2024-03-11T19:32:05.190 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T19:32:05.190 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} --archive /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293 --owner scheduled_teuthology@teuthology --name teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi --verbose -- /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293/orig.config.yaml 2024-03-11T19:32:05.248 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T19:32:05.345 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591293' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 94a539707733325322964c6a187c6a1b41be1daf subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b6a4997be2b8bb74411268cc42753f29ae9a78ad owner: scheduled_teuthology@teuthology priority: 250 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8179 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 297/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b6a4997be2b8bb74411268cc42753f29ae9a78ad syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJlNkDZxM21kiCbCyIGT8AvJPTJYmU5+z6GdS64/hYfKQSwUfkugIdbhrNHJ2xMRntb0q5YkB74XuEcHdyFDCiY= smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBJ/ri8dGEp9lL36CgRPgII75UmcwO8RWA7h7c4PWmi4uOXvfSO7VKlYI6RxG49mGU4XOl1fiEPHfJ+apzkF91Y= smithi161.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJaVzOJvnEd6jcckfqck5YksHEswmVBjlrHM8bwCvd8txKxJyeXQDSDhBQnBUwDA6mSh3j0tXJkKCiyrt/oPwmU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/fsync-tester.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_14:02:35 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T19:32:05.346 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa; will attempt to use it 2024-03-11T19:32:05.346 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks 2024-03-11T19:32:05.347 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T19:32:05.348 INFO:teuthology.task.internal:Checking packages... 2024-03-11T19:32:05.372 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-11T19:32:05.372 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T19:32:05.373 INFO:teuthology.packaging:ref: None 2024-03-11T19:32:05.373 INFO:teuthology.packaging:tag: None 2024-03-11T19:32:05.373 INFO:teuthology.packaging:branch: squid 2024-03-11T19:32:05.373 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T19:32:05.373 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-11T19:32:05.651 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-11T19:32:05.653 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T19:32:05.660 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T19:32:05.660 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T19:32:05.667 INFO:teuthology.task.internal:Saving configuration 2024-03-11T19:32:05.687 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T19:32:05.693 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T19:32:05.719 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 19:26:50.755900', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJlNkDZxM21kiCbCyIGT8AvJPTJYmU5+z6GdS64/hYfKQSwUfkugIdbhrNHJ2xMRntb0q5YkB74XuEcHdyFDCiY='} 2024-03-11T19:32:05.744 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 19:26:50.754045', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBJ/ri8dGEp9lL36CgRPgII75UmcwO8RWA7h7c4PWmi4uOXvfSO7VKlYI6RxG49mGU4XOl1fiEPHfJ+apzkF91Y='} 2024-03-11T19:32:05.769 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 19:26:50.757091', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJaVzOJvnEd6jcckfqck5YksHEswmVBjlrHM8bwCvd8txKxJyeXQDSDhBQnBUwDA6mSh3j0tXJkKCiyrt/oPwmU='} 2024-03-11T19:32:05.769 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T19:32:05.775 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-11T19:32:05.775 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-11T19:32:05.776 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-11T19:32:05.776 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T19:32:05.880 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f59f0161160>, signals=[15]) 2024-03-11T19:32:05.880 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T19:32:05.886 INFO:teuthology.task.internal:Opening connections... 2024-03-11T19:32:05.887 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2024-03-11T19:32:05.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T19:32:05.984 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-03-11T19:32:05.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T19:32:06.070 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2024-03-11T19:32:06.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T19:32:06.158 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T19:32:06.165 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2024-03-11T19:32:06.183 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2024-03-11T19:32:06.184 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2024-03-11T19:32:06.240 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2024-03-11T19:32:06.240 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="9" 2024-03-11T19:32:06.240 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2024-03-11T19:32:06.240 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2024-03-11T19:32:06.240 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="9" 2024-03-11T19:32:06.240 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el9" 2024-03-11T19:32:06.240 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T19:32:06.240 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2024-03-11T19:32:06.240 INFO:teuthology.orchestra.run.smithi097.stdout:LOGO="fedora-logo-icon" 2024-03-11T19:32:06.241 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T19:32:06.241 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2024-03-11T19:32:06.241 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T19:32:06.241 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T19:32:06.241 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T19:32:06.241 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2024-03-11T19:32:06.262 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-03-11T19:32:06.279 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-03-11T19:32:06.279 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-03-11T19:32:06.337 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-03-11T19:32:06.337 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-03-11T19:32:06.337 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-03-11T19:32:06.337 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-03-11T19:32:06.337 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-03-11T19:32:06.337 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-03-11T19:32:06.337 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T19:32:06.337 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-03-11T19:32:06.337 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-03-11T19:32:06.337 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T19:32:06.338 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-03-11T19:32:06.338 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T19:32:06.338 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T19:32:06.338 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T19:32:06.338 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-03-11T19:32:06.361 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2024-03-11T19:32:06.379 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2024-03-11T19:32:06.380 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2024-03-11T19:32:06.436 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Stream" 2024-03-11T19:32:06.436 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="9" 2024-03-11T19:32:06.436 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2024-03-11T19:32:06.436 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2024-03-11T19:32:06.436 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="9" 2024-03-11T19:32:06.436 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el9" 2024-03-11T19:32:06.436 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T19:32:06.436 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2024-03-11T19:32:06.437 INFO:teuthology.orchestra.run.smithi161.stdout:LOGO="fedora-logo-icon" 2024-03-11T19:32:06.437 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T19:32:06.437 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://centos.org/" 2024-03-11T19:32:06.437 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T19:32:06.437 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T19:32:06.437 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T19:32:06.437 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2024-03-11T19:32:06.461 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T19:32:06.472 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T19:32:06.478 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T19:32:06.478 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2024-03-11T19:32:06.480 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-03-11T19:32:06.482 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2024-03-11T19:32:06.498 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T19:32:06.504 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T19:32:06.504 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2024-03-11T19:32:06.537 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-03-11T19:32:06.539 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2024-03-11T19:32:06.822 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T19:32:06.892 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T19:32:06.908 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-11T19:32:06.908 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-11T19:32:06.908 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-11T19:32:06.908 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T19:32:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T19:32:06.925 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-11T19:32:07.229 DEBUG:teuthology.task.kernel:sha1 for client.0 is dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 2024-03-11T19:32:07.229 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412"... 2024-03-11T19:32:07.229 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-03-11T19:32:07.247 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-361.el9.x86_64 2024-03-11T19:32:07.247 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 2024-03-11T19:32:07.248 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-11T19:32:07.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum upgrade -y linux-firmware 2024-03-11T19:32:09.491 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-11T19:32:11.091 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-11T19:32:12.048 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-11T19:32:13.916 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-11T19:32:25.949 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 731 kB/s | 6.1 MB 00:08 2024-03-11T19:32:27.550 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2024-03-11T19:32:28.610 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-11T19:32:33.576 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-11T19:32:35.926 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T19:32:35.937 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T19:32:35.937 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-11T19:32:35.937 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T19:32:35.937 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2024-03-11T19:32:35.937 INFO:teuthology.orchestra.run.smithi097.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-11T19:32:35.937 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:32:35.938 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T19:32:35.938 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T19:32:35.938 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 1 Package 2024-03-11T19:32:35.938 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:32:35.938 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 375 M 2024-03-11T19:32:35.938 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T19:32:44.135 INFO:teuthology.orchestra.run.smithi097.stdout:linux-firmware-20240219-143.el9.noarch.rpm 46 MB/s | 375 MB 00:08 2024-03-11T19:32:44.136 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T19:32:44.137 INFO:teuthology.orchestra.run.smithi097.stdout:Total 46 MB/s | 375 MB 00:08 2024-03-11T19:32:44.137 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T19:32:44.229 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T19:32:44.229 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T19:32:46.686 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T19:32:46.686 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T19:32:52.964 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T19:32:53.187 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-11T19:32:53.285 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-11T19:32:54.555 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-11T19:32:54.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-11T19:32:54.917 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-11T19:32:54.917 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:32:54.917 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2024-03-11T19:32:54.917 INFO:teuthology.orchestra.run.smithi097.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-11T19:32:54.918 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:32:54.918 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T19:32:55.274 INFO:teuthology.task.kernel:Downloading kernel dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 on client.0... 2024-03-11T19:32:55.275 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 2024-03-11T19:32:55.495 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412/centos/9/flavors/default/x86_64/ 2024-03-11T19:32:55.495 DEBUG:teuthology.orchestra.run.smithi097:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc7_gdd5c1cd17167-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-11T19:32:55.498 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi097.front.sepia.ceph.com to complete... 2024-03-11T19:33:08.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11 19:33:08 URL:https://2.chacra.ceph.com/r/kernel/testing/dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412/centos/9/flavors/default/x86_64/kernel-6.8.0_rc7_gdd5c1cd17167-1.x86_64.rpm [336216379/336216379] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-11T19:33:08.605 INFO:teuthology.orchestra.run.smithi097.stderr:FINISHED --2024-03-11 19:33:08-- 2024-03-11T19:33:08.605 INFO:teuthology.orchestra.run.smithi097.stderr:Total wall clock time: 13s 2024-03-11T19:33:08.605 INFO:teuthology.orchestra.run.smithi097.stderr:Downloaded: 1 files, 321M in 13s (25.2 MB/s) 2024-03-11T19:33:08.608 INFO:teuthology.task.kernel:Installing kernel dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 on client.0... 2024-03-11T19:33:08.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-11T19:33:10.541 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying... ######################################## 2024-03-11T19:33:10.592 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing... ######################################## 2024-03-11T19:33:10.672 INFO:teuthology.orchestra.run.smithi097.stdout:Updating / installing... 2024-03-11T19:33:18.291 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-6.8.0_rc7_gdd5c1cd17167-1 ######################################## 2024-03-11T19:33:34.144 INFO:teuthology.orchestra.run.smithi097.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T19:33:56.520 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi097.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-11T19:33:56.520 DEBUG:teuthology.orchestra.run.smithi097:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-11T19:33:56.604 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167 2024-03-11T19:33:56.605 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/System.map 2024-03-11T19:33:56.605 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/config 2024-03-11T19:33:56.605 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel 2024-03-11T19:33:56.605 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch 2024-03-11T19:33:56.605 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86 2024-03-11T19:33:56.605 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto 2024-03-11T19:33:56.605 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-11T19:33:56.605 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-11T19:33:56.605 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-11T19:33:56.605 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-11T19:33:56.605 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-11T19:33:56.606 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kernel 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kernel/cpu 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kernel/cpu/mce 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kvm 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kvm/kvm.ko 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/ansi_cprng.ko 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/anubis.ko 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/arc4.ko 2024-03-11T19:33:56.607 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx 2024-03-11T19:33:56.608 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_memcpy.ko 2024-03-11T19:33:56.608 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_pq.ko 2024-03-11T19:33:56.608 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-11T19:33:56.608 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_tx.ko 2024-03-11T19:33:56.608 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_xor.ko 2024-03-11T19:33:56.608 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/raid6test.ko 2024-03-11T19:33:56.608 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/authenc.ko 2024-03-11T19:33:56.608 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/authencesn.ko 2024-03-11T19:33:56.608 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/blake2b_generic.ko 2024-03-11T19:33:56.608 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/blowfish_common.ko 2024-03-11T19:33:56.608 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/blowfish_generic.ko 2024-03-11T19:33:56.608 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/camellia_generic.ko 2024-03-11T19:33:56.608 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cast5_generic.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cast6_generic.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cast_common.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/ccm.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cmac.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/crc32_generic.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/crypto_engine.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/crypto_user.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cts.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/des_generic.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/dh_generic.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/echainiv.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/essiv.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/fcrypt.ko 2024-03-11T19:33:56.609 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/khazad.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/lz4.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/lz4hc.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/md4.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/michael_mic.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/pcbc.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/pcrypt.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/rmd160.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/seed.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/serpent_generic.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/tcrypt.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/tea.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/twofish_common.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/twofish_generic.ko 2024-03-11T19:33:56.610 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/vmac.ko 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/wp512.ko 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/xcbc.ko 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/xor.ko 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/xxhash_generic.ko 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/acpi_pad.ko 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/custom_method.ko 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/ec_sys.ko 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/platform_profile.ko 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/sbs.ko 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/sbshc.ko 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/video.ko 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata 2024-03-11T19:33:56.611 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/acard-ahci.ko 2024-03-11T19:33:56.612 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/ahci_platform.ko 2024-03-11T19:33:56.612 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/ata_generic.ko 2024-03-11T19:33:56.612 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/libahci_platform.ko 2024-03-11T19:33:56.612 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_acpi.ko 2024-03-11T19:33:56.612 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_ali.ko 2024-03-11T19:33:56.612 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_amd.ko 2024-03-11T19:33:56.612 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_artop.ko 2024-03-11T19:33:56.612 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_atiixp.ko 2024-03-11T19:33:56.612 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_atp867x.ko 2024-03-11T19:33:56.612 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_cmd640.ko 2024-03-11T19:33:56.612 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_cmd64x.ko 2024-03-11T19:33:56.612 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_cypress.ko 2024-03-11T19:33:56.612 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_efar.ko 2024-03-11T19:33:56.613 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_hpt366.ko 2024-03-11T19:33:56.613 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_hpt37x.ko 2024-03-11T19:33:56.613 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-11T19:33:56.613 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-11T19:33:56.613 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_it8213.ko 2024-03-11T19:33:56.613 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_it821x.ko 2024-03-11T19:33:56.613 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_jmicron.ko 2024-03-11T19:33:56.613 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_marvell.ko 2024-03-11T19:33:56.613 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_mpiix.ko 2024-03-11T19:33:56.613 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_netcell.ko 2024-03-11T19:33:56.613 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_ninja32.ko 2024-03-11T19:33:56.613 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_ns87410.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_ns87415.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_oldpiix.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_opti.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_optidma.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_pcmcia.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_piccolo.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_rdc.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_sch.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_serverworks.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_sil680.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_sis.ko 2024-03-11T19:33:56.614 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_sl82c105.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_triflex.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_via.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pdc_adma.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_inic162x.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_mv.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_nv.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_promise.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_qstor.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_sil.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_sil24.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_sis.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_svw.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_sx4.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_uli.ko 2024-03-11T19:33:56.615 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_via.ko 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_vsc.ko 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/auxdisplay 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/auxdisplay/ks0108.ko 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/base 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/base/regmap 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/bcma 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/bcma/bcma.ko 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/aoe 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/aoe/aoe.ko 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/brd.ko 2024-03-11T19:33:56.616 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/drbd 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/drbd/drbd.ko 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/floppy.ko 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/loop.ko 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/mtip32xx 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/nbd.ko 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/null_blk 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/null_blk/null_blk.ko 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/pktcdvd.ko 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/rbd.ko 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/virtio_blk.ko 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/xen-blkback 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-11T19:33:56.617 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/xen-blkfront.ko 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/zram 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/zram/zram.ko 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hangcheck-timer.ko 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/via-rng.ko 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-11T19:33:56.618 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/lp.ko 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/mwave 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/mwave/mwave.ko 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ppdev.ko 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tlclk.ko 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm.ko 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-11T19:33:56.619 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/uv_mmtimer.ko 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/virtio_console.ko 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/ccp 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/ccp/ccp.ko 2024-03-11T19:33:56.620 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat/qat_common 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/padlock-aes.ko 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/padlock-sha.ko 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/virtio 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dca 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dca/dca.ko 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/devfreq 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma 2024-03-11T19:33:56.621 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma/dw 2024-03-11T19:33:56.622 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-11T19:33:56.622 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma/ioat 2024-03-11T19:33:56.622 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-11T19:33:56.622 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac 2024-03-11T19:33:56.622 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/amd64_edac.ko 2024-03-11T19:33:56.622 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/e752x_edac.ko 2024-03-11T19:33:56.622 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/edac_mce_amd.ko 2024-03-11T19:33:56.622 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i3000_edac.ko 2024-03-11T19:33:56.622 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i3200_edac.ko 2024-03-11T19:33:56.622 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i5100_edac.ko 2024-03-11T19:33:56.622 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i5400_edac.ko 2024-03-11T19:33:56.622 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i7300_edac.ko 2024-03-11T19:33:56.622 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i7core_edac.ko 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i82975x_edac.ko 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/ie31200_edac.ko 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/sb_edac.ko 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/x38_edac.ko 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/firmware 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/firmware/edd.ko 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/gpio 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/gpio/gpio-ich.ko 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-appleir.ko 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-aureal.ko 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-axff.ko 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-dr.ko 2024-03-11T19:33:56.623 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-elecom.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-elo.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-emsff.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-gaff.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-gt683r.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-gyration.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-holtekff.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-hyperv.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-icade.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-keytouch.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-kye.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-lcpower.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-led.ko 2024-03-11T19:33:56.624 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-lenovo.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-multitouch.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-ortek.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-penmount.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-petalynx.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-picolcd.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-pl.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-plantronics.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-primax.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-rmi.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-common.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-11T19:33:56.625 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-saitek.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-samsung.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-sjoy.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-sony.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-speedlink.ko 2024-03-11T19:33:56.626 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-steelseries.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-sunplus.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-tivo.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-tmff.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-topseed.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-twinhan.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-uclogic.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-waltop.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-wiimote.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-xinmo.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-zpff.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-zydacron.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/uhid.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/wacom.ko 2024-03-11T19:33:56.627 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hv 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hv/hv_balloon.ko 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hv/hv_utils.ko 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hv/hv_vmbus.ko 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/abituguru.ko 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/abituguru3.ko 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ad7414.ko 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ad7418.ko 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adc128d818.ko 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm1025.ko 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm1026.ko 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm1029.ko 2024-03-11T19:33:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm1031.ko 2024-03-11T19:33:56.629 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm9240.ko 2024-03-11T19:33:56.629 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ads7828.ko 2024-03-11T19:33:56.629 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7410.ko 2024-03-11T19:33:56.629 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7411.ko 2024-03-11T19:33:56.629 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7462.ko 2024-03-11T19:33:56.629 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7470.ko 2024-03-11T19:33:56.629 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7475.ko 2024-03-11T19:33:56.629 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7x10.ko 2024-03-11T19:33:56.629 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/amc6821.ko 2024-03-11T19:33:56.629 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/applesmc.ko 2024-03-11T19:33:56.629 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/asb100.ko 2024-03-11T19:33:56.629 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/asc7621.ko 2024-03-11T19:33:56.629 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/atxp1.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/coretemp.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/dme1737.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ds1621.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ds620.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/emc1403.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/emc6w201.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/f71805f.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/f71882fg.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/f75375s.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/fam15h_power.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/fschmd.ko 2024-03-11T19:33:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/g760a.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/g762.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/gl518sm.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/gl520sm.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/i5500_temp.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/i5k_amb.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ibmaem.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ibmpex.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ina209.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ina2xx.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/it87.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/k10temp.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/k8temp.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lineage-pem.ko 2024-03-11T19:33:56.631 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm63.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm73.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm75.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm77.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm78.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm80.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm83.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm85.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm87.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm90.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm92.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm93.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm95234.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm95241.ko 2024-03-11T19:33:56.632 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm95245.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc2945.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4151.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4215.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4222.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4245.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4260.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4261.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max16065.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max1619.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max1668.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max197.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max6639.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max6650.ko 2024-03-11T19:33:56.633 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max6697.ko 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/mcp3021.ko 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/nct6683.ko 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/nct6775-core.ko 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/nct6775.ko 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/nct7802.ko 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pc87360.ko 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pc87427.ko 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pcf8591.ko 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-11T19:33:56.634 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/powr1220.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sch5627.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sch5636.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sht15.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sht21.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/shtc1.ko 2024-03-11T19:33:56.635 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sis5595.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/smsc47b397.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/smsc47m1.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/smsc47m192.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/thmc50.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/tmp102.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/tmp103.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/tmp401.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/tmp421.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/via-cputemp.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/via686a.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/vt1211.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/vt8231.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83627ehf.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83627hf.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83781d.ko 2024-03-11T19:33:56.636 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83791d.ko 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83792d.ko 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83793.ko 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83795.ko 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83l785ts.ko 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83l786ng.ko 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/algos 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-11T19:33:56.637 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-11T19:33:56.638 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-dev.ko 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-mux.ko 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-smbus.ko 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-stub.ko 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/ib_core.ko 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-11T19:33:56.639 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/cxgb4 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mlx4 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mlx5 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mthca 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/ipoib 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-11T19:33:56.640 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/iser 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/isert 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/srp 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/srpt 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/gameport 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/gameport/gameport.ko 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/joydev.ko 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/keyboard 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-11T19:33:56.641 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/appletouch.ko 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/bcm5974.ko 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/cyapatp.ko 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/sermouse.ko 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/rmi4 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio/altera_ps2.ko 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio/arc_ps2.ko 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-11T19:33:56.642 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio/serio_raw.ko 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/sparse-keymap.ko 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/leds-blinkm.ko 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/leds-lm3530.ko 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/leds-lp3944.ko 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/leds-ss4200.ko 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-11T19:33:56.643 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/bcache 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/bcache/bcache.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-bio-prison.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-cache-smq.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-cache.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-crypt.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-delay.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-flakey.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-log-userspace.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-multipath.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-queue-length.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-raid.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-round-robin.ko 2024-03-11T19:33:56.644 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-service-time.ko 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-switch.ko 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-thin-pool.ko 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-verity.ko 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/persistent-data 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/raid0.ko 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/raid1.ko 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/raid10.ko 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/raid456.ko 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptbase.ko 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptctl.ko 2024-03-11T19:33:56.645 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptfc.ko 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptlan.ko 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptsas.ko 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptscsih.ko 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptspi.ko 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/lpc_ich.ko 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/lpc_sch.ko 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/mfd-core.ko 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/sm501.ko 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/viperboard.ko 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/vx855.ko 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/wl1273-core.ko 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/altera-stapl 2024-03-11T19:33:56.646 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/apds9802als.ko 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/apds990x.ko 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/bh1770glc.ko 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/cb710 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/cb710/cb710.ko 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/eeprom 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/eeprom/at24.ko 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/eeprom/max6875.ko 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/enclosure.ko 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/hpilo.ko 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/isl29003.ko 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/isl29020.ko 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/lis3lv02d 2024-03-11T19:33:56.647 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/mei 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/mei/mei-me.ko 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/mei/mei-txe.ko 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/mei/mei.ko 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-gru 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-xp 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/tifm_7xx1.ko 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/tifm_core.ko 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/tsl2550.ko 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/vmw_balloon.ko 2024-03-11T19:33:56.648 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/vmw_vmci 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/bonding 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/bonding/bonding.ko 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/b53 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/mv88e6xxx 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dummy.ko 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/eql.ko 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet 2024-03-11T19:33:56.649 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/adaptec 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/agere 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/alteon 2024-03-11T19:33:56.650 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/altera 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/amd 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/alx 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atlx 2024-03-11T19:33:56.651 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/genet 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/brocade 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/brocade/bna 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-11T19:33:56.652 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio 2024-03-11T19:33:56.653 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-11T19:33:56.653 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-11T19:33:56.653 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-11T19:33:56.653 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-11T19:33:56.653 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-11T19:33:56.653 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-11T19:33:56.653 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-11T19:33:56.653 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-11T19:33:56.653 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-11T19:33:56.653 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-11T19:33:56.653 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/cisco 2024-03-11T19:33:56.653 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/cisco/enic 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dlink 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dnet.ko 2024-03-11T19:33:56.654 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/emulex 2024-03-11T19:33:56.655 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/emulex/benet 2024-03-11T19:33:56.655 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-11T19:33:56.655 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/ethoc.ko 2024-03-11T19:33:56.655 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/fealnx.ko 2024-03-11T19:33:56.655 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel 2024-03-11T19:33:56.655 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-11T19:33:56.655 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e1000 2024-03-11T19:33:56.655 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-11T19:33:56.655 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e1000e 2024-03-11T19:33:56.655 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-11T19:33:56.655 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/fm10k 2024-03-11T19:33:56.655 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-11T19:33:56.655 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/i40e 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/iavf 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/igb 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/igbvf 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/jme.ko 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/marvell 2024-03-11T19:33:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/micrel 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/myricom 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-11T19:33:56.657 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/natsemi 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/neterion 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/nvidia 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/packetengines 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-11T19:33:56.658 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/rdc 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/rocker 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sfc 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/silan 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-11T19:33:56.659 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sis 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/stmicro 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-11T19:33:56.660 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/tehuti 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/ti 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/via 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/wiznet 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/xircom 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-11T19:33:56.661 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/geneve.ko 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/hyperv 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ieee802154 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ifb.ko 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ipvlan 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/macvlan.ko 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/macvtap.ko 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio.ko 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-11T19:33:56.662 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mii.ko 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/net_failover.ko 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/netconsole.ko 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/nlmon.ko 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/pcs 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/amd.ko 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/bcm87xx.ko 2024-03-11T19:33:56.663 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/broadcom.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/cicada.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/davicom.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/dp83640.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/et1011c.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/icplus.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/lxt.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/marvell.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/micrel.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/national.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/phylink.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/qsemi.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/realtek.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/smsc.ko 2024-03-11T19:33:56.664 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/ste10Xp.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/vitesse.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_async.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/pppoe.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/pppox.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/pptp.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/slip 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/slip/slhc.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/slip/slip.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/sungem_phy.ko 2024-03-11T19:33:56.665 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/tap.ko 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team.ko 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_random.ko 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/tun.ko 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/veth.ko 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/virtio_net.ko 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/vmxnet3 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/vxlan 2024-03-11T19:33:56.666 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/vxlan/vxlan.ko 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/xen-netback 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/xen-netfront.ko 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ntb 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ntb/ntb.ko 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/nvme 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/nvme/host 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/nvme/host/nvme-core.ko 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/nvme/host/nvme.ko 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport/parport.ko 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport/parport_cs.ko 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport/parport_pc.ko 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport/parport_serial.ko 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci 2024-03-11T19:33:56.667 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/hotplug 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/pcie 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/xen-pcifront.ko 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pcmcia 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pcmcia/i82092.ko 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pcmcia/pd6729.ko 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pinctrl 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pinctrl/intel 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-11T19:33:56.668 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/acerhdf.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/hdaps.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/intel 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/intel_ips.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-11T19:33:56.669 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/wmi.ko 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/powercap 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/clients 2024-03-11T19:33:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/clients/pps_parport.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/pps_core.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ptp 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ptp/ptp.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ptp/ptp_kvm.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-11T19:33:56.671 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-em3027.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-max6900.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-11T19:33:56.672 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-x1205.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/3w-9xxx.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/3w-sas.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/3w-xxxx.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/BusLogic.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/a100u2w.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aacraid 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-11T19:33:56.673 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/advansys.ko 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aic7xxx 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/am53c974.ko 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/arcmsr 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/atp870u.ko 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/be2iscsi 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bfa 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bfa/bfa.ko 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bnx2fc 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bnx2i 2024-03-11T19:33:56.674 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/ch.ko 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/csiostor 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/dc395x.ko 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-11T19:33:56.675 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/dmx3191d.ko 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/esas2r 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/esp_scsi.ko 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fcoe 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fnic 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fnic/fnic.ko 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/hpsa.ko 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/hptiop.ko 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/hv_storvsc.ko 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/initio.ko 2024-03-11T19:33:56.676 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/ipr.ko 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/ips.ko 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/isci 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/isci/isci.ko 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libfc 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libfc/libfc.ko 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libiscsi.ko 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libsas 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libsas/libsas.ko 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/lpfc 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid.ko 2024-03-11T19:33:56.677 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-11T19:33:56.678 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-11T19:33:56.678 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-11T19:33:56.678 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mpt3sas 2024-03-11T19:33:56.678 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-11T19:33:56.678 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mvsas 2024-03-11T19:33:56.678 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-11T19:33:56.678 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mvumi.ko 2024-03-11T19:33:56.678 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/pm8001 2024-03-11T19:33:56.678 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-11T19:33:56.678 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/pmcraid.ko 2024-03-11T19:33:56.678 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla1280.ko 2024-03-11T19:33:56.678 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla2xxx 2024-03-11T19:33:56.679 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-11T19:33:56.679 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-11T19:33:56.679 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla4xxx 2024-03-11T19:33:56.679 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-11T19:33:56.679 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/raid_class.ko 2024-03-11T19:33:56.679 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_debug.ko 2024-03-11T19:33:56.679 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-11T19:33:56.679 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-11T19:33:56.679 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-11T19:33:56.679 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-11T19:33:56.679 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-11T19:33:56.679 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/ses.ko 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/st.ko 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/stex.ko 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/sym53c8xx_2 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/virtio_scsi.ko 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/wd719x.ko 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ssb 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ssb/ssb.ko 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/iscsi 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/loopback 2024-03-11T19:33:56.680 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_file.ko 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_iblock.ko 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_mod.ko 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_pscsi.ko 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_user.ko 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/tcm_fc 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-11T19:33:56.681 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/ipwireless 2024-03-11T19:33:56.682 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/n_gsm.ko 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/n_hdlc.ko 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/nozomi.ko 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/8250 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/arc_uart.ko 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/jsm 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs/core 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs/host 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-11T19:33:56.683 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio.ko 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio_aec.ko 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio_cif.ko 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio_pci_generic.ko 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio_sercos3.ko 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class/cdc-acm.ko 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class/usblp.ko 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class/usbtmc.ko 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/host 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/image 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/image/mdc800.ko 2024-03-11T19:33:56.684 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/image/microtek.ko 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/adutux.ko 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/appledisplay.ko 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/emi26.ko 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/emi62.ko 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/ezusb.ko 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/idmouse.ko 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/iowarrior.ko 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/ldusb.ko 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/legousbtower.ko 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/sisusbvga 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-11T19:33:56.685 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/usb3503.ko 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/usblcd.ko 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/uss720.ko 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/yurex.ko 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/phy 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/phy/phy-generic.ko 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/aircable.ko 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ark3116.ko 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ch341.ko 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/cp210x.ko 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/cyberjack.ko 2024-03-11T19:33:56.686 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/empeg.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/io_ti.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ipaq.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ipw.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ir-usb.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/keyspan.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/mct_u232.ko 2024-03-11T19:33:56.687 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/mos7720.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/mos7840.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/navman.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/omninet.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/opticon.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/option.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/oti6858.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/pl2303.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/qcaux.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/qcserial.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/quatech2.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/safe_serial.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/sierra.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-11T19:33:56.688 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ssu100.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/symbolserial.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/usb_debug.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/visor.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/whiteheat.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/uas.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-11T19:33:56.689 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-karma.ko 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/usb-storage.ko 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/usbip 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio 2024-03-11T19:33:56.690 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/pci 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/vfio.ko 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost/vhost.ko 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost/vhost_net.ko 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost/vhost_scsi.ko 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video/backlight 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video/backlight/apple_bl.ko 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video/backlight/lcd.ko 2024-03-11T19:33:56.691 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio.ko 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_balloon.ko 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_mmio.ko 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_pci.ko 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_ring.ko 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/masters 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/masters/ds2482.ko 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/masters/ds2490.ko 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-11T19:33:56.692 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/wire.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/hpwdt.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/i6300esb.ko 2024-03-11T19:33:56.693 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/ib700wdt.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/ibmasr.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/it87_wdt.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/machzwd.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/nv_tco.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/softdog.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-11T19:33:56.694 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/via_wdt.ko 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/wdt_pci.ko 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/xen_wdt.ko 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-evtchn.ko 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-gntalloc.ko 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-gntdev.ko 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-pciback 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-privcmd.ko 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xenfs 2024-03-11T19:33:56.695 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/9p 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/9p/9p.ko 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/affs 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/affs/affs.ko 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/befs 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/befs/befs.ko 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/binfmt_misc.ko 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/btrfs 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/btrfs/btrfs.ko 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/cachefiles 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/cachefiles/cachefiles.ko 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ceph 2024-03-11T19:33:56.696 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ceph/ceph.ko 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/coda 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/coda/coda.ko 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/cramfs 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/cramfs/cramfs.ko 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/dlm 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/dlm/dlm.ko 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ecryptfs 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/f2fs 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/f2fs/f2fs.ko 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fat 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fat/fat.ko 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fat/msdos.ko 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fat/vfat.ko 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fuse 2024-03-11T19:33:56.697 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fuse/cuse.ko 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fuse/fuse.ko 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/gfs2 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/gfs2/gfs2.ko 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/hfs 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/hfs/hfs.ko 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/hfsplus 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/hfsplus/hfsplus.ko 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/isofs 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/isofs/isofs.ko 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/jfs 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/jfs/jfs.ko 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/lockd 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/lockd/lockd.ko 2024-03-11T19:33:56.698 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/minix 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/minix/minix.ko 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/netfs 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/netfs/netfs.ko 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/blocklayout 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/filelayout 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/flexfilelayout 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/nfs.ko 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/nfsv3.ko 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/nfsv4.ko 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs_common 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs_common/grace.ko 2024-03-11T19:33:56.699 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs_common/nfs_acl.ko 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfsd 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfsd/nfsd.ko 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nilfs2 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nilfs2/nilfs2.ko 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-celtic.ko 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-centeuro.ko 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-croatian.ko 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-cyrillic.ko 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-gaelic.ko 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-greek.ko 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-iceland.ko 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-inuit.ko 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-roman.ko 2024-03-11T19:33:56.700 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-romanian.ko 2024-03-11T19:33:56.701 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-turkish.ko 2024-03-11T19:33:56.701 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp1250.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp1251.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp1255.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp737.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp775.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp850.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp852.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp855.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp857.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp860.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp861.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp862.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp863.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp864.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp865.ko 2024-03-11T19:33:56.702 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp866.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp869.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp874.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp932.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp936.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp949.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp950.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_euc-jp.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-1.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-13.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-14.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-15.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-2.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-3.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-4.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-5.ko 2024-03-11T19:33:56.703 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-6.ko 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-7.ko 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-9.ko 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_koi8-r.ko 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_koi8-ru.ko 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_koi8-u.ko 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_utf8.ko 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/cluster 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/dlm 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/dlmfs 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-11T19:33:56.704 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/ocfs2.ko 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/overlayfs 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/overlayfs/overlay.ko 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/pstore 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/pstore/ramoops.ko 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/reiserfs 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/reiserfs/reiserfs.ko 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/romfs 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/romfs/romfs.ko 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/client 2024-03-11T19:33:56.705 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/client/cifs.ko 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/common 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/common/cifs_arc4.ko 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/common/cifs_md4.ko 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/squashfs 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/squashfs/squashfs.ko 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/sysv 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/sysv/sysv.ko 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/udf 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/udf/udf.ko 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ufs 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ufs/ufs.ko 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/xfs 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/xfs/xfs.ko 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/rcu 2024-03-11T19:33:56.706 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/rcu/rcutorture.ko 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/torture.ko 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/trace 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/asn1_encoder.ko 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crc-itu-t.ko 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crc8.ko 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crypto 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crypto/libarc4.ko 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crypto/libdes.ko 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/libcrc32c.ko 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/lru_cache.ko 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/lz4 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/lz4/lz4_compress.ko 2024-03-11T19:33:56.707 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/lz4/lz4hc_compress.ko 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/math 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/math/cordic.ko 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/raid6 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/raid6/raid6_pq.ko 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/reed_solomon 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/ts_bm.ko 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/ts_fsm.ko 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/ts_kmp.ko 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/zstd 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/zstd/zstd_compress.ko 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/mm 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/mm/hwpoison-inject.ko 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net 2024-03-11T19:33:56.708 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/6lowpan.ko 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_dest.ko 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_fragment.ko 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_hop.ko 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_mobility.ko 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_routing.ko 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_udp.ko 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/garp.ko 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/mrp.ko 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/p8022.ko 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/psnap.ko 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/stp.ko 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/8021q 2024-03-11T19:33:56.709 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/8021q/8021q.ko 2024-03-11T19:33:56.710 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p 2024-03-11T19:33:56.710 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p/9pnet.ko 2024-03-11T19:33:56.710 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p/9pnet_fd.ko 2024-03-11T19:33:56.710 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p/9pnet_rdma.ko 2024-03-11T19:33:56.710 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p/9pnet_virtio.ko 2024-03-11T19:33:56.710 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/batman-adv 2024-03-11T19:33:56.710 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/batman-adv/batman-adv.ko 2024-03-11T19:33:56.710 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge 2024-03-11T19:33:56.710 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/br_netfilter.ko 2024-03-11T19:33:56.710 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/bridge.ko 2024-03-11T19:33:56.710 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter 2024-03-11T19:33:56.710 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-11T19:33:56.711 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-11T19:33:56.711 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-11T19:33:56.711 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-11T19:33:56.711 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-11T19:33:56.711 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-11T19:33:56.711 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-11T19:33:56.711 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-11T19:33:56.711 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-11T19:33:56.711 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-11T19:33:56.711 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-11T19:33:56.711 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-11T19:33:56.711 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-11T19:33:56.711 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebtables.ko 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ceph 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ceph/libceph.ko 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/core 2024-03-11T19:33:56.712 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/core/failover.ko 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/core/pktgen.ko 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp/dccp.ko 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp/dccp_diag.ko 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp/dccp_ipv4.ko 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp/dccp_ipv6.ko 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dns_resolver 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dns_resolver/dns_resolver.ko 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/dsa_core.ko 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/tag_brcm.ko 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/tag_dsa.ko 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/tag_none.ko 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/tag_trailer.ko 2024-03-11T19:33:56.713 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154/6lowpan 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154/ieee802154.ko 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ah4.ko 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/esp4.ko 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/fou.ko 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/gre.ko 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/inet_diag.ko 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ip_gre.ko 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ip_tunnel.ko 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ip_vti.ko 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ipcomp.ko 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ipip.ko 2024-03-11T19:33:56.714 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-11T19:33:56.715 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-11T19:33:56.716 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-11T19:33:56.716 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-11T19:33:56.716 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-11T19:33:56.716 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-11T19:33:56.716 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-11T19:33:56.716 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-11T19:33:56.716 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_bic.ko 2024-03-11T19:33:56.716 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_dctcp.ko 2024-03-11T19:33:56.716 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_diag.ko 2024-03-11T19:33:56.716 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_highspeed.ko 2024-03-11T19:33:56.716 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_htcp.ko 2024-03-11T19:33:56.716 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_hybla.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_illinois.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_lp.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_scalable.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_vegas.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_veno.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_westwood.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_yeah.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tunnel4.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/udp_diag.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/udp_tunnel.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ah6.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/esp6.ko 2024-03-11T19:33:56.717 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/fou6.ko 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ip6_tunnel.ko 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ip6_vti.ko 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ipcomp6.ko 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-11T19:33:56.718 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-11T19:33:56.719 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/sit.ko 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/tunnel6.ko 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/key 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/key/af_key.ko 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_core.ko 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_eth.ko 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_ip.ko 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_ip6.ko 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_netlink.ko 2024-03-11T19:33:56.720 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_ppp.ko 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/llc 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/llc/llc.ko 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/mac802154 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/mac802154/mac802154.ko 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/mpls 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/mpls/mpls_gso.ko 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set.ko 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-11T19:33:56.721 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-11T19:33:56.722 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conncount.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-11T19:33:56.723 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_flow_table.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_log_syslog.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-11T19:33:56.724 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_irc.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_sip.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_tables.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink_log.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_chain_nat.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_compat.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_ct.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_fib.ko 2024-03-11T19:33:56.725 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_hash.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_limit.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_log.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_masq.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_nat.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_numgen.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_queue.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_quota.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_redir.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_reject.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_reject_inet.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_socket.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_tunnel.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_AUDIT.ko 2024-03-11T19:33:56.726 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_CT.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_DSCP.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_HL.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_HMARK.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_LED.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_LOG.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_NETMAP.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_NFLOG.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-11T19:33:56.727 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_RATEEST.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_SECMARK.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TEE.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TPROXY.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TRACE.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_addrtype.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_bpf.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_cgroup.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_cluster.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_comment.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_connbytes.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_connlabel.ko 2024-03-11T19:33:56.728 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_connlimit.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_connmark.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_conntrack.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_cpu.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_dccp.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_devgroup.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_dscp.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_ecn.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_esp.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_hashlimit.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_helper.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_hl.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_ipcomp.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_iprange.ko 2024-03-11T19:33:56.729 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_ipvs.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_l2tp.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_length.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_limit.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_mac.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_mark.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_multiport.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_nat.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_nfacct.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_osf.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_owner.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_physdev.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_pkttype.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_policy.ko 2024-03-11T19:33:56.730 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_quota.ko 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_rateest.ko 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_realm.ko 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_recent.ko 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_sctp.ko 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_set.ko 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_socket.ko 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_state.ko 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_statistic.ko 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_string.ko 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_tcpmss.ko 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_time.ko 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_u32.ko 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netlink 2024-03-11T19:33:56.731 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netlink/netlink_diag.ko 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/nsh 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/nsh/nsh.ko 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch/openvswitch.ko 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch/vport-geneve.ko 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch/vport-gre.ko 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch/vport-vxlan.ko 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/packet 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/packet/af_packet_diag.ko 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/rds 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/rds/rds.ko 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/rds/rds_rdma.ko 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/rds/rds_tcp.ko 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched 2024-03-11T19:33:56.732 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_csum.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_gact.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_mirred.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_nat.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_pedit.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_police.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_simple.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_skbedit.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_vlan.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_basic.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_bpf.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_flow.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_fw.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_route.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_u32.ko 2024-03-11T19:33:56.733 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_cmp.ko 2024-03-11T19:33:56.734 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_ipset.ko 2024-03-11T19:33:56.734 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_meta.ko 2024-03-11T19:33:56.734 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_nbyte.ko 2024-03-11T19:33:56.734 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_text.ko 2024-03-11T19:33:56.734 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_u32.ko 2024-03-11T19:33:56.734 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_choke.ko 2024-03-11T19:33:56.734 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_codel.ko 2024-03-11T19:33:56.734 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_drr.ko 2024-03-11T19:33:56.734 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_fq.ko 2024-03-11T19:33:56.734 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_gred.ko 2024-03-11T19:33:56.734 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_hfsc.ko 2024-03-11T19:33:56.734 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_hhf.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_htb.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_ingress.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_mqprio.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_mqprio_lib.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_multiq.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_netem.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_pie.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_plug.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_prio.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_qfq.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_red.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_sfb.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_sfq.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_tbf.ko 2024-03-11T19:33:56.735 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_teql.ko 2024-03-11T19:33:56.736 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sctp 2024-03-11T19:33:56.736 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sctp/sctp.ko 2024-03-11T19:33:56.736 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sctp/sctp_diag.ko 2024-03-11T19:33:56.736 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc 2024-03-11T19:33:56.736 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/auth_gss 2024-03-11T19:33:56.736 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-11T19:33:56.736 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-11T19:33:56.736 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/sunrpc.ko 2024-03-11T19:33:56.736 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/xprtrdma 2024-03-11T19:33:56.736 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-11T19:33:56.736 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/tipc 2024-03-11T19:33:56.736 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/tipc/diag.ko 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/tipc/tipc.ko 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/unix 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/unix/unix_diag.ko 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vsock.ko 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/xfrm 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys/encrypted-keys 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-11T19:33:56.737 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys/trusted-keys 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys/trusted-keys/trusted.ko 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/virt 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/virt/lib 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/virt/lib/irqbypass.ko 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.alias 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.alias.bin 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.builtin 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.builtin.alias.bin 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.builtin.bin 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.builtin.modinfo 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.dep 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.dep.bin 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.devname 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.order 2024-03-11T19:33:56.738 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.softdep 2024-03-11T19:33:56.739 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.symbols 2024-03-11T19:33:56.739 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.symbols.bin 2024-03-11T19:33:56.739 INFO:teuthology.orchestra.run.smithi097.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/vmlinuz 2024-03-11T19:33:56.743 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc7-gdd5c1cd17167 2024-03-11T19:33:56.743 DEBUG:teuthology.orchestra.run.smithi097:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-11T19:33:56.765 INFO:teuthology.orchestra.run.smithi097.stdout:postinstall scriptlet (using /bin/sh): 2024-03-11T19:33:56.765 INFO:teuthology.orchestra.run.smithi097.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-11T19:33:56.765 INFO:teuthology.orchestra.run.smithi097.stdout: /usr/bin/kernel-install add 6.8.0-rc7-gdd5c1cd17167 /lib/modules/6.8.0-rc7-gdd5c1cd17167/vmlinuz 2024-03-11T19:33:56.765 INFO:teuthology.orchestra.run.smithi097.stdout:fi 2024-03-11T19:33:56.765 INFO:teuthology.orchestra.run.smithi097.stdout:for file in vmlinuz System.map config; do 2024-03-11T19:33:56.765 INFO:teuthology.orchestra.run.smithi097.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc7-gdd5c1cd17167/${file}" "/boot/${file}-6.8.0-rc7-gdd5c1cd17167"; then 2024-03-11T19:33:56.766 INFO:teuthology.orchestra.run.smithi097.stdout: cp "/lib/modules/6.8.0-rc7-gdd5c1cd17167/${file}" "/boot/${file}-6.8.0-rc7-gdd5c1cd17167" 2024-03-11T19:33:56.766 INFO:teuthology.orchestra.run.smithi097.stdout: fi 2024-03-11T19:33:56.766 INFO:teuthology.orchestra.run.smithi097.stdout:done 2024-03-11T19:33:56.766 INFO:teuthology.orchestra.run.smithi097.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-11T19:33:56.766 INFO:teuthology.orchestra.run.smithi097.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-11T19:33:56.766 INFO:teuthology.orchestra.run.smithi097.stdout:new-kernel-pkg --remove 6.8.0-rc7-gdd5c1cd17167 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc7-gdd5c1cd17167.img 2024-03-11T19:33:56.766 INFO:teuthology.orchestra.run.smithi097.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-11T19:33:56.766 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-install remove 6.8.0-rc7-gdd5c1cd17167 2024-03-11T19:33:56.766 INFO:teuthology.orchestra.run.smithi097.stdout:fi 2024-03-11T19:33:56.766 INFO:teuthology.orchestra.run.smithi097.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-11T19:33:56.766 INFO:teuthology.orchestra.run.smithi097.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-11T19:33:56.766 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/update-bootloader --remove 6.8.0-rc7-gdd5c1cd17167 2024-03-11T19:33:56.767 INFO:teuthology.orchestra.run.smithi097.stdout:fi 2024-03-11T19:33:56.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo rpm -qi grub2-tools 2024-03-11T19:33:56.842 INFO:teuthology.orchestra.run.smithi097.stdout:Name : grub2-tools 2024-03-11T19:33:56.842 INFO:teuthology.orchestra.run.smithi097.stdout:Epoch : 1 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:Version : 2.06 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:Release : 68.el9 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: x86_64 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:Group : Unspecified 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:Size : 8264254 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:License : GPLv3+ 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:Packager : builder@centos.org 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:Vendor : CentOS 2024-03-11T19:33:56.843 INFO:teuthology.orchestra.run.smithi097.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T19:33:56.844 INFO:teuthology.orchestra.run.smithi097.stdout:Summary : Support tools for GRUB. 2024-03-11T19:33:56.844 INFO:teuthology.orchestra.run.smithi097.stdout:Description : 2024-03-11T19:33:56.844 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:33:56.844 INFO:teuthology.orchestra.run.smithi097.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T19:33:56.844 INFO:teuthology.orchestra.run.smithi097.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T19:33:56.844 INFO:teuthology.orchestra.run.smithi097.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T19:33:56.844 INFO:teuthology.orchestra.run.smithi097.stdout:hardware devices. 2024-03-11T19:33:56.844 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:33:56.844 INFO:teuthology.orchestra.run.smithi097.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T19:33:56.845 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T19:33:56.845 INFO:teuthology.task.kernel:Updating grub on smithi097 to boot 6.8.0-rc7-gdd5c1cd17167 2024-03-11T19:33:56.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T19:33:57.415 INFO:teuthology.orchestra.run.smithi097.stderr:Generating grub configuration file ... 2024-03-11T19:33:58.787 INFO:teuthology.orchestra.run.smithi097.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T19:33:58.826 INFO:teuthology.orchestra.run.smithi097.stderr:done 2024-03-11T19:33:58.828 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-03-11T19:33:58.843 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.uXqjMGfvab 2024-03-11T19:33:58.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uXqjMGfvab 2024-03-11T19:33:58.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.uXqjMGfvab 2024-03-11T19:33:59.050 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.uXqjMGfvab is 6KB 2024-03-11T19:33:59.099 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.uXqjMGfvab 2024-03-11T19:33:59.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T19:33:59.178 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T19:33:59.179 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T19:33:59.179 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T19:33:59.179 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc7-gdd5c1cd17167.conf 2024-03-11T19:33:59.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc7-gdd5c1cd17167 2024-03-11T19:33:59.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo shutdown -r now 2024-03-11T19:34:29.376 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-11T19:34:29.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-11T19:34:29.377 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T19:34:47.842 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-03-11T19:34:56.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-11T19:34:56.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T19:34:59.902 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-03-11T19:35:11.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-11T19:35:11.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T19:35:43.818 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-03-11T19:35:44.474 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-11T19:35:44.474 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc7-gdd5c1cd17167"... 2024-03-11T19:35:44.474 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-03-11T19:35:44.512 INFO:teuthology.orchestra.run.smithi097.stdout:6.8.0-rc7-gdd5c1cd17167 2024-03-11T19:35:44.513 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc7-gdd5c1cd17167 vs 6.8.0-rc7-gdd5c1cd17167 2024-03-11T19:35:44.513 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T19:35:44.513 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T19:35:45.514 DEBUG:teuthology.parallel:result is None 2024-03-11T19:35:45.514 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T19:35:45.521 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T19:35:45.521 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T19:35:45.529 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T19:35:45.531 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T19:35:45.572 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T19:35:45.580 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T19:35:45.586 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T19:35:45.587 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T19:35:45.590 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T19:35:45.593 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T19:35:45.656 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T19:35:45.662 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T19:35:45.663 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T19:35:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T19:35:45.680 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T19:35:45.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T19:35:45.698 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T19:35:45.706 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T19:35:45.723 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T19:35:45.757 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T19:35:45.778 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T19:35:45.806 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T19:35:45.831 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T19:35:45.836 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T19:35:45.930 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T19:35:45.934 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T19:35:45.942 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T19:35:45.942 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T19:35:45.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T19:35:45.949 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T19:35:46.017 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T19:35:46.026 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T19:35:46.027 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T19:35:46.031 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T19:35:46.034 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T19:35:46.066 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T19:35:46.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T19:35:46.260 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T19:35:46.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T19:35:46.436 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:35:46.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T19:35:46.544 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T19:35:46.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T19:35:46.648 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T19:35:46.692 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T19:35:46.736 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-11T19:35:46.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T19:35:46.805 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T19:35:46.845 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T19:35:47.204 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T19:35:47.249 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T19:35:47.295 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-11T19:35:47.295 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T19:35:47.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2024-03-11T19:35:47.370 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-03-11T19:35:47.373 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2024-03-11T19:35:47.419 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T19:35:47.448 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T19:35:47.486 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T19:35:47.770 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T19:35:47.778 INFO:teuthology.task.internal:Starting timer... 2024-03-11T19:35:47.778 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T19:35:47.788 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T19:35:47.799 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T19:35:47.799 DEBUG:teuthology.orchestra.run.smithi097:> sudo service auditd rotate 2024-03-11T19:35:47.900 INFO:teuthology.orchestra.run.smithi097.stdout:Rotating logs: 2024-03-11T19:35:47.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-03-11T19:35:47.971 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-03-11T19:35:47.973 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2024-03-11T19:35:48.036 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: 2024-03-11T19:35:48.038 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T19:35:48.038 DEBUG:teuthology.orchestra.run.smithi097:> /usr/sbin/getenforce 2024-03-11T19:35:48.098 INFO:teuthology.orchestra.run.smithi097.stdout:Permissive 2024-03-11T19:35:48.098 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-03-11T19:35:48.132 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-03-11T19:35:48.132 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2024-03-11T19:35:48.155 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2024-03-11T19:35:48.156 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi097.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi161.front.sepia.ceph.com': 'permissive'} 2024-03-11T19:35:48.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T19:35:48.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T19:35:48.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T19:35:48.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T19:35:48.257 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T19:35:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T19:35:48.286 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T19:35:48.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo /usr/sbin/setenforce permissive 2024-03-11T19:35:48.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-03-11T19:35:48.426 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2024-03-11T19:35:48.455 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T19:35:48.465 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T19:35:48.482 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T19:35:48.483 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi097.front.sepia.ceph.com,smithi114.front.sepia.ceph.com,smithi161.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T19:41:39.765 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi097.front.sepia.ceph.com'), Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi161.front.sepia.ceph.com')] 2024-03-11T19:41:39.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-11T19:41:39.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T19:41:39.976 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-03-11T19:41:40.136 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-11T19:41:40.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-11T19:41:40.137 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T19:41:40.244 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-03-11T19:41:40.315 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-11T19:41:40.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-11T19:41:40.316 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T19:41:40.410 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-03-11T19:41:40.481 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-11T19:41:40.481 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T19:41:40.748 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T19:41:40.749 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T19:41:40.749 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T19:41:40.759 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T19:41:40.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T19:41:40.765 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T19:41:40.765 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T19:41:40.798 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T19:41:40.805 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T19:41:40.817 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T19:41:40.822 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T19:41:40.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-03-11T19:41:40.853 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T19:41:40.854 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: ntpd: command not found 2024-03-11T19:41:40.862 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-03-11T19:41:40.868 INFO:teuthology.orchestra.run.smithi161.stdout:506 Cannot talk to daemon 2024-03-11T19:41:40.879 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T19:41:40.884 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T19:41:40.896 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T19:41:40.901 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T19:41:40.905 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T19:41:40.940 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-03-11T19:41:40.943 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T19:41:40.943 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-03-11T19:41:40.943 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:41:40.943 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:41:40.943 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:41:40.943 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:41:40.945 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-03-11T19:41:40.949 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T19:41:40.949 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-03-11T19:41:40.949 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:41:40.949 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:41:40.950 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:41:40.950 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:41:40.998 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: ntpd: command not found 2024-03-11T19:41:41.040 INFO:teuthology.orchestra.run.smithi097.stdout:506 Cannot talk to daemon 2024-03-11T19:41:41.091 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T19:41:41.142 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T19:41:41.268 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-03-11T19:41:41.275 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T19:41:41.275 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-03-11T19:41:41.276 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:41:41.276 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:41:41.276 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:41:41.276 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T19:41:41.277 INFO:teuthology.run_tasks:Running task install... 2024-03-11T19:41:41.288 DEBUG:teuthology.task.install:project ceph 2024-03-11T19:41:41.288 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'}} 2024-03-11T19:41:41.288 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T19:41:41.288 INFO:teuthology.task.install:Using flavor: default 2024-03-11T19:41:41.295 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-11T19:41:41.295 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-11T19:41:41.295 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-11T19:41:41.296 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T19:41:41.298 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-11T19:41:41.298 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T19:41:41.299 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-11T19:41:41.299 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T19:41:41.444 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-11T19:41:41.444 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-11T19:41:41.454 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-11T19:41:41.454 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-11T19:41:41.460 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-11T19:41:41.460 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-11T19:41:41.563 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T19:41:41.563 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:41:41.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T19:41:41.580 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T19:41:41.580 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-11T19:41:41.580 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T19:41:41.582 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T19:41:41.582 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-11T19:41:41.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T19:41:41.611 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-11T19:41:41.612 DEBUG:teuthology.orchestra.run.smithi161:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T19:41:41.613 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-11T19:41:41.613 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T19:41:41.658 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-11T19:41:41.659 DEBUG:teuthology.orchestra.run.smithi097:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T19:41:41.680 DEBUG:teuthology.orchestra.run.smithi161:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T19:41:41.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T19:41:41.761 DEBUG:teuthology.orchestra.run.smithi161:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T19:41:41.762 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T19:41:41.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T19:41:41.827 INFO:teuthology.orchestra.run.smithi161.stdout:check_obsoletes = 1 2024-03-11T19:41:41.828 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum clean all 2024-03-11T19:41:41.829 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-03-11T19:41:41.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-03-11T19:41:41.916 DEBUG:teuthology.orchestra.run.smithi097:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T19:41:42.002 INFO:teuthology.orchestra.run.smithi097.stdout:check_obsoletes = 1 2024-03-11T19:41:42.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum clean all 2024-03-11T19:41:42.063 INFO:teuthology.orchestra.run.smithi161.stdout:74 files removed 2024-03-11T19:41:42.064 INFO:teuthology.orchestra.run.smithi114.stdout:74 files removed 2024-03-11T19:41:42.089 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-11T19:41:42.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-11T19:41:42.361 INFO:teuthology.orchestra.run.smithi097.stdout:74 files removed 2024-03-11T19:41:42.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-11T19:41:42.791 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 201 kB/s | 82 kB 00:00 2024-03-11T19:41:42.812 INFO:teuthology.orchestra.run.smithi161.stdout:ceph packages for x86_64 190 kB/s | 82 kB 00:00 2024-03-11T19:41:43.146 INFO:teuthology.orchestra.run.smithi161.stdout:ceph noarch packages 34 kB/s | 10 kB 00:00 2024-03-11T19:41:43.158 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-11T19:41:43.392 INFO:teuthology.orchestra.run.smithi097.stdout:ceph packages for x86_64 169 kB/s | 82 kB 00:00 2024-03-11T19:41:43.517 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-03-11T19:41:43.555 INFO:teuthology.orchestra.run.smithi161.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-11T19:41:43.892 INFO:teuthology.orchestra.run.smithi097.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-11T19:41:43.961 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-11T19:41:43.999 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-11T19:41:44.385 INFO:teuthology.orchestra.run.smithi097.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-03-11T19:41:45.037 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-11T19:41:45.361 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-11T19:41:45.558 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-11T19:41:46.855 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-11T19:41:47.940 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-03-11T19:41:47.993 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-11T19:41:50.552 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.0 MB 00:04 2024-03-11T19:41:52.672 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-11T19:41:57.970 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.1 MB 00:01 2024-03-11T19:41:59.346 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 15 kB 00:00 2024-03-11T19:42:01.172 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 19 MB 00:12 2024-03-11T19:42:04.067 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 4.5 MB/s | 21 MB 00:04 2024-03-11T19:42:05.773 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 6.1 MB 00:01 2024-03-11T19:42:07.666 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 15 kB 00:00 2024-03-11T19:42:08.684 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 991 kB/s | 19 MB 00:19 2024-03-11T19:42:10.000 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-11T19:42:10.000 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-11T19:42:11.545 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T19:42:11.546 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T19:42:11.560 INFO:teuthology.orchestra.run.smithi114.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-11T19:42:11.562 INFO:teuthology.orchestra.run.smithi114.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-11T19:42:11.563 INFO:teuthology.orchestra.run.smithi114.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-11T19:42:11.564 INFO:teuthology.orchestra.run.smithi114.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-11T19:42:11.566 INFO:teuthology.orchestra.run.smithi114.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-11T19:42:11.569 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-11T19:42:11.570 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-11T19:42:11.649 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-11T19:42:11.665 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-11T19:42:11.665 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-11T19:42:11.666 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-11T19:42:11.666 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-11T19:42:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-11T19:42:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-11T19:42:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-11T19:42:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-11T19:42:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-11T19:42:11.666 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-11T19:42:11.667 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-11T19:42:11.668 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-11T19:42:11.669 INFO:teuthology.orchestra.run.smithi114.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-11T19:42:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-11T19:42:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-11T19:42:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-11T19:42:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-11T19:42:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T19:42:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-11T19:42:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-11T19:42:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-11T19:42:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-11T19:42:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-11T19:42:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-11T19:42:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-11T19:42:11.670 INFO:teuthology.orchestra.run.smithi114.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-11T19:42:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-11T19:42:11.671 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-11T19:42:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T19:42:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-11T19:42:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T19:42:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-11T19:42:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-11T19:42:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-11T19:42:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-11T19:42:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-11T19:42:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-11T19:42:11.671 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-11T19:42:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-11T19:42:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T19:42:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T19:42:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T19:42:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T19:42:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T19:42:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T19:42:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T19:42:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T19:42:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T19:42:11.672 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-11T19:42:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T19:42:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T19:42:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T19:42:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T19:42:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T19:42:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-11T19:42:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T19:42:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-11T19:42:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T19:42:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-11T19:42:11.673 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T19:42:11.674 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T19:42:11.675 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T19:42:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T19:42:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T19:42:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T19:42:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T19:42:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T19:42:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T19:42:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T19:42:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T19:42:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T19:42:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T19:42:11.676 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T19:42:11.677 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T19:42:11.677 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T19:42:11.677 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T19:42:11.677 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T19:42:11.677 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T19:42:11.677 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T19:42:11.677 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T19:42:11.677 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T19:42:11.677 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T19:42:11.677 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T19:42:11.677 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T19:42:11.678 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T19:42:11.679 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T19:42:11.679 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T19:42:11.679 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T19:42:11.679 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-11T19:42:11.679 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T19:42:11.679 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T19:42:11.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T19:42:11.680 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T19:42:11.680 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T19:42:11.680 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T19:42:11.680 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T19:42:11.681 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T19:42:11.681 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T19:42:11.681 INFO:teuthology.orchestra.run.smithi114.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-11T19:42:11.681 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T19:42:11.681 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-11T19:42:11.681 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-03-11T19:42:11.681 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T19:42:11.681 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:42:11.681 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-11T19:42:11.681 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-11T19:42:11.681 INFO:teuthology.orchestra.run.smithi114.stdout:Install 150 Packages 2024-03-11T19:42:11.681 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 24 Packages 2024-03-11T19:42:11.682 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:42:11.682 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 223 M 2024-03-11T19:42:11.682 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-11T19:42:12.573 INFO:teuthology.orchestra.run.smithi114.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-03-11T19:42:12.832 INFO:teuthology.orchestra.run.smithi114.stdout:(2/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 3.3 MB/s | 886 kB 00:00 2024-03-11T19:42:12.900 INFO:teuthology.orchestra.run.smithi114.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 2.1 MB/s | 145 kB 00:00 2024-03-11T19:42:13.208 INFO:teuthology.orchestra.run.smithi114.stdout:(4/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 6.9 MB/s | 2.1 MB 00:00 2024-03-11T19:42:13.383 INFO:teuthology.orchestra.run.smithi114.stdout:(5/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 8.5 MB/s | 1.5 MB 00:00 2024-03-11T19:42:13.584 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.1 MB 00:01 2024-03-11T19:42:13.892 INFO:teuthology.orchestra.run.smithi114.stdout:(6/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 9.2 MB/s | 4.7 MB 00:00 2024-03-11T19:42:14.258 INFO:teuthology.orchestra.run.smithi114.stdout:(7/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 2.9 MB/s | 5.3 MB 00:01 2024-03-11T19:42:14.458 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-11T19:42:14.601 INFO:teuthology.orchestra.run.smithi114.stdout:(8/174): ceph-common-19.0.0-1607.g94a53970.el9. 9.0 MB/s | 20 MB 00:02 2024-03-11T19:42:14.668 INFO:teuthology.orchestra.run.smithi114.stdout:(9/174): ceph-selinux-19.0.0-1607.g94a53970.el9 379 kB/s | 25 kB 00:00 2024-03-11T19:42:15.269 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 15 kB 00:00 2024-03-11T19:42:15.334 INFO:teuthology.orchestra.run.smithi114.stdout:(10/174): ceph-osd-19.0.0-1607.g94a53970.el9.x8 12 MB/s | 17 MB 00:01 2024-03-11T19:42:15.393 INFO:teuthology.orchestra.run.smithi114.stdout:(11/174): cephfs-mirror-19.0.0-1607.g94a53970.e 3.7 MB/s | 217 kB 00:00 2024-03-11T19:42:15.451 INFO:teuthology.orchestra.run.smithi114.stdout:(12/174): cephfs-shell-19.0.0-1607.g94a53970.el 464 kB/s | 27 kB 00:00 2024-03-11T19:42:15.510 INFO:teuthology.orchestra.run.smithi114.stdout:(13/174): libcephfs-devel-19.0.0-1607.g94a53970 534 kB/s | 31 kB 00:00 2024-03-11T19:42:15.610 INFO:teuthology.orchestra.run.smithi114.stdout:(14/174): libcephfs2-19.0.0-1607.g94a53970.el9. 7.3 MB/s | 742 kB 00:00 2024-03-11T19:42:15.669 INFO:teuthology.orchestra.run.smithi114.stdout:(15/174): libcephsqlite-19.0.0-1607.g94a53970.e 2.7 MB/s | 163 kB 00:00 2024-03-11T19:42:15.736 INFO:teuthology.orchestra.run.smithi114.stdout:(16/174): librados-devel-19.0.0-1607.g94a53970. 1.9 MB/s | 127 kB 00:00 2024-03-11T19:42:15.820 INFO:teuthology.orchestra.run.smithi114.stdout:(17/174): libradosstriper1-19.0.0-1607.g94a5397 5.8 MB/s | 498 kB 00:00 2024-03-11T19:42:16.178 INFO:teuthology.orchestra.run.smithi114.stdout:(18/174): ceph-radosgw-19.0.0-1607.g94a53970.el 5.1 MB/s | 9.8 MB 00:01 2024-03-11T19:42:16.320 INFO:teuthology.orchestra.run.smithi114.stdout:(19/174): librgw2-19.0.0-1607.g94a53970.el9.x86 10 MB/s | 5.1 MB 00:00 2024-03-11T19:42:16.345 INFO:teuthology.orchestra.run.smithi114.stdout:(20/174): python3-ceph-argparse-19.0.0-1607.g94 270 kB/s | 45 kB 00:00 2024-03-11T19:42:16.387 INFO:teuthology.orchestra.run.smithi114.stdout:(21/174): python3-ceph-common-19.0.0-1607.g94a5 1.9 MB/s | 131 kB 00:00 2024-03-11T19:42:16.438 INFO:teuthology.orchestra.run.smithi114.stdout:(22/174): python3-cephfs-19.0.0-1607.g94a53970. 1.7 MB/s | 161 kB 00:00 2024-03-11T19:42:16.463 INFO:teuthology.orchestra.run.smithi114.stdout:(23/174): python3-rados-19.0.0-1607.g94a53970.e 4.2 MB/s | 320 kB 00:00 2024-03-11T19:42:16.530 INFO:teuthology.orchestra.run.smithi114.stdout:(24/174): python3-rbd-19.0.0-1607.g94a53970.el9 3.2 MB/s | 298 kB 00:00 2024-03-11T19:42:16.555 INFO:teuthology.orchestra.run.smithi114.stdout:(25/174): python3-rgw-19.0.0-1607.g94a53970.el9 1.1 MB/s | 99 kB 00:00 2024-03-11T19:42:16.630 INFO:teuthology.orchestra.run.smithi114.stdout:(26/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 853 kB/s | 85 kB 00:00 2024-03-11T19:42:16.773 INFO:teuthology.orchestra.run.smithi114.stdout:(27/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 1.2 MB/s | 171 kB 00:00 2024-03-11T19:42:16.873 INFO:teuthology.orchestra.run.smithi114.stdout:(28/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 9.7 MB/s | 3.1 MB 00:00 2024-03-11T19:42:16.889 INFO:teuthology.orchestra.run.smithi114.stdout:(29/174): ceph-grafana-dashboards-19.0.0-1607.g 269 kB/s | 31 kB 00:00 2024-03-11T19:42:16.905 INFO:teuthology.orchestra.run.smithi161.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T19:42:16.906 INFO:teuthology.orchestra.run.smithi161.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T19:42:16.920 INFO:teuthology.orchestra.run.smithi161.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-11T19:42:16.921 INFO:teuthology.orchestra.run.smithi161.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-11T19:42:16.922 INFO:teuthology.orchestra.run.smithi161.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-11T19:42:16.923 INFO:teuthology.orchestra.run.smithi161.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-11T19:42:16.925 INFO:teuthology.orchestra.run.smithi161.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-11T19:42:16.928 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-11T19:42:16.929 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-11T19:42:16.931 INFO:teuthology.orchestra.run.smithi114.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 2.3 MB/s | 138 kB 00:00 2024-03-11T19:42:17.009 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-11T19:42:17.021 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-03-11T19:42:17.021 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-11T19:42:17.021 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-03-11T19:42:17.021 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-11T19:42:17.021 INFO:teuthology.orchestra.run.smithi161.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-11T19:42:17.021 INFO:teuthology.orchestra.run.smithi161.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-11T19:42:17.022 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-11T19:42:17.022 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-11T19:42:17.022 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-11T19:42:17.022 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-11T19:42:17.022 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-11T19:42:17.022 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-11T19:42:17.022 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-11T19:42:17.022 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-11T19:42:17.022 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-11T19:42:17.022 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-11T19:42:17.022 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-11T19:42:17.023 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-11T19:42:17.023 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-11T19:42:17.023 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-11T19:42:17.023 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-11T19:42:17.023 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-11T19:42:17.023 INFO:teuthology.orchestra.run.smithi161.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-11T19:42:17.023 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-11T19:42:17.023 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-11T19:42:17.023 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-11T19:42:17.023 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-11T19:42:17.023 INFO:teuthology.orchestra.run.smithi161.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-11T19:42:17.023 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-11T19:42:17.023 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrading: 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-11T19:42:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T19:42:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-11T19:42:17.026 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T19:42:17.027 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T19:42:17.028 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-11T19:42:17.029 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T19:42:17.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T19:42:17.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T19:42:17.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T19:42:17.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T19:42:17.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T19:42:17.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T19:42:17.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T19:42:17.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T19:42:17.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T19:42:17.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T19:42:17.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-11T19:42:17.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T19:42:17.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T19:42:17.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T19:42:17.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T19:42:17.033 INFO:teuthology.orchestra.run.smithi161.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T19:42:17.034 INFO:teuthology.orchestra.run.smithi161.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T19:42:17.034 INFO:teuthology.orchestra.run.smithi161.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T19:42:17.034 INFO:teuthology.orchestra.run.smithi161.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-11T19:42:17.034 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T19:42:17.034 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-11T19:42:17.034 INFO:teuthology.orchestra.run.smithi161.stdout:Installing weak dependencies: 2024-03-11T19:42:17.034 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T19:42:17.034 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:42:17.034 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-11T19:42:17.034 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-03-11T19:42:17.034 INFO:teuthology.orchestra.run.smithi161.stdout:Install 150 Packages 2024-03-11T19:42:17.034 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade 24 Packages 2024-03-11T19:42:17.034 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:42:17.035 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 223 M 2024-03-11T19:42:17.035 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-11T19:42:17.349 INFO:teuthology.orchestra.run.smithi114.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 3.9 MB/s | 1.8 MB 00:00 2024-03-11T19:42:17.466 INFO:teuthology.orchestra.run.smithi114.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1607.g94 2.1 MB/s | 247 kB 00:00 2024-03-11T19:42:17.566 INFO:teuthology.orchestra.run.smithi114.stdout:(33/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 494 kB/s | 49 kB 00:00 2024-03-11T19:42:17.593 INFO:teuthology.orchestra.run.smithi161.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-03-11T19:42:17.658 INFO:teuthology.orchestra.run.smithi114.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1607.g9 167 kB/s | 15 kB 00:00 2024-03-11T19:42:17.935 INFO:teuthology.orchestra.run.smithi161.stdout:(2/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 2.5 MB/s | 886 kB 00:00 2024-03-11T19:42:18.002 INFO:teuthology.orchestra.run.smithi161.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 2.1 MB/s | 145 kB 00:00 2024-03-11T19:42:18.092 INFO:teuthology.orchestra.run.smithi114.stdout:(35/174): ceph-test-19.0.0-1607.g94a53970.el9.x 14 MB/s | 47 MB 00:03 2024-03-11T19:42:18.183 INFO:teuthology.orchestra.run.smithi114.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 5.9 MB/s | 7.4 MB 00:01 2024-03-11T19:42:18.209 INFO:teuthology.orchestra.run.smithi114.stdout:(37/174): ceph-volume-19.0.0-1607.g94a53970.el9 479 kB/s | 264 kB 00:00 2024-03-11T19:42:18.234 INFO:teuthology.orchestra.run.smithi114.stdout:(38/174): cephfs-top-19.0.0-1607.g94a53970.el9. 364 kB/s | 18 kB 00:00 2024-03-11T19:42:18.267 INFO:teuthology.orchestra.run.smithi114.stdout:(39/174): cephadm-19.0.0-1607.g94a53970.el9.noa 4.2 MB/s | 748 kB 00:00 2024-03-11T19:42:18.318 INFO:teuthology.orchestra.run.smithi161.stdout:(4/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 6.7 MB/s | 2.1 MB 00:00 2024-03-11T19:42:18.443 INFO:teuthology.orchestra.run.smithi114.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 855 kB/s | 106 kB 00:00 2024-03-11T19:42:18.476 INFO:teuthology.orchestra.run.smithi114.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-11T19:42:18.493 INFO:teuthology.orchestra.run.smithi161.stdout:(5/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 4.6 MB/s | 5.3 MB 00:01 2024-03-11T19:42:18.501 INFO:teuthology.orchestra.run.smithi114.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 920 kB 00:00 2024-03-11T19:42:18.527 INFO:teuthology.orchestra.run.smithi114.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 307 kB 00:00 2024-03-11T19:42:18.535 INFO:teuthology.orchestra.run.smithi161.stdout:(6/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 6.9 MB/s | 1.5 MB 00:00 2024-03-11T19:42:18.552 INFO:teuthology.orchestra.run.smithi114.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 424 kB/s | 32 kB 00:00 2024-03-11T19:42:18.594 INFO:teuthology.orchestra.run.smithi114.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 257 kB/s | 17 kB 00:00 2024-03-11T19:42:18.644 INFO:teuthology.orchestra.run.smithi114.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-11T19:42:18.669 INFO:teuthology.orchestra.run.smithi114.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 86 kB/s | 10 kB 00:00 2024-03-11T19:42:18.695 INFO:teuthology.orchestra.run.smithi114.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-03-11T19:42:18.720 INFO:teuthology.orchestra.run.smithi114.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-11T19:42:18.753 INFO:teuthology.orchestra.run.smithi114.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-11T19:42:18.779 INFO:teuthology.orchestra.run.smithi114.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 578 kB 00:00 2024-03-11T19:42:18.804 INFO:teuthology.orchestra.run.smithi114.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 941 kB/s | 46 kB 00:00 2024-03-11T19:42:18.829 INFO:teuthology.orchestra.run.smithi114.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-11T19:42:18.855 INFO:teuthology.orchestra.run.smithi114.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-11T19:42:18.880 INFO:teuthology.orchestra.run.smithi114.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-11T19:42:18.896 INFO:teuthology.orchestra.run.smithi114.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 514 kB/s | 34 kB 00:00 2024-03-11T19:42:18.953 INFO:teuthology.orchestra.run.smithi161.stdout:(7/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 10 MB/s | 4.7 MB 00:00 2024-03-11T19:42:18.972 INFO:teuthology.orchestra.run.smithi114.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 41 MB/s | 4.7 MB 00:00 2024-03-11T19:42:19.014 INFO:teuthology.orchestra.run.smithi114.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 13 MB/s | 1.5 MB 00:00 2024-03-11T19:42:19.039 INFO:teuthology.orchestra.run.smithi114.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 5.7 MB/s | 811 kB 00:00 2024-03-11T19:42:19.064 INFO:teuthology.orchestra.run.smithi114.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2024-03-11T19:42:19.115 INFO:teuthology.orchestra.run.smithi114.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-03-11T19:42:19.190 INFO:teuthology.orchestra.run.smithi114.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-03-11T19:42:19.215 INFO:teuthology.orchestra.run.smithi114.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 526 kB/s | 78 kB 00:00 2024-03-11T19:42:19.316 INFO:teuthology.orchestra.run.smithi114.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-11T19:42:19.341 INFO:teuthology.orchestra.run.smithi114.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 199 kB/s | 24 kB 00:00 2024-03-11T19:42:19.366 INFO:teuthology.orchestra.run.smithi114.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 122 kB/s | 18 kB 00:00 2024-03-11T19:42:19.416 INFO:teuthology.orchestra.run.smithi114.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-11T19:42:19.441 INFO:teuthology.orchestra.run.smithi114.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 423 kB/s | 32 kB 00:00 2024-03-11T19:42:19.575 INFO:teuthology.orchestra.run.smithi114.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-03-11T19:42:19.601 INFO:teuthology.orchestra.run.smithi114.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-11T19:42:19.626 INFO:teuthology.orchestra.run.smithi114.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 965 kB/s | 174 kB 00:00 2024-03-11T19:42:19.650 INFO:teuthology.orchestra.run.smithi114.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 923 kB/s | 69 kB 00:00 2024-03-11T19:42:19.676 INFO:teuthology.orchestra.run.smithi114.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 827 kB/s | 41 kB 00:00 2024-03-11T19:42:19.709 INFO:teuthology.orchestra.run.smithi114.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 238 kB 00:00 2024-03-11T19:42:19.735 INFO:teuthology.orchestra.run.smithi114.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-11T19:42:19.760 INFO:teuthology.orchestra.run.smithi114.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-11T19:42:19.783 INFO:teuthology.orchestra.run.smithi114.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 646 kB/s | 47 kB 00:00 2024-03-11T19:42:19.808 INFO:teuthology.orchestra.run.smithi114.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 587 kB/s | 43 kB 00:00 2024-03-11T19:42:19.844 INFO:teuthology.orchestra.run.smithi161.stdout:(8/174): ceph-radosgw-19.0.0-1607.g94a53970.el9 11 MB/s | 9.8 MB 00:00 2024-03-11T19:42:19.850 INFO:teuthology.orchestra.run.smithi114.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 580 kB/s | 39 kB 00:00 2024-03-11T19:42:20.011 INFO:teuthology.orchestra.run.smithi161.stdout:(9/174): ceph-osd-19.0.0-1607.g94a53970.el9.x86 11 MB/s | 17 MB 00:01 2024-03-11T19:42:20.036 INFO:teuthology.orchestra.run.smithi161.stdout:(10/174): ceph-selinux-19.0.0-1607.g94a53970.el 131 kB/s | 25 kB 00:00 2024-03-11T19:42:20.068 INFO:teuthology.orchestra.run.smithi114.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 55 MB/s | 16 MB 00:00 2024-03-11T19:42:20.093 INFO:teuthology.orchestra.run.smithi114.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 145 kB/s | 41 kB 00:00 2024-03-11T19:42:20.112 INFO:teuthology.orchestra.run.smithi161.stdout:(11/174): cephfs-mirror-19.0.0-1607.g94a53970.e 2.8 MB/s | 217 kB 00:00 2024-03-11T19:42:20.118 INFO:teuthology.orchestra.run.smithi114.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 300 kB 00:00 2024-03-11T19:42:20.144 INFO:teuthology.orchestra.run.smithi114.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-11T19:42:20.169 INFO:teuthology.orchestra.run.smithi114.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 835 kB/s | 63 kB 00:00 2024-03-11T19:42:20.178 INFO:teuthology.orchestra.run.smithi161.stdout:(12/174): cephfs-shell-19.0.0-1607.g94a53970.el 407 kB/s | 27 kB 00:00 2024-03-11T19:42:20.194 INFO:teuthology.orchestra.run.smithi114.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 584 kB/s | 44 kB 00:00 2024-03-11T19:42:20.219 INFO:teuthology.orchestra.run.smithi114.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-11T19:42:20.253 INFO:teuthology.orchestra.run.smithi114.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-03-11T19:42:20.262 INFO:teuthology.orchestra.run.smithi161.stdout:(13/174): libcephfs-devel-19.0.0-1607.g94a53970 374 kB/s | 31 kB 00:00 2024-03-11T19:42:20.295 INFO:teuthology.orchestra.run.smithi114.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-11T19:42:20.320 INFO:teuthology.orchestra.run.smithi114.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-11T19:42:20.346 INFO:teuthology.orchestra.run.smithi114.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-11T19:42:20.370 INFO:teuthology.orchestra.run.smithi114.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-11T19:42:20.662 INFO:teuthology.orchestra.run.smithi161.stdout:(14/174): libcephfs2-19.0.0-1607.g94a53970.el9. 1.8 MB/s | 742 kB 00:00 2024-03-11T19:42:20.737 INFO:teuthology.orchestra.run.smithi161.stdout:(15/174): libcephsqlite-19.0.0-1607.g94a53970.e 2.1 MB/s | 163 kB 00:00 2024-03-11T19:42:20.804 INFO:teuthology.orchestra.run.smithi161.stdout:(16/174): librados-devel-19.0.0-1607.g94a53970. 1.9 MB/s | 127 kB 00:00 2024-03-11T19:42:20.897 INFO:teuthology.orchestra.run.smithi161.stdout:(17/174): libradosstriper1-19.0.0-1607.g94a5397 5.3 MB/s | 498 kB 00:00 2024-03-11T19:42:20.962 INFO:teuthology.orchestra.run.smithi114.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 304 kB/s | 187 kB 00:00 2024-03-11T19:42:20.988 INFO:teuthology.orchestra.run.smithi114.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 317 kB/s | 195 kB 00:00 2024-03-11T19:42:21.088 INFO:teuthology.orchestra.run.smithi114.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 996 kB/s | 125 kB 00:00 2024-03-11T19:42:21.213 INFO:teuthology.orchestra.run.smithi114.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 926 kB/s | 803 kB 00:00 2024-03-11T19:42:21.255 INFO:teuthology.orchestra.run.smithi114.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 80 kB/s | 21 kB 00:00 2024-03-11T19:42:21.297 INFO:teuthology.orchestra.run.smithi114.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 122 kB/s | 25 kB 00:00 2024-03-11T19:42:21.456 INFO:teuthology.orchestra.run.smithi114.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-11T19:42:21.480 INFO:teuthology.orchestra.run.smithi161.stdout:(18/174): librgw2-19.0.0-1607.g94a53970.el9.x86 8.8 MB/s | 5.1 MB 00:00 2024-03-11T19:42:21.531 INFO:teuthology.orchestra.run.smithi114.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 13 MB/s | 4.1 MB 00:00 2024-03-11T19:42:21.555 INFO:teuthology.orchestra.run.smithi161.stdout:(19/174): python3-ceph-argparse-19.0.0-1607.g94 604 kB/s | 45 kB 00:00 2024-03-11T19:42:21.639 INFO:teuthology.orchestra.run.smithi161.stdout:(20/174): python3-ceph-common-19.0.0-1607.g94a5 1.5 MB/s | 131 kB 00:00 2024-03-11T19:42:21.639 INFO:teuthology.orchestra.run.smithi114.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.2 MB/s | 216 kB 00:00 2024-03-11T19:42:21.665 INFO:teuthology.orchestra.run.smithi114.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 261 kB/s | 35 kB 00:00 2024-03-11T19:42:21.690 INFO:teuthology.orchestra.run.smithi114.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 631 kB/s | 248 kB 00:00 2024-03-11T19:42:21.715 INFO:teuthology.orchestra.run.smithi114.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 590 kB/s | 44 kB 00:00 2024-03-11T19:42:21.722 INFO:teuthology.orchestra.run.smithi161.stdout:(21/174): python3-cephfs-19.0.0-1607.g94a53970. 1.9 MB/s | 161 kB 00:00 2024-03-11T19:42:21.732 INFO:teuthology.orchestra.run.smithi114.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-03-11T19:42:21.814 INFO:teuthology.orchestra.run.smithi161.stdout:(22/174): python3-rados-19.0.0-1607.g94a53970.e 3.4 MB/s | 320 kB 00:00 2024-03-11T19:42:21.841 INFO:teuthology.orchestra.run.smithi114.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 148 kB/s | 22 kB 00:00 2024-03-11T19:42:21.906 INFO:teuthology.orchestra.run.smithi161.stdout:(23/174): python3-rbd-19.0.0-1607.g94a53970.el9 3.2 MB/s | 298 kB 00:00 2024-03-11T19:42:21.982 INFO:teuthology.orchestra.run.smithi161.stdout:(24/174): python3-rgw-19.0.0-1607.g94a53970.el9 1.3 MB/s | 99 kB 00:00 2024-03-11T19:42:22.066 INFO:teuthology.orchestra.run.smithi114.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-03-11T19:42:22.067 INFO:teuthology.orchestra.run.smithi161.stdout:(25/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-11T19:42:22.092 INFO:teuthology.orchestra.run.smithi114.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-03-11T19:42:22.117 INFO:teuthology.orchestra.run.smithi114.stdout:(108/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-03-11T19:42:22.134 INFO:teuthology.orchestra.run.smithi114.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 290 kB/s | 19 kB 00:00 2024-03-11T19:42:22.184 INFO:teuthology.orchestra.run.smithi114.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 394 kB/s | 26 kB 00:00 2024-03-11T19:42:22.201 INFO:teuthology.orchestra.run.smithi114.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 745 kB/s | 49 kB 00:00 2024-03-11T19:42:22.270 INFO:teuthology.orchestra.run.smithi114.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-03-11T19:42:22.285 INFO:teuthology.orchestra.run.smithi114.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 671 kB/s | 67 kB 00:00 2024-03-11T19:42:22.310 INFO:teuthology.orchestra.run.smithi114.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-03-11T19:42:22.352 INFO:teuthology.orchestra.run.smithi114.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-11T19:42:22.377 INFO:teuthology.orchestra.run.smithi114.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 5.8 MB/s | 548 kB 00:00 2024-03-11T19:42:22.402 INFO:teuthology.orchestra.run.smithi114.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 472 kB/s | 43 kB 00:00 2024-03-11T19:42:22.428 INFO:teuthology.orchestra.run.smithi114.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 435 kB/s | 32 kB 00:00 2024-03-11T19:42:22.453 INFO:teuthology.orchestra.run.smithi114.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-03-11T19:42:22.478 INFO:teuthology.orchestra.run.smithi114.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-11T19:42:22.503 INFO:teuthology.orchestra.run.smithi114.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-11T19:42:22.529 INFO:teuthology.orchestra.run.smithi114.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 6.0 MB/s | 303 kB 00:00 2024-03-11T19:42:22.554 INFO:teuthology.orchestra.run.smithi114.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 473 kB/s | 35 kB 00:00 2024-03-11T19:42:22.579 INFO:teuthology.orchestra.run.smithi114.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.7 MB/s | 203 kB 00:00 2024-03-11T19:42:22.649 INFO:teuthology.orchestra.run.smithi161.stdout:(26/174): ceph-test-19.0.0-1607.g94a53970.el9.x 18 MB/s | 47 MB 00:02 2024-03-11T19:42:22.671 INFO:teuthology.orchestra.run.smithi114.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 18 MB/s | 2.0 MB 00:00 2024-03-11T19:42:22.696 INFO:teuthology.orchestra.run.smithi114.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-03-11T19:42:22.716 INFO:teuthology.orchestra.run.smithi161.stdout:(27/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 4.7 MB/s | 3.1 MB 00:00 2024-03-11T19:42:22.721 INFO:teuthology.orchestra.run.smithi114.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 76 kB/s | 11 kB 00:00 2024-03-11T19:42:22.747 INFO:teuthology.orchestra.run.smithi114.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-03-11T19:42:22.749 INFO:teuthology.orchestra.run.smithi161.stdout:(28/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 1.7 MB/s | 171 kB 00:00 2024-03-11T19:42:22.772 INFO:teuthology.orchestra.run.smithi114.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-03-11T19:42:22.783 INFO:teuthology.orchestra.run.smithi161.stdout:(29/174): ceph-grafana-dashboards-19.0.0-1607.g 471 kB/s | 31 kB 00:00 2024-03-11T19:42:22.797 INFO:teuthology.orchestra.run.smithi114.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2024-03-11T19:42:22.808 INFO:teuthology.orchestra.run.smithi161.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 2.3 MB/s | 138 kB 00:00 2024-03-11T19:42:22.822 INFO:teuthology.orchestra.run.smithi114.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 262 kB/s | 20 kB 00:00 2024-03-11T19:42:22.864 INFO:teuthology.orchestra.run.smithi114.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-03-11T19:42:22.890 INFO:teuthology.orchestra.run.smithi114.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 504 kB/s | 46 kB 00:00 2024-03-11T19:42:22.915 INFO:teuthology.orchestra.run.smithi114.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 865 kB/s | 79 kB 00:00 2024-03-11T19:42:22.940 INFO:teuthology.orchestra.run.smithi114.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 771 kB/s | 58 kB 00:00 2024-03-11T19:42:22.967 INFO:teuthology.orchestra.run.smithi114.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.1 MB/s | 272 kB 00:00 2024-03-11T19:42:22.975 INFO:teuthology.orchestra.run.smithi161.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 9.3 MB/s | 1.8 MB 00:00 2024-03-11T19:42:22.993 INFO:teuthology.orchestra.run.smithi114.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 213 kB/s | 16 kB 00:00 2024-03-11T19:42:23.018 INFO:teuthology.orchestra.run.smithi114.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-03-11T19:42:23.043 INFO:teuthology.orchestra.run.smithi114.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 569 kB/s | 28 kB 00:00 2024-03-11T19:42:23.059 INFO:teuthology.orchestra.run.smithi161.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1607.g94 2.9 MB/s | 247 kB 00:00 2024-03-11T19:42:23.068 INFO:teuthology.orchestra.run.smithi114.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 412 kB/s | 31 kB 00:00 2024-03-11T19:42:23.093 INFO:teuthology.orchestra.run.smithi114.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-03-11T19:42:23.119 INFO:teuthology.orchestra.run.smithi114.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-11T19:42:23.142 INFO:teuthology.orchestra.run.smithi161.stdout:(33/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 593 kB/s | 49 kB 00:00 2024-03-11T19:42:23.144 INFO:teuthology.orchestra.run.smithi114.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 477 kB/s | 36 kB 00:00 2024-03-11T19:42:23.169 INFO:teuthology.orchestra.run.smithi114.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-03-11T19:42:23.194 INFO:teuthology.orchestra.run.smithi114.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-11T19:42:23.220 INFO:teuthology.orchestra.run.smithi114.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-11T19:42:23.251 INFO:teuthology.orchestra.run.smithi161.stdout:(34/174): ceph-mgr-diskprediction-local-19.0.0- 17 MB/s | 7.4 MB 00:00 2024-03-11T19:42:23.253 INFO:teuthology.orchestra.run.smithi114.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-11T19:42:23.267 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 2.6 MB/s | 21 MB 00:07 2024-03-11T19:42:23.270 INFO:teuthology.orchestra.run.smithi114.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 404 kB/s | 20 kB 00:00 2024-03-11T19:42:23.276 INFO:teuthology.orchestra.run.smithi161.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1607.g9 115 kB/s | 15 kB 00:00 2024-03-11T19:42:23.295 INFO:teuthology.orchestra.run.smithi114.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-11T19:42:23.302 INFO:teuthology.orchestra.run.smithi161.stdout:(36/174): ceph-volume-19.0.0-1607.g94a53970.el9 5.2 MB/s | 264 kB 00:00 2024-03-11T19:42:23.329 INFO:teuthology.orchestra.run.smithi114.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-03-11T19:42:23.344 INFO:teuthology.orchestra.run.smithi161.stdout:(37/174): cephadm-19.0.0-1607.g94a53970.el9.noa 11 MB/s | 748 kB 00:00 2024-03-11T19:42:23.360 INFO:teuthology.orchestra.run.smithi161.stdout:(38/174): cephfs-top-19.0.0-1607.g94a53970.el9. 312 kB/s | 18 kB 00:00 2024-03-11T19:42:23.430 INFO:teuthology.orchestra.run.smithi114.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.6 MB/s | 321 kB 00:00 2024-03-11T19:42:23.536 INFO:teuthology.orchestra.run.smithi161.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 662 kB 00:00 2024-03-11T19:42:23.569 INFO:teuthology.orchestra.run.smithi161.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.3 MB/s | 920 kB 00:00 2024-03-11T19:42:23.595 INFO:teuthology.orchestra.run.smithi161.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-11T19:42:23.620 INFO:teuthology.orchestra.run.smithi161.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-11T19:42:23.645 INFO:teuthology.orchestra.run.smithi161.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 638 kB/s | 32 kB 00:00 2024-03-11T19:42:23.729 INFO:teuthology.orchestra.run.smithi161.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-11T19:42:23.754 INFO:teuthology.orchestra.run.smithi161.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 158 kB/s | 17 kB 00:00 2024-03-11T19:42:23.779 INFO:teuthology.orchestra.run.smithi161.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 200 kB/s | 10 kB 00:00 2024-03-11T19:42:23.813 INFO:teuthology.orchestra.run.smithi161.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.6 MB/s | 159 kB 00:00 2024-03-11T19:42:23.821 INFO:teuthology.orchestra.run.smithi114.stdout:(152/174): librados2-19.0.0-1607.g94a53970.el9. 6.4 MB/s | 3.4 MB 00:00 2024-03-11T19:42:23.838 INFO:teuthology.orchestra.run.smithi161.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 752 kB/s | 44 kB 00:00 2024-03-11T19:42:23.864 INFO:teuthology.orchestra.run.smithi161.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-11T19:42:23.871 INFO:teuthology.orchestra.run.smithi114.stdout:(153/174): librbd1-19.0.0-1607.g94a53970.el9.x8 5.5 MB/s | 3.1 MB 00:00 2024-03-11T19:42:23.889 INFO:teuthology.orchestra.run.smithi161.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-11T19:42:23.914 INFO:teuthology.orchestra.run.smithi161.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 940 kB/s | 46 kB 00:00 2024-03-11T19:42:23.939 INFO:teuthology.orchestra.run.smithi161.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-11T19:42:23.965 INFO:teuthology.orchestra.run.smithi161.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-11T19:42:23.990 INFO:teuthology.orchestra.run.smithi161.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-11T19:42:24.015 INFO:teuthology.orchestra.run.smithi161.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2024-03-11T19:42:24.207 INFO:teuthology.orchestra.run.smithi161.stdout:(56/174): ceph-common-19.0.0-1607.g94a53970.el9 2.9 MB/s | 20 MB 00:06 2024-03-11T19:42:24.222 INFO:teuthology.orchestra.run.smithi114.stdout:(154/174): NetworkManager-1.46.0-1.el9.x86_64.r 2.9 MB/s | 2.3 MB 00:00 2024-03-11T19:42:24.247 INFO:teuthology.orchestra.run.smithi114.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 113 kB/s | 42 kB 00:00 2024-03-11T19:42:24.274 INFO:teuthology.orchestra.run.smithi161.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-03-11T19:42:24.316 INFO:teuthology.orchestra.run.smithi161.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 5.1 MB/s | 1.5 MB 00:00 2024-03-11T19:42:24.331 INFO:teuthology.orchestra.run.smithi114.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-03-11T19:42:24.350 INFO:teuthology.orchestra.run.smithi161.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 5.7 MB/s | 811 kB 00:00 2024-03-11T19:42:24.375 INFO:teuthology.orchestra.run.smithi161.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-03-11T19:42:24.408 INFO:teuthology.orchestra.run.smithi161.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-11T19:42:24.423 INFO:teuthology.orchestra.run.smithi114.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 407 kB/s | 71 kB 00:00 2024-03-11T19:42:24.448 INFO:teuthology.orchestra.run.smithi114.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 312 kB/s | 36 kB 00:00 2024-03-11T19:42:24.509 INFO:teuthology.orchestra.run.smithi161.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 36 MB/s | 5.7 MB 00:00 2024-03-11T19:42:24.523 INFO:teuthology.orchestra.run.smithi114.stdout:(159/174): NetworkManager-libnm-1.46.0-1.el9.x8 2.6 MB/s | 1.8 MB 00:00 2024-03-11T19:42:24.542 INFO:teuthology.orchestra.run.smithi161.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 466 kB/s | 78 kB 00:00 2024-03-11T19:42:24.556 INFO:teuthology.orchestra.run.smithi114.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-11T19:42:24.634 INFO:teuthology.orchestra.run.smithi161.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-03-11T19:42:24.659 INFO:teuthology.orchestra.run.smithi161.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 166 kB/s | 24 kB 00:00 2024-03-11T19:42:24.684 INFO:teuthology.orchestra.run.smithi161.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 130 kB/s | 18 kB 00:00 2024-03-11T19:42:24.699 INFO:teuthology.orchestra.run.smithi114.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 131 kB/s | 23 kB 00:00 2024-03-11T19:42:24.709 INFO:teuthology.orchestra.run.smithi161.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-11T19:42:24.724 INFO:teuthology.orchestra.run.smithi114.stdout:(162/174): elfutils-libelf-0.190-2.el9.x86_64.r 638 kB/s | 192 kB 00:00 2024-03-11T19:42:24.735 INFO:teuthology.orchestra.run.smithi161.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 629 kB/s | 32 kB 00:00 2024-03-11T19:42:24.749 INFO:teuthology.orchestra.run.smithi114.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 563 kB/s | 108 kB 00:00 2024-03-11T19:42:24.800 INFO:teuthology.orchestra.run.smithi114.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-11T19:42:24.810 INFO:teuthology.orchestra.run.smithi161.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-11T19:42:24.836 INFO:teuthology.orchestra.run.smithi161.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-11T19:42:24.850 INFO:teuthology.orchestra.run.smithi114.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-11T19:42:24.860 INFO:teuthology.orchestra.run.smithi161.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2024-03-11T19:42:24.875 INFO:teuthology.orchestra.run.smithi114.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 509 kB/s | 64 kB 00:00 2024-03-11T19:42:24.900 INFO:teuthology.orchestra.run.smithi114.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 286 kB/s | 29 kB 00:00 2024-03-11T19:42:24.902 INFO:teuthology.orchestra.run.smithi161.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 758 kB/s | 69 kB 00:00 2024-03-11T19:42:24.927 INFO:teuthology.orchestra.run.smithi161.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 618 kB/s | 41 kB 00:00 2024-03-11T19:42:24.953 INFO:teuthology.orchestra.run.smithi161.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.5 MB/s | 238 kB 00:00 2024-03-11T19:42:24.978 INFO:teuthology.orchestra.run.smithi161.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-11T19:42:25.003 INFO:teuthology.orchestra.run.smithi161.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 951 kB/s | 47 kB 00:00 2024-03-11T19:42:25.028 INFO:teuthology.orchestra.run.smithi161.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-11T19:42:25.045 INFO:teuthology.orchestra.run.smithi161.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 639 kB/s | 43 kB 00:00 2024-03-11T19:42:25.109 INFO:teuthology.orchestra.run.smithi114.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 9.2 MB/s | 2.2 MB 00:00 2024-03-11T19:42:25.168 INFO:teuthology.orchestra.run.smithi114.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 8.6 MB/s | 2.3 MB 00:00 2024-03-11T19:42:25.276 INFO:teuthology.orchestra.run.smithi114.stdout:(170/174): openssl-3.0.7-27.el9.x86_64.rpm 2.8 MB/s | 1.2 MB 00:00 2024-03-11T19:42:25.301 INFO:teuthology.orchestra.run.smithi114.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 233 kB/s | 31 kB 00:00 2024-03-11T19:42:25.304 INFO:teuthology.orchestra.run.smithi161.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 52 MB/s | 16 MB 00:00 2024-03-11T19:42:25.321 INFO:teuthology.orchestra.run.smithi161.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 142 kB/s | 39 kB 00:00 2024-03-11T19:42:25.337 INFO:teuthology.orchestra.run.smithi161.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 142 kB/s | 41 kB 00:00 2024-03-11T19:42:25.363 INFO:teuthology.orchestra.run.smithi161.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-11T19:42:25.388 INFO:teuthology.orchestra.run.smithi161.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-11T19:42:25.401 INFO:teuthology.orchestra.run.smithi114.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 744 kB/s | 74 kB 00:00 2024-03-11T19:42:25.413 INFO:teuthology.orchestra.run.smithi161.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 839 kB/s | 63 kB 00:00 2024-03-11T19:42:25.438 INFO:teuthology.orchestra.run.smithi161.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-03-11T19:42:25.464 INFO:teuthology.orchestra.run.smithi161.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-11T19:42:25.485 INFO:teuthology.orchestra.run.smithi114.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.1 MB/s | 1.1 MB 00:00 2024-03-11T19:42:25.489 INFO:teuthology.orchestra.run.smithi161.stdout:(87/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-11T19:42:25.522 INFO:teuthology.orchestra.run.smithi161.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-11T19:42:25.548 INFO:teuthology.orchestra.run.smithi161.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-11T19:42:25.573 INFO:teuthology.orchestra.run.smithi161.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-11T19:42:25.598 INFO:teuthology.orchestra.run.smithi161.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-11T19:42:25.602 INFO:teuthology.orchestra.run.smithi114.stdout:(174/174): util-linux-core-2.37.4-18.el9.x86_64 944 kB/s | 465 kB 00:00 2024-03-11T19:42:25.608 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-11T19:42:25.609 INFO:teuthology.orchestra.run.smithi114.stdout:Total 16 MB/s | 223 MB 00:13 2024-03-11T19:42:26.090 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-11T19:42:26.362 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-11T19:42:26.363 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-11T19:42:26.598 INFO:teuthology.orchestra.run.smithi161.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 178 kB/s | 187 kB 00:01 2024-03-11T19:42:26.624 INFO:teuthology.orchestra.run.smithi161.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 191 kB/s | 195 kB 00:01 2024-03-11T19:42:27.490 INFO:teuthology.orchestra.run.smithi161.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 424 kB/s | 803 kB 00:01 2024-03-11T19:42:27.516 INFO:teuthology.orchestra.run.smithi161.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 136 kB/s | 125 kB 00:00 2024-03-11T19:42:28.073 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-11T19:42:28.074 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-11T19:42:29.011 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-03-11T19:42:31.721 INFO:teuthology.orchestra.run.smithi097.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T19:42:31.723 INFO:teuthology.orchestra.run.smithi097.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T19:42:31.746 INFO:teuthology.orchestra.run.smithi097.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-11T19:42:31.748 INFO:teuthology.orchestra.run.smithi097.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-11T19:42:31.749 INFO:teuthology.orchestra.run.smithi097.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-11T19:42:31.751 INFO:teuthology.orchestra.run.smithi097.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-11T19:42:31.754 INFO:teuthology.orchestra.run.smithi097.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-11T19:42:31.758 INFO:teuthology.orchestra.run.smithi097.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-11T19:42:31.759 INFO:teuthology.orchestra.run.smithi097.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-11T19:42:31.846 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T19:42:31.878 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-03-11T19:42:31.878 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-11T19:42:31.878 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-03-11T19:42:31.878 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T19:42:31.878 INFO:teuthology.orchestra.run.smithi097.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-11T19:42:31.878 INFO:teuthology.orchestra.run.smithi097.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-11T19:42:31.878 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-11T19:42:31.879 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-11T19:42:31.880 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-11T19:42:31.881 INFO:teuthology.orchestra.run.smithi097.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-11T19:42:31.882 INFO:teuthology.orchestra.run.smithi097.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-11T19:42:31.882 INFO:teuthology.orchestra.run.smithi097.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-11T19:42:31.882 INFO:teuthology.orchestra.run.smithi097.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-11T19:42:31.882 INFO:teuthology.orchestra.run.smithi097.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-11T19:42:31.882 INFO:teuthology.orchestra.run.smithi097.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-11T19:42:31.882 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T19:42:31.882 INFO:teuthology.orchestra.run.smithi097.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-11T19:42:31.882 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-11T19:42:31.882 INFO:teuthology.orchestra.run.smithi097.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-11T19:42:31.882 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-11T19:42:31.882 INFO:teuthology.orchestra.run.smithi097.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-11T19:42:31.882 INFO:teuthology.orchestra.run.smithi097.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-11T19:42:31.882 INFO:teuthology.orchestra.run.smithi097.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-11T19:42:31.883 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T19:42:31.884 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T19:42:31.885 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T19:42:31.886 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T19:42:31.887 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T19:42:31.888 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T19:42:31.889 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout:Installing weak dependencies: 2024-03-11T19:42:31.890 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T19:42:31.891 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:42:31.891 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T19:42:31.891 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-03-11T19:42:31.891 INFO:teuthology.orchestra.run.smithi097.stdout:Install 150 Packages 2024-03-11T19:42:31.891 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 24 Packages 2024-03-11T19:42:31.891 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:42:31.891 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 223 M 2024-03-11T19:42:31.891 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T19:42:32.596 INFO:teuthology.orchestra.run.smithi097.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-03-11T19:42:32.797 INFO:teuthology.orchestra.run.smithi097.stdout:(2/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 4.3 MB/s | 886 kB 00:00 2024-03-11T19:42:32.863 INFO:teuthology.orchestra.run.smithi097.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 2.2 MB/s | 145 kB 00:00 2024-03-11T19:42:33.080 INFO:teuthology.orchestra.run.smithi097.stdout:(4/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 7.4 MB/s | 5.3 MB 00:00 2024-03-11T19:42:33.164 INFO:teuthology.orchestra.run.smithi097.stdout:(5/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 7.1 MB/s | 2.1 MB 00:00 2024-03-11T19:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout:(6/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 9.0 MB/s | 1.5 MB 00:00 2024-03-11T19:42:33.564 INFO:teuthology.orchestra.run.smithi097.stdout:(7/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 12 MB/s | 4.7 MB 00:00 2024-03-11T19:42:34.556 INFO:teuthology.orchestra.run.smithi097.stdout:(8/174): ceph-osd-19.0.0-1607.g94a53970.el9.x86 13 MB/s | 17 MB 00:01 2024-03-11T19:42:34.606 INFO:teuthology.orchestra.run.smithi097.stdout:(9/174): ceph-selinux-19.0.0-1607.g94a53970.el9 515 kB/s | 25 kB 00:00 2024-03-11T19:42:34.757 INFO:teuthology.orchestra.run.smithi097.stdout:(10/174): ceph-radosgw-19.0.0-1607.g94a53970.el 8.2 MB/s | 9.8 MB 00:01 2024-03-11T19:42:34.832 INFO:teuthology.orchestra.run.smithi097.stdout:(11/174): cephfs-mirror-19.0.0-1607.g94a53970.e 2.9 MB/s | 217 kB 00:00 2024-03-11T19:42:34.890 INFO:teuthology.orchestra.run.smithi097.stdout:(12/174): cephfs-shell-19.0.0-1607.g94a53970.el 469 kB/s | 27 kB 00:00 2024-03-11T19:42:34.923 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-11T19:42:34.949 INFO:teuthology.orchestra.run.smithi097.stdout:(13/174): libcephfs-devel-19.0.0-1607.g94a53970 540 kB/s | 31 kB 00:00 2024-03-11T19:42:35.048 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-11T19:42:35.082 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-11T19:42:35.100 INFO:teuthology.orchestra.run.smithi097.stdout:(14/174): libcephfs2-19.0.0-1607.g94a53970.el9. 4.9 MB/s | 742 kB 00:00 2024-03-11T19:42:35.113 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T19:42:35.122 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T19:42:35.163 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-11T19:42:35.175 INFO:teuthology.orchestra.run.smithi097.stdout:(15/174): libcephsqlite-19.0.0-1607.g94a53970.e 2.2 MB/s | 163 kB 00:00 2024-03-11T19:42:35.216 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-11T19:42:35.250 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-11T19:42:35.258 INFO:teuthology.orchestra.run.smithi097.stdout:(16/174): librados-devel-19.0.0-1607.g94a53970. 1.5 MB/s | 127 kB 00:00 2024-03-11T19:42:35.275 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-11T19:42:35.326 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-11T19:42:35.350 INFO:teuthology.orchestra.run.smithi097.stdout:(17/174): libradosstriper1-19.0.0-1607.g94a5397 5.4 MB/s | 498 kB 00:00 2024-03-11T19:42:35.644 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-11T19:42:35.710 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T19:42:35.822 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T19:42:35.852 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T19:42:35.927 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T19:42:35.951 INFO:teuthology.orchestra.run.smithi097.stdout:(18/174): librgw2-19.0.0-1607.g94a53970.el9.x86 8.5 MB/s | 5.1 MB 00:00 2024-03-11T19:42:35.956 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T19:42:35.992 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-11T19:42:36.009 INFO:teuthology.orchestra.run.smithi097.stdout:(19/174): python3-ceph-argparse-19.0.0-1607.g94 788 kB/s | 45 kB 00:00 2024-03-11T19:42:36.076 INFO:teuthology.orchestra.run.smithi097.stdout:(20/174): python3-ceph-common-19.0.0-1607.g94a5 1.9 MB/s | 131 kB 00:00 2024-03-11T19:42:36.160 INFO:teuthology.orchestra.run.smithi097.stdout:(21/174): python3-cephfs-19.0.0-1607.g94a53970. 1.9 MB/s | 161 kB 00:00 2024-03-11T19:42:36.166 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-11T19:42:36.221 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-11T19:42:36.249 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T19:42:36.252 INFO:teuthology.orchestra.run.smithi097.stdout:(22/174): python3-rados-19.0.0-1607.g94a53970.e 3.5 MB/s | 320 kB 00:00 2024-03-11T19:42:36.352 INFO:teuthology.orchestra.run.smithi097.stdout:(23/174): python3-rbd-19.0.0-1607.g94a53970.el9 2.9 MB/s | 298 kB 00:00 2024-03-11T19:42:36.370 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T19:42:36.392 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T19:42:36.427 INFO:teuthology.orchestra.run.smithi097.stdout:(24/174): python3-rgw-19.0.0-1607.g94a53970.el9 1.3 MB/s | 99 kB 00:00 2024-03-11T19:42:36.456 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T19:42:36.496 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-11T19:42:36.518 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T19:42:36.519 INFO:teuthology.orchestra.run.smithi097.stdout:(25/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 937 kB/s | 85 kB 00:00 2024-03-11T19:42:36.534 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T19:42:36.747 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T19:42:36.775 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T19:42:36.815 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-11T19:42:36.914 INFO:teuthology.orchestra.run.smithi097.stdout:(26/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 7.9 MB/s | 3.1 MB 00:00 2024-03-11T19:42:36.991 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-11T19:42:37.003 INFO:teuthology.orchestra.run.smithi097.stdout:(27/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 1.9 MB/s | 171 kB 00:00 2024-03-11T19:42:37.193 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T19:42:37.253 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T19:42:37.326 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T19:42:37.379 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T19:42:37.496 INFO:teuthology.orchestra.run.smithi097.stdout:(28/174): ceph-test-19.0.0-1607.g94a53970.el9.x 16 MB/s | 47 MB 00:02 2024-03-11T19:42:37.520 INFO:teuthology.orchestra.run.smithi097.stdout:(29/174): ceph-grafana-dashboards-19.0.0-1607.g 61 kB/s | 31 kB 00:00 2024-03-11T19:42:37.554 INFO:teuthology.orchestra.run.smithi097.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 2.4 MB/s | 138 kB 00:00 2024-03-11T19:42:37.593 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T19:42:37.688 INFO:teuthology.orchestra.run.smithi097.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 11 MB/s | 1.8 MB 00:00 2024-03-11T19:42:37.703 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-11T19:42:37.754 INFO:teuthology.orchestra.run.smithi097.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1607.g94 3.7 MB/s | 247 kB 00:00 2024-03-11T19:42:37.802 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-11T19:42:37.830 INFO:teuthology.orchestra.run.smithi097.stdout:(33/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 664 kB/s | 49 kB 00:00 2024-03-11T19:42:37.842 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-11T19:42:37.869 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-11T19:42:37.912 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-11T19:42:37.913 INFO:teuthology.orchestra.run.smithi097.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1607.g9 186 kB/s | 15 kB 00:00 2024-03-11T19:42:37.989 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-11T19:42:38.042 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-11T19:42:38.072 INFO:teuthology.orchestra.run.smithi097.stdout:(35/174): ceph-mgr-diskprediction-local-19.0.0- 14 MB/s | 7.4 MB 00:00 2024-03-11T19:42:38.079 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-11T19:42:38.154 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-11T19:42:38.187 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-11T19:42:38.230 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-11T19:42:38.263 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-11T19:42:38.291 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-11T19:42:38.347 INFO:teuthology.orchestra.run.smithi097.stdout:(36/174): ceph-common-19.0.0-1607.g94a53970.el9 3.3 MB/s | 20 MB 00:05 2024-03-11T19:42:38.362 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-11T19:42:38.381 INFO:teuthology.orchestra.run.smithi097.stdout:(37/174): ceph-volume-19.0.0-1607.g94a53970.el9 565 kB/s | 264 kB 00:00 2024-03-11T19:42:38.396 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-11T19:42:38.423 INFO:teuthology.orchestra.run.smithi097.stdout:(38/174): cephadm-19.0.0-1607.g94a53970.el9.noa 2.1 MB/s | 748 kB 00:00 2024-03-11T19:42:38.439 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-11T19:42:38.448 INFO:teuthology.orchestra.run.smithi097.stdout:(39/174): cephfs-top-19.0.0-1607.g94a53970.el9. 184 kB/s | 18 kB 00:00 2024-03-11T19:42:38.468 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-11T19:42:38.541 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-11T19:42:38.582 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-11T19:42:38.659 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-11T19:42:38.707 INFO:teuthology.orchestra.run.smithi097.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 411 kB/s | 106 kB 00:00 2024-03-11T19:42:38.773 INFO:teuthology.orchestra.run.smithi097.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 1.7 MB/s | 662 kB 00:00 2024-03-11T19:42:38.795 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-11T19:42:38.815 INFO:teuthology.orchestra.run.smithi097.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 2.3 MB/s | 920 kB 00:00 2024-03-11T19:42:38.840 INFO:teuthology.orchestra.run.smithi097.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 307 kB 00:00 2024-03-11T19:42:38.853 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-11T19:42:38.874 INFO:teuthology.orchestra.run.smithi097.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 335 kB/s | 32 kB 00:00 2024-03-11T19:42:38.908 INFO:teuthology.orchestra.run.smithi097.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 261 kB/s | 17 kB 00:00 2024-03-11T19:42:39.000 INFO:teuthology.orchestra.run.smithi097.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-03-11T19:42:39.025 INFO:teuthology.orchestra.run.smithi097.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 72 kB/s | 10 kB 00:00 2024-03-11T19:42:39.059 INFO:teuthology.orchestra.run.smithi097.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-11T19:42:39.109 INFO:teuthology.orchestra.run.smithi097.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 544 kB/s | 44 kB 00:00 2024-03-11T19:42:39.142 INFO:teuthology.orchestra.run.smithi097.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 5.6 MB/s | 661 kB 00:00 2024-03-11T19:42:39.184 INFO:teuthology.orchestra.run.smithi097.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 4.8 MB/s | 578 kB 00:00 2024-03-11T19:42:39.218 INFO:teuthology.orchestra.run.smithi097.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 668 kB/s | 46 kB 00:00 2024-03-11T19:42:39.243 INFO:teuthology.orchestra.run.smithi097.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 282 kB 00:00 2024-03-11T19:42:39.276 INFO:teuthology.orchestra.run.smithi097.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.1 MB/s | 187 kB 00:00 2024-03-11T19:42:39.310 INFO:teuthology.orchestra.run.smithi097.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2024-03-11T19:42:39.335 INFO:teuthology.orchestra.run.smithi097.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 381 kB/s | 34 kB 00:00 2024-03-11T19:42:39.408 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-11T19:42:39.487 INFO:teuthology.orchestra.run.smithi097.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 23 MB/s | 4.7 MB 00:00 2024-03-11T19:42:39.491 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-11T19:42:39.536 INFO:teuthology.orchestra.run.smithi097.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 8.2 MB/s | 1.5 MB 00:00 2024-03-11T19:42:39.571 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-11T19:42:39.578 INFO:teuthology.orchestra.run.smithi097.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 3.5 MB/s | 811 kB 00:00 2024-03-11T19:42:39.617 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T19:42:39.620 INFO:teuthology.orchestra.run.smithi097.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.0 MB/s | 134 kB 00:00 2024-03-11T19:42:39.626 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T19:42:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-11T19:42:39.704 INFO:teuthology.orchestra.run.smithi097.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.2 MB/s | 1.0 MB 00:00 2024-03-11T19:42:39.705 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-11T19:42:39.744 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-11T19:42:39.782 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-11T19:42:39.812 INFO:teuthology.orchestra.run.smithi097.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 25 MB/s | 5.7 MB 00:00 2024-03-11T19:42:39.837 INFO:teuthology.orchestra.run.smithi097.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 382 kB/s | 78 kB 00:00 2024-03-11T19:42:39.979 INFO:teuthology.orchestra.run.smithi097.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-11T19:42:39.996 INFO:teuthology.orchestra.run.smithi097.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 168 kB/s | 24 kB 00:00 2024-03-11T19:42:40.013 INFO:teuthology.orchestra.run.smithi097.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 112 kB/s | 18 kB 00:00 2024-03-11T19:42:40.041 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-11T19:42:40.071 INFO:teuthology.orchestra.run.smithi097.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 144 kB 00:00 2024-03-11T19:42:40.097 INFO:teuthology.orchestra.run.smithi097.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 386 kB/s | 32 kB 00:00 2024-03-11T19:42:40.214 INFO:teuthology.orchestra.run.smithi097.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-11T19:42:40.217 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-11T19:42:40.239 INFO:teuthology.orchestra.run.smithi097.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-11T19:42:40.270 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-11T19:42:40.291 INFO:teuthology.orchestra.run.smithi097.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 967 kB/s | 174 kB 00:00 2024-03-11T19:42:40.314 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-11T19:42:40.339 INFO:teuthology.orchestra.run.smithi097.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.4 MB/s | 69 kB 00:00 2024-03-11T19:42:40.364 INFO:teuthology.orchestra.run.smithi097.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 576 kB/s | 41 kB 00:00 2024-03-11T19:42:40.389 INFO:teuthology.orchestra.run.smithi097.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.4 MB/s | 238 kB 00:00 2024-03-11T19:42:40.431 INFO:teuthology.orchestra.run.smithi097.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.2 MB/s | 132 kB 00:00 2024-03-11T19:42:40.456 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-11T19:42:40.456 INFO:teuthology.orchestra.run.smithi097.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-11T19:42:40.482 INFO:teuthology.orchestra.run.smithi097.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 520 kB/s | 47 kB 00:00 2024-03-11T19:42:40.540 INFO:teuthology.orchestra.run.smithi097.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 751 kB/s | 43 kB 00:00 2024-03-11T19:42:40.574 INFO:teuthology.orchestra.run.smithi097.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 429 kB/s | 39 kB 00:00 2024-03-11T19:42:40.592 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-11T19:42:40.777 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-11T19:42:40.832 INFO:teuthology.orchestra.run.smithi097.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-11T19:42:40.895 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-11T19:42:40.895 INFO:teuthology.orchestra.run.smithi097.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 131 kB/s | 41 kB 00:00 2024-03-11T19:42:40.895 INFO:teuthology.orchestra.run.smithi097.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 976 kB/s | 300 kB 00:00 2024-03-11T19:42:40.916 INFO:teuthology.orchestra.run.smithi097.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2024-03-11T19:42:40.918 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-11T19:42:40.942 INFO:teuthology.orchestra.run.smithi097.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 772 kB/s | 63 kB 00:00 2024-03-11T19:42:40.954 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-11T19:42:40.967 INFO:teuthology.orchestra.run.smithi097.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 600 kB/s | 44 kB 00:00 2024-03-11T19:42:40.992 INFO:teuthology.orchestra.run.smithi097.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-11T19:42:41.004 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-11T19:42:41.017 INFO:teuthology.orchestra.run.smithi097.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.2 MB/s | 241 kB 00:00 2024-03-11T19:42:41.037 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-11T19:42:41.081 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-11T19:42:41.090 INFO:teuthology.orchestra.run.smithi097.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 9.6 MB/s | 1.1 MB 00:00 2024-03-11T19:42:41.123 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-11T19:42:41.126 INFO:teuthology.orchestra.run.smithi097.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-03-11T19:42:41.143 INFO:teuthology.orchestra.run.smithi097.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 124 kB 00:00 2024-03-11T19:42:41.177 INFO:teuthology.orchestra.run.smithi097.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-11T19:42:41.264 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-11T19:42:41.311 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-11T19:42:41.341 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-11T19:42:41.378 INFO:teuthology.orchestra.run.smithi097.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 812 kB/s | 187 kB 00:00 2024-03-11T19:42:41.383 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-11T19:42:41.402 INFO:teuthology.orchestra.run.smithi097.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 869 kB/s | 195 kB 00:00 2024-03-11T19:42:41.480 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-11T19:42:41.501 INFO:teuthology.orchestra.run.smithi097.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-03-11T19:42:41.515 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-11T19:42:41.555 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-11T19:42:41.589 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-11T19:42:41.611 INFO:teuthology.orchestra.run.smithi097.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-11T19:42:41.614 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-11T19:42:41.690 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-11T19:42:41.770 INFO:teuthology.orchestra.run.smithi097.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 59 kB/s | 21 kB 00:00 2024-03-11T19:42:41.803 INFO:teuthology.orchestra.run.smithi097.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 109 kB/s | 25 kB 00:00 2024-03-11T19:42:41.818 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-11T19:42:41.912 INFO:teuthology.orchestra.run.smithi097.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.3 MB/s | 248 kB 00:00 2024-03-11T19:42:41.930 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-11T19:42:41.946 INFO:teuthology.orchestra.run.smithi097.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-11T19:42:41.973 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-11T19:42:41.996 INFO:teuthology.orchestra.run.smithi097.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.6 MB/s | 216 kB 00:00 2024-03-11T19:42:42.021 INFO:teuthology.orchestra.run.smithi097.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 470 kB/s | 35 kB 00:00 2024-03-11T19:42:42.031 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-11T19:42:42.094 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-11T19:42:42.130 INFO:teuthology.orchestra.run.smithi097.stdout:(102/174): openssl-devel-3.0.7-27.el9.x86_64.rp 8.0 MB/s | 4.1 MB 00:00 2024-03-11T19:42:42.155 INFO:teuthology.orchestra.run.smithi097.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 281 kB/s | 44 kB 00:00 2024-03-11T19:42:42.222 INFO:teuthology.orchestra.run.smithi097.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 467 kB/s | 94 kB 00:00 2024-03-11T19:42:42.287 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-11T19:42:42.289 INFO:teuthology.orchestra.run.smithi097.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-11T19:42:42.344 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-11T19:42:42.487 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-11T19:42:42.525 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-11T19:42:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-11T19:42:42.581 INFO:teuthology.orchestra.run.smithi097.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.0 MB/s | 863 kB 00:00 2024-03-11T19:42:42.587 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-11T19:42:42.616 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-11T19:42:42.650 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-11T19:42:42.689 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-11T19:42:42.721 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-11T19:42:42.731 INFO:teuthology.orchestra.run.smithi097.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 831 kB/s | 309 kB 00:00 2024-03-11T19:42:42.751 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-11T19:42:42.773 INFO:teuthology.orchestra.run.smithi097.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 997 kB/s | 548 kB 00:00 2024-03-11T19:42:42.798 INFO:teuthology.orchestra.run.smithi097.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 90 kB/s | 19 kB 00:00 2024-03-11T19:42:42.815 INFO:teuthology.orchestra.run.smithi097.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 627 kB/s | 26 kB 00:00 2024-03-11T19:42:42.828 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-11T19:42:42.849 INFO:teuthology.orchestra.run.smithi097.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 977 kB/s | 49 kB 00:00 2024-03-11T19:42:42.874 INFO:teuthology.orchestra.run.smithi097.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-11T19:42:42.886 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-11T19:42:42.925 INFO:teuthology.orchestra.run.smithi097.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-11T19:42:42.927 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-11T19:42:42.978 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-11T19:42:43.012 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-11T19:42:43.105 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-11T19:42:43.125 INFO:teuthology.orchestra.run.smithi097.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 2.7 MB/s | 548 kB 00:00 2024-03-11T19:42:43.175 INFO:teuthology.orchestra.run.smithi097.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 838 kB 00:00 2024-03-11T19:42:43.194 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-11T19:42:43.200 INFO:teuthology.orchestra.run.smithi097.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 584 kB/s | 43 kB 00:00 2024-03-11T19:42:43.226 INFO:teuthology.orchestra.run.smithi097.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 651 kB/s | 32 kB 00:00 2024-03-11T19:42:43.251 INFO:teuthology.orchestra.run.smithi097.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 303 kB/s | 15 kB 00:00 2024-03-11T19:42:43.252 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-11T19:42:43.283 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-11T19:42:43.301 INFO:teuthology.orchestra.run.smithi097.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.3 MB/s | 172 kB 00:00 2024-03-11T19:42:43.332 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-11T19:42:43.369 INFO:teuthology.orchestra.run.smithi097.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-11T19:42:43.410 INFO:teuthology.orchestra.run.smithi097.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 868 kB/s | 35 kB 00:00 2024-03-11T19:42:43.442 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-11T19:42:43.469 INFO:teuthology.orchestra.run.smithi097.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.8 MB/s | 303 kB 00:00 2024-03-11T19:42:43.494 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-11T19:42:43.647 INFO:teuthology.orchestra.run.smithi097.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 863 kB/s | 203 kB 00:00 2024-03-11T19:42:43.729 INFO:teuthology.orchestra.run.smithi097.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.8 MB/s | 144 kB 00:00 2024-03-11T19:42:43.770 INFO:teuthology.orchestra.run.smithi097.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 262 kB/s | 11 kB 00:00 2024-03-11T19:42:43.812 INFO:teuthology.orchestra.run.smithi097.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 433 kB/s | 18 kB 00:00 2024-03-11T19:42:43.871 INFO:teuthology.orchestra.run.smithi097.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 404 kB/s | 23 kB 00:00 2024-03-11T19:42:43.912 INFO:teuthology.orchestra.run.smithi097.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 476 kB/s | 19 kB 00:00 2024-03-11T19:42:43.954 INFO:teuthology.orchestra.run.smithi097.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 483 kB/s | 20 kB 00:00 2024-03-11T19:42:44.188 INFO:teuthology.orchestra.run.smithi097.stdout:(130/174): python3-grpcio-1.46.7-10.el9.x86_64. 2.8 MB/s | 2.0 MB 00:00 2024-03-11T19:42:44.250 INFO:teuthology.orchestra.run.smithi097.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 762 kB/s | 46 kB 00:00 2024-03-11T19:42:44.257 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-11T19:42:44.297 INFO:teuthology.orchestra.run.smithi097.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.7 MB/s | 79 kB 00:00 2024-03-11T19:42:44.306 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-11T19:42:44.339 INFO:teuthology.orchestra.run.smithi097.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-03-11T19:42:44.340 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-11T19:42:44.381 INFO:teuthology.orchestra.run.smithi097.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 2.4 MB/s | 1.0 MB 00:00 2024-03-11T19:42:44.389 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-11T19:42:44.423 INFO:teuthology.orchestra.run.smithi097.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 402 kB/s | 16 kB 00:00 2024-03-11T19:42:44.473 INFO:teuthology.orchestra.run.smithi097.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-11T19:42:44.479 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-11T19:42:44.498 INFO:teuthology.orchestra.run.smithi097.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-11T19:42:44.516 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-11T19:42:44.532 INFO:teuthology.orchestra.run.smithi097.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 486 kB/s | 28 kB 00:00 2024-03-11T19:42:44.557 INFO:teuthology.orchestra.run.smithi097.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 531 kB/s | 31 kB 00:00 2024-03-11T19:42:44.691 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-11T19:42:44.724 INFO:teuthology.orchestra.run.smithi097.stdout:(140/174): libarrow-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 4.4 MB 00:01 2024-03-11T19:42:44.730 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-11T19:42:44.749 INFO:teuthology.orchestra.run.smithi097.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 309 kB/s | 59 kB 00:00 2024-03-11T19:42:44.774 INFO:teuthology.orchestra.run.smithi097.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 779 kB/s | 188 kB 00:00 2024-03-11T19:42:44.800 INFO:teuthology.orchestra.run.smithi097.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 480 kB/s | 36 kB 00:00 2024-03-11T19:42:44.807 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-11T19:42:44.825 INFO:teuthology.orchestra.run.smithi097.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 679 kB/s | 51 kB 00:00 2024-03-11T19:42:44.867 INFO:teuthology.orchestra.run.smithi097.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-03-11T19:42:44.901 INFO:teuthology.orchestra.run.smithi097.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-11T19:42:44.926 INFO:teuthology.orchestra.run.smithi097.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 346 kB/s | 20 kB 00:00 2024-03-11T19:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.4 MB/s | 427 kB 00:00 2024-03-11T19:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-11T19:42:45.475 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-11T19:42:45.580 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-11T19:42:45.661 INFO:teuthology.orchestra.run.smithi097.stdout:(150/174): librados2-19.0.0-1607.g94a53970.el9. 5.1 MB/s | 3.4 MB 00:00 2024-03-11T19:42:45.711 INFO:teuthology.orchestra.run.smithi097.stdout:(151/174): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-03-11T19:42:45.836 INFO:teuthology.orchestra.run.smithi097.stdout:(152/174): librbd1-19.0.0-1607.g94a53970.el9.x8 4.6 MB/s | 3.1 MB 00:00 2024-03-11T19:42:45.955 INFO:teuthology.orchestra.run.smithi097.stdout:(153/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.1 MB/s | 321 kB 00:00 2024-03-11T19:42:46.111 INFO:teuthology.orchestra.run.smithi097.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 270 kB/s | 42 kB 00:00 2024-03-11T19:42:46.245 INFO:teuthology.orchestra.run.smithi097.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.8 MB/s | 248 kB 00:00 2024-03-11T19:42:46.329 INFO:teuthology.orchestra.run.smithi097.stdout:(156/174): NetworkManager-libnm-1.46.0-1.el9.x8 3.7 MB/s | 1.8 MB 00:00 2024-03-11T19:42:46.356 INFO:teuthology.orchestra.run.smithi097.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 660 kB/s | 71 kB 00:00 2024-03-11T19:42:46.422 INFO:teuthology.orchestra.run.smithi097.stdout:(158/174): NetworkManager-1.46.0-1.el9.x86_64.r 3.4 MB/s | 2.3 MB 00:00 2024-03-11T19:42:46.446 INFO:teuthology.orchestra.run.smithi097.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 313 kB/s | 36 kB 00:00 2024-03-11T19:42:46.480 INFO:teuthology.orchestra.run.smithi097.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.5 MB/s | 192 kB 00:00 2024-03-11T19:42:46.513 INFO:teuthology.orchestra.run.smithi097.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-11T19:42:46.539 INFO:teuthology.orchestra.run.smithi097.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 251 kB/s | 23 kB 00:00 2024-03-11T19:42:46.564 INFO:teuthology.orchestra.run.smithi097.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-11T19:42:46.632 INFO:teuthology.orchestra.run.smithi097.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 155 kB 00:00 2024-03-11T19:42:46.668 INFO:teuthology.orchestra.run.smithi097.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 621 kB/s | 64 kB 00:00 2024-03-11T19:42:46.693 INFO:teuthology.orchestra.run.smithi097.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 473 kB/s | 29 kB 00:00 2024-03-11T19:42:46.718 INFO:teuthology.orchestra.run.smithi097.stdout:(167/174): libmount-2.37.4-18.el9.x86_64.rpm 758 kB/s | 136 kB 00:00 2024-03-11T19:42:46.869 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-11T19:42:46.936 INFO:teuthology.orchestra.run.smithi097.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.5 MB/s | 1.2 MB 00:00 2024-03-11T19:42:46.965 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-11T19:42:46.995 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-11T19:42:47.026 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-11T19:42:47.053 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-11T19:42:47.061 INFO:teuthology.orchestra.run.smithi097.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.9 MB/s | 2.2 MB 00:00 2024-03-11T19:42:47.147 INFO:teuthology.orchestra.run.smithi097.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 5.4 MB/s | 2.3 MB 00:00 2024-03-11T19:42:47.170 INFO:teuthology.orchestra.run.smithi097.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 288 kB/s | 31 kB 00:00 2024-03-11T19:42:47.203 INFO:teuthology.orchestra.run.smithi097.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 1.7 MB/s | 465 kB 00:00 2024-03-11T19:42:47.354 INFO:teuthology.orchestra.run.smithi097.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.2 MB/s | 1.1 MB 00:00 2024-03-11T19:42:47.375 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-11T19:42:47.400 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T19:42:47.479 INFO:teuthology.orchestra.run.smithi097.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 241 kB/s | 74 kB 00:00 2024-03-11T19:42:47.491 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T19:42:47.494 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T19:42:47.494 INFO:teuthology.orchestra.run.smithi097.stdout:Total 14 MB/s | 223 MB 00:15 2024-03-11T19:42:47.521 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-11T19:42:47.557 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-11T19:42:48.042 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-11T19:42:48.070 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T19:42:48.141 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T19:42:48.171 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-11T19:42:48.906 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T19:42:49.280 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T19:42:49.280 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T19:42:50.315 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T19:42:50.351 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T19:42:50.891 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T19:42:50.929 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T19:42:51.010 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T19:42:51.010 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T19:42:51.010 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T19:42:51.011 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:42:51.038 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T19:42:51.657 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T19:42:51.658 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T19:42:59.366 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T19:42:59.494 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-11T19:42:59.543 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-11T19:42:59.580 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T19:42:59.625 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T19:42:59.711 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-11T19:42:59.819 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-11T19:42:59.872 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-11T19:42:59.895 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-11T19:42:59.969 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-11T19:43:00.373 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-11T19:43:00.419 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T19:43:00.577 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T19:43:00.611 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T19:43:00.739 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T19:43:00.779 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T19:43:00.840 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-11T19:43:01.157 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-11T19:43:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-11T19:43:01.244 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T19:43:01.394 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T19:43:01.429 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T19:43:01.528 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T19:43:01.573 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-11T19:43:01.598 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T19:43:01.660 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T19:43:01.982 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T19:43:02.031 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T19:43:02.082 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-11T19:43:02.477 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-11T19:43:02.656 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T19:43:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-03-11T19:43:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-03-11T19:43:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-03-11T19:43:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-03-11T19:43:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-03-11T19:43:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-03-11T19:43:02.657 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-03-11T19:43:02.657 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:43:02.668 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T19:43:02.771 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T19:43:02.835 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T19:43:02.850 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T19:43:02.851 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:02.851 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T19:43:02.851 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:43:02.987 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T19:43:03.071 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T19:43:03.403 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T19:43:03.592 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-11T19:43:03.726 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-11T19:43:03.764 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-11T19:43:03.800 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-11T19:43:03.866 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-11T19:43:03.990 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-11T19:43:04.055 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-11T19:43:04.097 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-11T19:43:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-11T19:43:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-11T19:43:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T19:43:04.305 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-11T19:43:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T19:43:04.322 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:04.322 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T19:43:04.322 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T19:43:04.322 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T19:43:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:43:04.349 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-11T19:43:04.382 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-11T19:43:04.495 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-11T19:43:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T19:43:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T19:43:04.553 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:04.553 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T19:43:04.553 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T19:43:04.553 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T19:43:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:43:04.614 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-11T19:43:04.705 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-11T19:43:04.744 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-11T19:43:04.873 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-11T19:43:04.937 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-11T19:43:04.989 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T19:43:05.012 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T19:43:05.012 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:05.012 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T19:43:05.012 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T19:43:05.012 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T19:43:05.012 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:43:05.037 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-11T19:43:05.107 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-11T19:43:05.224 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-11T19:43:05.290 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-11T19:43:05.458 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T19:43:05.490 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T19:43:05.523 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-11T19:43:05.584 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-11T19:43:05.612 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T19:43:05.851 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T19:43:05.879 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T19:43:05.981 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-11T19:43:06.078 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-11T19:43:06.209 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-11T19:43:06.297 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T19:43:06.325 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T19:43:06.375 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-11T19:43:06.443 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-11T19:43:06.499 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-11T19:43:06.547 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-11T19:43:06.844 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T19:43:06.871 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T19:43:06.944 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-11T19:43:06.978 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T19:43:07.142 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-11T19:43:07.162 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-11T19:43:07.182 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T19:43:07.205 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T19:43:07.205 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:07.206 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T19:43:07.206 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T19:43:07.206 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T19:43:07.206 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:43:07.258 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T19:43:07.260 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-11T19:43:07.269 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T19:43:07.315 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-11T19:43:07.317 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-11T19:43:07.518 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-11T19:43:07.565 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-11T19:43:07.638 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-11T19:43:07.733 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-11T19:43:07.782 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-11T19:43:07.820 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-11T19:43:07.913 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-11T19:43:07.972 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-11T19:43:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-11T19:43:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-11T19:43:08.175 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-11T19:43:08.227 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T19:43:08.250 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T19:43:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T19:43:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T19:43:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T19:43:08.251 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:43:08.294 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T19:43:08.298 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-11T19:43:08.316 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T19:43:08.316 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:08.316 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T19:43:08.316 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:43:08.352 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-11T19:43:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T19:43:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T19:43:08.384 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:08.384 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-11T19:43:08.384 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T19:43:08.384 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T19:43:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:43:08.389 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-11T19:43:08.438 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-11T19:43:08.566 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-11T19:43:08.605 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-11T19:43:08.670 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-11T19:43:08.696 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T19:43:08.713 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-11T19:43:08.718 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T19:43:08.718 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:08.718 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T19:43:08.718 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T19:43:08.719 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T19:43:08.719 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:43:08.746 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-11T19:43:08.919 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-11T19:43:09.089 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-11T19:43:09.231 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-11T19:43:09.289 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-11T19:43:09.352 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-11T19:43:09.425 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-11T19:43:09.678 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-11T19:43:10.029 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-11T19:43:10.196 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-11T19:43:10.241 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-11T19:43:10.315 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-11T19:43:10.353 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-11T19:43:10.396 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-11T19:43:10.443 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-11T19:43:10.509 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-11T19:43:10.569 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-11T19:43:10.616 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-11T19:43:10.733 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-11T19:43:10.777 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-11T19:43:10.813 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-11T19:43:10.874 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-11T19:43:10.916 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-11T19:43:11.058 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-11T19:43:11.175 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-11T19:43:11.232 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-11T19:43:11.268 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-11T19:43:11.324 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-11T19:43:11.463 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-11T19:43:11.525 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-11T19:43:12.485 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-11T19:43:12.534 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-11T19:43:12.567 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-11T19:43:12.623 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-11T19:43:12.733 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-11T19:43:12.784 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-11T19:43:13.015 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-11T19:43:13.047 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-11T19:43:13.124 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T19:43:13.139 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-11T19:43:13.165 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-11T19:43:13.196 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-11T19:43:13.223 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-11T19:43:13.425 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-11T19:43:13.547 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-11T19:43:13.588 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-11T19:43:13.627 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-11T19:43:13.680 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-11T19:43:13.723 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-11T19:43:13.756 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-11T19:43:13.788 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-11T19:43:13.817 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-11T19:43:13.872 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-11T19:43:13.931 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-11T19:43:13.934 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-11T19:43:13.965 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-11T19:43:14.059 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-11T19:43:14.567 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T19:43:14.605 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T19:43:14.625 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T19:43:15.478 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-11T19:43:15.559 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-11T19:43:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-11T19:43:15.601 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-11T19:43:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-11T19:43:15.685 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-11T19:43:15.699 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-11T19:43:15.729 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-11T19:43:15.730 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-11T19:43:15.754 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-11T19:43:15.779 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-11T19:43:15.794 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-11T19:43:15.805 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-11T19:43:15.805 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T19:43:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T19:43:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-11T19:43:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T19:43:16.008 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T19:43:16.034 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-11T19:43:16.067 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-11T19:43:16.092 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-11T19:43:16.117 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-11T19:43:16.150 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-11T19:43:16.168 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-11T19:43:16.190 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-11T19:43:16.222 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-11T19:43:16.227 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T19:43:16.247 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-11T19:43:16.272 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-11T19:43:16.298 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-11T19:43:16.323 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-11T19:43:16.330 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T19:43:16.330 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T19:43:16.333 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T19:43:16.367 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-11T19:43:16.424 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-11T19:43:16.509 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T19:43:16.509 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T19:43:16.594 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T19:43:16.617 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-11T19:43:16.642 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-11T19:43:16.675 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-11T19:43:16.702 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-11T19:43:16.766 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T19:43:16.940 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-11T19:43:16.971 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T19:43:17.071 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T19:43:17.098 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-11T19:43:19.585 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T19:43:19.630 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T19:43:20.283 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T19:43:20.333 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T19:43:20.481 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T19:43:20.481 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T19:43:20.482 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T19:43:20.482 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:20.526 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T19:43:21.551 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T19:43:21.551 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-11T19:43:21.552 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-11T19:43:21.552 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-11T19:43:21.552 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-11T19:43:21.552 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-11T19:43:21.552 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-11T19:43:21.552 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-11T19:43:21.552 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-11T19:43:21.552 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-11T19:43:21.552 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T19:43:21.552 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T19:43:21.552 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T19:43:21.552 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-11T19:43:21.553 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-11T19:43:21.569 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-11T19:43:21.569 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-11T19:43:21.570 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-11T19:43:21.570 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-11T19:43:21.570 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-11T19:43:21.570 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-11T19:43:21.570 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-11T19:43:21.570 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-11T19:43:21.570 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-11T19:43:21.570 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-11T19:43:21.570 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-11T19:43:21.570 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-11T19:43:21.570 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-11T19:43:21.570 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-11T19:43:21.570 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-11T19:43:21.571 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-11T19:43:21.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-11T19:43:21.573 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-11T19:43:21.574 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-11T19:43:21.575 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-11T19:43:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-11T19:43:21.577 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-11T19:43:21.578 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-11T19:43:21.578 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-11T19:43:21.578 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-11T19:43:21.578 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-11T19:43:21.578 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-11T19:43:21.578 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-11T19:43:21.578 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-11T19:43:21.578 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T19:43:21.578 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-11T19:43:21.578 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T19:43:21.578 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-11T19:43:21.578 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-11T19:43:21.578 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-11T19:43:21.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-11T19:43:21.580 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-11T19:43:21.581 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-11T19:43:21.581 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-11T19:43:21.581 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-11T19:43:21.581 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-11T19:43:21.581 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-11T19:43:21.581 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-11T19:43:21.581 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-11T19:43:21.581 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-11T19:43:21.581 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-11T19:43:21.581 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-11T19:43:21.581 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-11T19:43:21.581 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-11T19:43:24.556 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-11T19:43:24.556 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:43:24.556 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-03-11T19:43:24.556 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-11T19:43:24.556 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-11T19:43:24.556 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-11T19:43:24.556 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-11T19:43:24.556 INFO:teuthology.orchestra.run.smithi114.stdout: acl-2.3.1-4.el9.x86_64 2024-03-11T19:43:24.556 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-11T19:43:24.557 INFO:teuthology.orchestra.run.smithi114.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout: automake-1.16.2-8.el9.noarch 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout: bison-3.7.4-5.el9.x86_64 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.558 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.559 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: flex-2.6.4-9.el9.x86_64 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: jq-1.6-16.el9.x86_64 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-11T19:43:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T19:43:24.561 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T19:43:24.561 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.561 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.561 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.561 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T19:43:24.561 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T19:43:24.561 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T19:43:24.561 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T19:43:24.561 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T19:43:24.561 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.561 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.561 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T19:43:24.561 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T19:43:24.562 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-13.11-1.el9.x86_64 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T19:43:24.563 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T19:43:24.564 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T19:43:24.565 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T19:43:24.566 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T19:43:24.567 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T19:43:24.568 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.568 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.568 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:24.568 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T19:43:24.568 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T19:43:24.568 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T19:43:24.568 INFO:teuthology.orchestra.run.smithi114.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-11T19:43:24.568 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-11T19:43:24.568 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T19:43:24.571 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-11T19:43:24.571 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-11T19:43:24.571 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-11T19:43:25.042 DEBUG:teuthology.parallel:result is None 2024-03-11T19:43:33.647 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T19:43:33.647 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /sys 2024-03-11T19:43:33.647 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /proc 2024-03-11T19:43:33.647 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /mnt 2024-03-11T19:43:33.647 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /var/tmp 2024-03-11T19:43:33.647 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /home 2024-03-11T19:43:33.647 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /root 2024-03-11T19:43:33.647 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /tmp 2024-03-11T19:43:33.647 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:33.843 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T19:43:33.892 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T19:43:33.892 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:33.892 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T19:43:33.892 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:35.427 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T19:43:35.506 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T19:43:35.507 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:35.507 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T19:43:35.507 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T19:43:35.507 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T19:43:35.507 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:35.728 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T19:43:35.793 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T19:43:35.793 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:35.793 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T19:43:35.793 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T19:43:35.793 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T19:43:35.794 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:36.270 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T19:43:36.334 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T19:43:36.334 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:36.335 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T19:43:36.335 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T19:43:36.335 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T19:43:36.335 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:36.447 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-11T19:43:37.049 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T19:43:37.090 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T19:43:37.161 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-11T19:43:37.269 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-11T19:43:37.302 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T19:43:37.626 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T19:43:37.661 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T19:43:38.778 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T19:43:38.811 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T19:43:38.953 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T19:43:39.126 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-11T19:43:39.164 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T19:43:39.251 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T19:43:39.252 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:39.252 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T19:43:39.252 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T19:43:39.252 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T19:43:39.252 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:39.306 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T19:43:39.344 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T19:43:39.441 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-11T19:43:40.390 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-11T19:43:40.425 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T19:43:40.495 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:40.543 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T19:43:40.614 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T19:43:40.614 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:40.614 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T19:43:40.614 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:40.719 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T19:43:40.792 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T19:43:40.792 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:40.792 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-11T19:43:40.793 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T19:43:40.793 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T19:43:40.793 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:41.169 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T19:43:41.239 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T19:43:41.239 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:43:41.239 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T19:43:41.239 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T19:43:41.239 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T19:43:41.239 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:45.987 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T19:43:46.044 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-11T19:43:46.076 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-11T19:43:46.115 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-11T19:43:46.387 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-11T19:43:46.511 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-11T19:43:46.560 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-11T19:43:46.618 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-11T19:43:46.676 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-11T19:43:46.720 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-11T19:43:46.746 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-11T19:43:46.792 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-11T19:43:46.844 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-11T19:43:46.934 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-11T19:43:47.082 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-11T19:43:47.127 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-11T19:43:48.027 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T19:43:48.075 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T19:43:48.139 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T19:43:49.470 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-11T19:43:49.547 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-11T19:43:49.600 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-11T19:43:49.717 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-11T19:43:49.766 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-11T19:43:49.791 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-11T19:43:49.825 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-11T19:43:49.865 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-11T19:43:49.866 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T19:43:50.004 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T19:43:50.046 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-11T19:43:50.047 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T19:43:50.209 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T19:43:50.253 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-11T19:43:50.299 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-11T19:43:50.333 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-11T19:43:50.365 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-11T19:43:50.436 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-11T19:43:50.538 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-11T19:43:50.596 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-11T19:43:50.629 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-11T19:43:50.662 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-11T19:43:50.696 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-11T19:43:50.729 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-11T19:43:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T19:43:50.766 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T19:43:51.080 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T19:43:51.081 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T19:43:51.219 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T19:43:51.248 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-11T19:43:51.307 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-11T19:43:51.366 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-11T19:43:51.409 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-11T19:43:51.528 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T19:43:51.781 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T19:43:51.782 INFO:teuthology.orchestra.run.smithi097.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-11T19:43:51.782 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:56.533 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-11T19:43:56.534 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-11T19:43:56.534 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-11T19:43:56.534 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-11T19:43:56.534 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-11T19:43:56.534 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-11T19:43:56.534 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-11T19:43:56.534 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-11T19:43:56.534 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-11T19:43:56.534 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T19:43:56.534 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T19:43:56.534 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T19:43:56.534 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-11T19:43:56.535 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-11T19:43:56.535 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-11T19:43:56.535 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T19:43:56.535 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-11T19:43:56.535 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-11T19:43:56.535 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-11T19:43:56.535 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-11T19:43:56.535 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-11T19:43:56.535 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-11T19:43:56.535 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-11T19:43:56.535 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-11T19:43:56.535 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-11T19:43:56.551 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-11T19:43:56.552 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-11T19:43:56.552 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-11T19:43:56.552 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-11T19:43:56.552 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-11T19:43:56.552 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-11T19:43:56.552 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-11T19:43:56.552 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-11T19:43:56.552 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-11T19:43:56.552 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-11T19:43:56.552 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-11T19:43:56.552 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-11T19:43:56.552 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-11T19:43:56.553 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-11T19:43:56.554 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-11T19:43:56.555 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-11T19:43:56.556 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-11T19:43:56.557 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-11T19:43:56.558 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-11T19:43:56.559 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-11T19:43:56.559 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-11T19:43:56.559 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-11T19:43:56.559 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-11T19:43:56.559 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-11T19:43:56.559 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-11T19:43:56.559 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-11T19:43:56.559 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-11T19:43:56.559 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-11T19:43:56.560 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-11T19:43:56.560 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-11T19:43:56.560 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-11T19:43:56.560 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-11T19:43:56.560 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-11T19:43:56.560 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-11T19:43:56.560 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-11T19:43:56.560 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-11T19:43:56.560 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-11T19:43:56.560 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-11T19:43:56.560 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-11T19:43:56.561 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-11T19:43:56.561 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-11T19:43:56.561 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-11T19:43:56.561 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-11T19:43:56.561 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-11T19:43:56.561 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-11T19:43:56.561 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T19:43:56.561 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-11T19:43:56.561 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T19:43:56.561 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-11T19:43:56.562 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-11T19:43:56.562 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-11T19:43:56.562 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-11T19:43:56.562 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-11T19:43:56.562 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-11T19:43:56.562 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-11T19:43:56.562 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-11T19:43:56.562 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-11T19:43:56.562 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-11T19:43:56.562 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-11T19:43:56.562 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-11T19:43:56.562 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-11T19:43:56.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-11T19:43:56.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-11T19:43:56.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-11T19:43:56.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-11T19:43:56.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-11T19:43:56.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-11T19:43:56.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-11T19:43:56.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-11T19:43:56.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-11T19:43:56.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-11T19:43:56.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-11T19:43:56.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-11T19:43:56.564 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-11T19:43:56.564 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-11T19:43:56.564 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-11T19:43:56.564 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-11T19:43:56.564 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-11T19:43:56.564 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-11T19:43:56.564 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-11T19:43:56.564 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-11T19:43:56.564 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-11T19:43:56.564 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-11T19:43:56.564 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-11T19:43:56.564 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-11T19:43:56.565 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-11T19:43:56.565 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-11T19:43:56.565 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-11T19:43:56.565 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-11T19:43:56.565 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-11T19:43:56.565 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-11T19:43:56.565 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-11T19:43:59.244 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-11T19:43:59.244 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:59.244 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2024-03-11T19:43:59.244 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-11T19:43:59.244 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-11T19:43:59.244 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-11T19:43:59.244 INFO:teuthology.orchestra.run.smithi097.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-11T19:43:59.244 INFO:teuthology.orchestra.run.smithi097.stdout: acl-2.3.1-4.el9.x86_64 2024-03-11T19:43:59.244 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-11T19:43:59.244 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-11T19:43:59.244 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-11T19:43:59.244 INFO:teuthology.orchestra.run.smithi097.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-11T19:43:59.245 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout: automake-1.16.2-8.el9.noarch 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout: bison-3.7.4-5.el9.x86_64 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.246 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.247 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: flex-2.6.4-9.el9.x86_64 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: jq-1.6-16.el9.x86_64 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T19:43:59.248 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-11T19:43:59.249 INFO:teuthology.orchestra.run.smithi097.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-13.11-1.el9.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T19:43:59.250 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-11T19:43:59.251 INFO:teuthology.orchestra.run.smithi097.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T19:43:59.252 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T19:43:59.253 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T19:43:59.254 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T19:43:59.255 INFO:teuthology.orchestra.run.smithi097.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T19:43:59.256 INFO:teuthology.orchestra.run.smithi097.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T19:43:59.256 INFO:teuthology.orchestra.run.smithi097.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-11T19:43:59.256 INFO:teuthology.orchestra.run.smithi097.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-11T19:43:59.256 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T19:43:59.256 INFO:teuthology.orchestra.run.smithi097.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-11T19:43:59.256 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:43:59.256 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T19:43:59.908 DEBUG:teuthology.parallel:result is None 2024-03-11T19:44:08.760 INFO:teuthology.orchestra.run.smithi161.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 215 B/s | 21 kB 01:42 2024-03-11T19:44:08.777 INFO:teuthology.orchestra.run.smithi161.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 257 B/s | 25 kB 01:41 2024-03-11T19:44:08.894 INFO:teuthology.orchestra.run.smithi161.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-03-11T19:44:08.919 INFO:teuthology.orchestra.run.smithi161.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.8 MB/s | 248 kB 00:00 2024-03-11T19:44:08.961 INFO:teuthology.orchestra.run.smithi161.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 839 kB/s | 35 kB 00:00 2024-03-11T19:44:08.986 INFO:teuthology.orchestra.run.smithi161.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.3 MB/s | 216 kB 00:00 2024-03-11T19:44:09.070 INFO:teuthology.orchestra.run.smithi161.stdout:(102/174): openssl-devel-3.0.7-27.el9.x86_64.rp 42 kB/s | 4.1 MB 01:41 2024-03-11T19:44:09.095 INFO:teuthology.orchestra.run.smithi161.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 333 kB/s | 44 kB 00:00 2024-03-11T19:44:09.120 INFO:teuthology.orchestra.run.smithi161.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 700 kB/s | 94 kB 00:00 2024-03-11T19:44:09.421 INFO:teuthology.orchestra.run.smithi161.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 64 kB/s | 22 kB 00:00 2024-03-11T19:44:09.796 INFO:teuthology.orchestra.run.smithi161.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 824 kB/s | 309 kB 00:00 2024-03-11T19:44:09.821 INFO:teuthology.orchestra.run.smithi161.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 782 kB/s | 548 kB 00:00 2024-03-11T19:44:09.846 INFO:teuthology.orchestra.run.smithi161.stdout:(108/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-03-11T19:44:09.871 INFO:teuthology.orchestra.run.smithi161.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-11T19:44:10.105 INFO:teuthology.orchestra.run.smithi161.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 100 kB/s | 26 kB 00:00 2024-03-11T19:44:10.130 INFO:teuthology.orchestra.run.smithi161.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 188 kB/s | 49 kB 00:00 2024-03-11T19:44:10.189 INFO:teuthology.orchestra.run.smithi161.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-11T19:44:10.214 INFO:teuthology.orchestra.run.smithi161.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 619 kB/s | 67 kB 00:00 2024-03-11T19:44:10.272 INFO:teuthology.orchestra.run.smithi161.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-11T19:44:10.314 INFO:teuthology.orchestra.run.smithi161.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.6 MB/s | 838 kB 00:00 2024-03-11T19:44:10.339 INFO:teuthology.orchestra.run.smithi161.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-03-11T19:44:10.364 INFO:teuthology.orchestra.run.smithi161.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 480 kB/s | 43 kB 00:00 2024-03-11T19:44:10.389 INFO:teuthology.orchestra.run.smithi161.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 433 kB/s | 32 kB 00:00 2024-03-11T19:44:10.415 INFO:teuthology.orchestra.run.smithi161.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-03-11T19:44:10.440 INFO:teuthology.orchestra.run.smithi161.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-11T19:44:10.465 INFO:teuthology.orchestra.run.smithi161.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-03-11T19:44:10.491 INFO:teuthology.orchestra.run.smithi161.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-11T19:44:10.516 INFO:teuthology.orchestra.run.smithi161.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 471 kB/s | 35 kB 00:00 2024-03-11T19:44:10.541 INFO:teuthology.orchestra.run.smithi161.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.7 MB/s | 203 kB 00:00 2024-03-11T19:44:10.633 INFO:teuthology.orchestra.run.smithi161.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-11T19:44:10.658 INFO:teuthology.orchestra.run.smithi161.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-03-11T19:44:10.684 INFO:teuthology.orchestra.run.smithi161.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 76 kB/s | 11 kB 00:00 2024-03-11T19:44:10.725 INFO:teuthology.orchestra.run.smithi161.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2024-03-11T19:44:10.776 INFO:teuthology.orchestra.run.smithi161.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-03-11T19:44:10.801 INFO:teuthology.orchestra.run.smithi161.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-03-11T19:44:11.033 INFO:teuthology.orchestra.run.smithi161.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-03-11T19:44:11.033 INFO:teuthology.orchestra.run.smithi161.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-03-11T19:44:11.033 INFO:teuthology.orchestra.run.smithi161.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 395 kB/s | 46 kB 00:00 2024-03-11T19:44:11.033 INFO:teuthology.orchestra.run.smithi161.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 672 kB/s | 79 kB 00:00 2024-03-11T19:44:11.033 INFO:teuthology.orchestra.run.smithi161.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 628 kB/s | 58 kB 00:00 2024-03-11T19:44:11.033 INFO:teuthology.orchestra.run.smithi161.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-11T19:44:11.045 INFO:teuthology.orchestra.run.smithi161.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-03-11T19:44:11.078 INFO:teuthology.orchestra.run.smithi161.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 979 kB/s | 90 kB 00:00 2024-03-11T19:44:11.112 INFO:teuthology.orchestra.run.smithi161.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 306 kB/s | 28 kB 00:00 2024-03-11T19:44:11.137 INFO:teuthology.orchestra.run.smithi161.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-03-11T19:44:11.179 INFO:teuthology.orchestra.run.smithi161.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-11T19:44:11.212 INFO:teuthology.orchestra.run.smithi161.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-03-11T19:44:11.238 INFO:teuthology.orchestra.run.smithi161.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-03-11T19:44:11.280 INFO:teuthology.orchestra.run.smithi161.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-03-11T19:44:11.305 INFO:teuthology.orchestra.run.smithi161.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-11T19:44:11.334 INFO:teuthology.orchestra.run.smithi161.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 975 kB/s | 90 kB 00:00 2024-03-11T19:44:11.364 INFO:teuthology.orchestra.run.smithi161.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-11T19:44:11.381 INFO:teuthology.orchestra.run.smithi161.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-11T19:44:11.423 INFO:teuthology.orchestra.run.smithi161.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.1 MB/s | 192 kB 00:00 2024-03-11T19:44:11.623 INFO:teuthology.orchestra.run.smithi161.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-03-11T19:44:11.665 INFO:teuthology.orchestra.run.smithi161.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-11T19:44:11.832 INFO:teuthology.orchestra.run.smithi161.stdout:(152/174): librbd1-19.0.0-1607.g94a53970.el9.x8 7.7 MB/s | 3.1 MB 00:00 2024-03-11T19:44:12.599 INFO:teuthology.orchestra.run.smithi161.stdout:(153/174): librados2-19.0.0-1607.g94a53970.el9. 2.8 MB/s | 3.4 MB 00:01 2024-03-11T19:44:13.432 INFO:teuthology.orchestra.run.smithi161.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 51 kB/s | 42 kB 00:00 2024-03-11T19:44:14.499 INFO:teuthology.orchestra.run.smithi161.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 233 kB/s | 248 kB 00:01 2024-03-11T19:44:15.282 INFO:teuthology.orchestra.run.smithi161.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 91 kB/s | 71 kB 00:00 2024-03-11T19:44:15.482 INFO:teuthology.orchestra.run.smithi161.stdout:(157/174): NetworkManager-libnm-1.46.0-1.el9.x8 510 kB/s | 1.8 MB 00:03 2024-03-11T19:44:16.449 INFO:teuthology.orchestra.run.smithi161.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 31 kB/s | 36 kB 00:01 2024-03-11T19:44:16.541 INFO:teuthology.orchestra.run.smithi161.stdout:(159/174): NetworkManager-1.46.0-1.el9.x86_64.r 481 kB/s | 2.3 MB 00:04 2024-03-11T19:44:16.733 INFO:teuthology.orchestra.run.smithi161.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 154 kB/s | 192 kB 00:01 2024-03-11T19:44:17.325 INFO:teuthology.orchestra.run.smithi161.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 29 kB/s | 23 kB 00:00 2024-03-11T19:44:17.633 INFO:teuthology.orchestra.run.smithi161.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 120 kB/s | 108 kB 00:00 2024-03-11T19:44:17.658 INFO:teuthology.orchestra.run.smithi161.stdout:(163/174): elfutils-libs-0.190-2.el9.x86_64.rpm 211 kB/s | 256 kB 00:01 2024-03-11T19:44:18.245 INFO:teuthology.orchestra.run.smithi161.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 168 kB/s | 155 kB 00:00 2024-03-11T19:44:18.428 INFO:teuthology.orchestra.run.smithi161.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 83 kB/s | 64 kB 00:00 2024-03-11T19:44:18.486 INFO:teuthology.orchestra.run.smithi161.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 159 kB/s | 136 kB 00:00 2024-03-11T19:44:18.920 INFO:teuthology.orchestra.run.smithi161.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 42 kB/s | 29 kB 00:00 2024-03-11T19:44:21.138 INFO:teuthology.orchestra.run.smithi161.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 458 kB/s | 1.2 MB 00:02 2024-03-11T19:44:22.545 INFO:teuthology.orchestra.run.smithi161.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 544 kB/s | 2.2 MB 00:04 2024-03-11T19:44:22.579 INFO:teuthology.orchestra.run.smithi161.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 323 kB/s | 465 kB 00:01 2024-03-11T19:44:22.912 INFO:teuthology.orchestra.run.smithi161.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 586 kB/s | 2.3 MB 00:03 2024-03-11T19:44:22.971 INFO:teuthology.orchestra.run.smithi161.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 74 kB 00:00 2024-03-11T19:44:23.237 INFO:teuthology.orchestra.run.smithi161.stdout:(173/174): util-linux-user-2.37.4-18.el9.x86_64 45 kB/s | 31 kB 00:00 2024-03-11T19:44:24.812 INFO:teuthology.orchestra.run.smithi161.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 492 kB/s | 1.1 MB 00:02 2024-03-11T19:44:24.818 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-11T19:44:24.818 INFO:teuthology.orchestra.run.smithi161.stdout:Total 1.7 MB/s | 223 MB 02:07 2024-03-11T19:44:25.268 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-11T19:44:25.541 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-11T19:44:25.541 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-11T19:44:27.295 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-11T19:44:27.295 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-11T19:44:34.095 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-11T19:44:34.224 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-11T19:44:34.250 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-11T19:44:34.289 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T19:44:34.298 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T19:44:34.338 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-11T19:44:34.391 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-11T19:44:34.425 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-11T19:44:34.451 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-11T19:44:34.502 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-11T19:44:34.819 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-11T19:44:34.877 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T19:44:34.980 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T19:44:35.003 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T19:44:35.076 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T19:44:35.107 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T19:44:35.143 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-11T19:44:35.312 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-11T19:44:35.388 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-11T19:44:35.416 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T19:44:35.511 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T19:44:35.534 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T19:44:35.590 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T19:44:35.629 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-11T19:44:35.652 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T19:44:35.667 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T19:44:35.860 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T19:44:35.891 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T19:44:35.923 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-11T19:44:36.028 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-11T19:44:36.125 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T19:44:36.185 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T19:44:36.246 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T19:44:36.311 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T19:44:36.525 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T19:44:36.633 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-11T19:44:36.752 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-11T19:44:36.791 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-11T19:44:36.817 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-11T19:44:36.853 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-11T19:44:36.926 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-11T19:44:36.981 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-11T19:44:37.019 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-11T19:44:37.083 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-11T19:44:37.119 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-11T19:44:37.162 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-11T19:44:37.194 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-11T19:44:37.223 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-11T19:44:37.292 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-11T19:44:37.335 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-11T19:44:37.379 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-11T19:44:37.407 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-11T19:44:37.477 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-11T19:44:37.522 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-11T19:44:37.594 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-11T19:44:37.728 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-11T19:44:37.774 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-11T19:44:38.292 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-11T19:44:38.370 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-11T19:44:38.449 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-11T19:44:38.489 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T19:44:38.497 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T19:44:38.529 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-11T19:44:38.569 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-11T19:44:38.607 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-11T19:44:38.637 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-11T19:44:38.863 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-11T19:44:39.053 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-11T19:44:39.115 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-11T19:44:39.158 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-11T19:44:39.292 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-11T19:44:39.344 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-11T19:44:39.398 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-11T19:44:39.464 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-11T19:44:39.495 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-11T19:44:39.531 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-11T19:44:39.578 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-11T19:44:39.614 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-11T19:44:39.657 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-11T19:44:39.699 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-11T19:44:39.833 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-11T19:44:39.879 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-11T19:44:39.909 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-11T19:44:39.950 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-11T19:44:40.044 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-11T19:44:40.084 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-11T19:44:40.124 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-11T19:44:40.149 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-11T19:44:40.174 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-11T19:44:40.237 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-11T19:44:40.415 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-11T19:44:40.523 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-11T19:44:40.565 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-11T19:44:40.614 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-11T19:44:40.670 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-11T19:44:40.855 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-11T19:44:40.902 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-11T19:44:41.032 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-11T19:44:41.067 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-11T19:44:41.098 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-11T19:44:41.130 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-11T19:44:41.158 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-11T19:44:41.192 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-11T19:44:41.231 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-11T19:44:41.262 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-11T19:44:41.301 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-11T19:44:41.377 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-11T19:44:41.411 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-11T19:44:41.445 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-11T19:44:41.486 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-11T19:44:41.520 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-11T19:44:41.609 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-11T19:44:41.700 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-11T19:44:41.743 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-11T19:44:41.774 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-11T19:44:41.822 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-11T19:44:41.920 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-11T19:44:41.967 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-11T19:44:42.684 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-11T19:44:42.768 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-11T19:44:42.768 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-11T19:44:42.812 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-11T19:44:42.891 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-11T19:44:42.932 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-11T19:44:43.111 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-11T19:44:43.138 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-11T19:44:43.215 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-11T19:44:43.911 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-11T19:44:44.017 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-11T19:44:45.228 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-11T19:44:45.313 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-11T19:44:45.343 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-11T19:44:45.375 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-11T19:44:45.402 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-11T19:44:45.709 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-11T19:44:45.749 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T19:44:45.840 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T19:44:45.870 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-11T19:44:45.906 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-11T19:44:46.360 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-11T19:44:46.393 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T19:44:46.464 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T19:44:46.486 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-11T19:44:48.615 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T19:44:48.649 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T19:44:49.183 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T19:44:49.219 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T19:44:49.301 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T19:44:49.301 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T19:44:49.301 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T19:44:49.301 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:44:49.329 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T19:45:01.142 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T19:45:01.142 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /sys 2024-03-11T19:45:01.142 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /proc 2024-03-11T19:45:01.142 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /mnt 2024-03-11T19:45:01.142 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /var/tmp 2024-03-11T19:45:01.142 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /home 2024-03-11T19:45:01.142 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /root 2024-03-11T19:45:01.143 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /tmp 2024-03-11T19:45:01.143 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:45:01.307 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T19:45:01.325 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T19:45:01.325 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:45:01.325 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T19:45:01.325 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:45:02.811 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T19:45:02.836 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T19:45:02.836 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:45:02.836 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T19:45:02.836 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T19:45:02.836 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T19:45:02.837 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:45:03.044 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T19:45:03.067 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T19:45:03.067 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:45:03.067 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T19:45:03.067 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T19:45:03.067 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T19:45:03.068 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:45:03.512 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T19:45:03.534 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T19:45:03.534 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:45:03.535 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T19:45:03.535 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T19:45:03.535 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T19:45:03.535 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:45:03.630 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-11T19:45:04.081 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T19:45:04.111 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T19:45:04.143 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-11T19:45:04.203 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-11T19:45:04.234 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T19:45:04.467 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T19:45:04.502 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T19:45:05.469 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T19:45:05.494 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T19:45:05.592 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T19:45:05.747 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-11T19:45:05.779 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T19:45:05.802 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T19:45:05.802 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:45:05.803 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T19:45:05.803 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T19:45:05.803 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T19:45:05.803 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:45:05.855 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T19:45:05.866 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T19:45:05.911 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-11T19:45:06.758 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-11T19:45:06.781 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T19:45:06.804 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T19:45:06.804 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:45:06.804 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T19:45:06.804 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T19:45:06.804 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T19:45:06.804 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:45:06.849 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T19:45:06.870 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T19:45:06.871 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:45:06.871 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T19:45:06.871 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:45:06.916 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T19:45:06.938 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T19:45:06.938 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:45:06.938 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-11T19:45:06.938 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T19:45:06.938 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T19:45:06.939 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:45:07.242 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T19:45:07.264 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T19:45:07.264 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T19:45:07.264 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T19:45:07.265 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T19:45:07.265 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T19:45:07.265 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:45:11.690 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T19:45:11.731 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-11T19:45:11.761 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-11T19:45:11.790 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-11T19:45:11.962 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-11T19:45:12.082 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-11T19:45:12.128 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-11T19:45:12.167 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-11T19:45:12.228 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-11T19:45:12.271 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-11T19:45:12.304 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-11T19:45:12.354 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-11T19:45:12.382 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-11T19:45:12.428 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-11T19:45:12.481 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-11T19:45:12.514 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-11T19:45:13.087 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T19:45:13.120 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T19:45:13.139 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T19:45:13.966 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-11T19:45:14.057 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-11T19:45:14.097 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-11T19:45:14.178 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-11T19:45:14.227 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-11T19:45:14.252 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-11T19:45:14.277 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-11T19:45:14.303 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-11T19:45:14.304 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T19:45:14.391 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T19:45:14.421 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-11T19:45:14.421 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T19:45:14.506 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T19:45:14.532 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-11T19:45:14.565 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-11T19:45:14.591 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-11T19:45:14.616 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-11T19:45:14.649 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-11T19:45:14.688 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-11T19:45:14.720 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-11T19:45:14.745 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-11T19:45:14.771 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-11T19:45:14.796 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-11T19:45:14.821 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-11T19:45:14.827 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T19:45:14.827 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T19:45:15.016 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T19:45:15.016 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T19:45:15.099 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T19:45:15.124 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-11T19:45:15.149 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-11T19:45:15.182 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-11T19:45:15.208 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-11T19:45:15.280 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T19:45:20.212 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T19:45:20.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-11T19:45:20.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-11T19:45:20.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-11T19:45:20.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-11T19:45:20.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-11T19:45:20.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-11T19:45:20.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-11T19:45:20.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-11T19:45:20.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-11T19:45:20.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T19:45:20.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T19:45:20.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T19:45:20.213 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-11T19:45:20.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-11T19:45:20.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-11T19:45:20.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T19:45:20.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-11T19:45:20.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-11T19:45:20.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-11T19:45:20.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-11T19:45:20.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-11T19:45:20.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-11T19:45:20.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-11T19:45:20.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-11T19:45:20.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-11T19:45:20.214 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-11T19:45:20.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-11T19:45:20.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-11T19:45:20.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-11T19:45:20.215 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-11T19:45:20.231 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-11T19:45:20.231 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-11T19:45:20.231 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-11T19:45:20.231 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-11T19:45:20.231 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-11T19:45:20.231 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-11T19:45:20.231 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-11T19:45:20.232 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-11T19:45:20.233 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-11T19:45:20.233 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-11T19:45:20.233 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-11T19:45:20.233 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-11T19:45:20.233 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-11T19:45:20.233 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-11T19:45:20.233 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-11T19:45:20.233 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-11T19:45:20.233 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-11T19:45:20.233 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-11T19:45:20.233 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-11T19:45:20.233 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-11T19:45:20.233 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-11T19:45:20.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-11T19:45:20.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-11T19:45:20.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-11T19:45:20.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-11T19:45:20.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-11T19:45:20.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-11T19:45:20.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-11T19:45:20.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-11T19:45:20.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-11T19:45:20.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-11T19:45:20.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-11T19:45:20.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-11T19:45:20.234 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-11T19:45:20.235 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-11T19:45:20.236 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-11T19:45:20.237 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-11T19:45:20.237 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-11T19:45:20.237 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-11T19:45:20.237 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-11T19:45:20.237 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-11T19:45:20.237 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-11T19:45:20.237 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-11T19:45:20.237 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-11T19:45:20.237 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-11T19:45:20.237 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-11T19:45:20.237 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-11T19:45:20.237 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-11T19:45:20.237 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-11T19:45:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-11T19:45:20.239 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-11T19:45:20.240 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-11T19:45:20.241 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-11T19:45:20.242 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-11T19:45:20.243 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-11T19:45:20.243 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-11T19:45:20.243 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-11T19:45:20.243 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-11T19:45:20.243 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-11T19:45:20.243 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-11T19:45:20.243 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-11T19:45:20.243 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-11T19:45:23.315 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-11T19:45:23.315 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:45:23.315 INFO:teuthology.orchestra.run.smithi161.stdout:Upgraded: 2024-03-11T19:45:23.315 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-11T19:45:23.315 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-11T19:45:23.315 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-11T19:45:23.315 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-11T19:45:23.315 INFO:teuthology.orchestra.run.smithi161.stdout: acl-2.3.1-4.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-11T19:45:23.316 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: automake-1.16.2-8.el9.noarch 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: bison-3.7.4-5.el9.x86_64 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.317 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:45:23.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:45:23.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:45:23.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:45:23.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:45:23.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:45:23.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.318 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:45:23.319 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.319 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.319 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.319 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:45:23.319 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:45:23.319 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.319 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.319 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T19:45:23.319 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-11T19:45:23.319 INFO:teuthology.orchestra.run.smithi161.stdout: flex-2.6.4-9.el9.x86_64 2024-03-11T19:45:23.319 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T19:45:23.320 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T19:45:23.320 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T19:45:23.320 INFO:teuthology.orchestra.run.smithi161.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-11T19:45:23.320 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T19:45:23.320 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T19:45:23.320 INFO:teuthology.orchestra.run.smithi161.stdout: jq-1.6-16.el9.x86_64 2024-03-11T19:45:23.320 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-11T19:45:23.320 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T19:45:23.321 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T19:45:23.321 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.321 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.321 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.321 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T19:45:23.321 INFO:teuthology.orchestra.run.smithi161.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T19:45:23.321 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T19:45:23.321 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T19:45:23.321 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T19:45:23.321 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.321 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.322 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T19:45:23.322 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.322 INFO:teuthology.orchestra.run.smithi161.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-11T19:45:23.322 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T19:45:23.322 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-11T19:45:23.322 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-11T19:45:23.322 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T19:45:23.322 INFO:teuthology.orchestra.run.smithi161.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T19:45:23.322 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T19:45:23.322 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T19:45:23.322 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T19:45:23.323 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T19:45:23.323 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T19:45:23.323 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-11T19:45:23.323 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T19:45:23.323 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-13.11-1.el9.x86_64 2024-03-11T19:45:23.323 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-11T19:45:23.323 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-11T19:45:23.323 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-11T19:45:23.323 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T19:45:23.323 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T19:45:23.323 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T19:45:23.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-11T19:45:23.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T19:45:23.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T19:45:23.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T19:45:23.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T19:45:23.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T19:45:23.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T19:45:23.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T19:45:23.324 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-11T19:45:23.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-11T19:45:23.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T19:45:23.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T19:45:23.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T19:45:23.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T19:45:23.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T19:45:23.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T19:45:23.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T19:45:23.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T19:45:23.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T19:45:23.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T19:45:23.325 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T19:45:23.326 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T19:45:23.326 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T19:45:23.326 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T19:45:23.326 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T19:45:23.326 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T19:45:23.326 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T19:45:23.326 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T19:45:23.326 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T19:45:23.326 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T19:45:23.326 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T19:45:23.326 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T19:45:23.327 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T19:45:23.327 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T19:45:23.327 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T19:45:23.327 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T19:45:23.327 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T19:45:23.327 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T19:45:23.327 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T19:45:23.327 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T19:45:23.327 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T19:45:23.327 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T19:45:23.327 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-11T19:45:23.327 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T19:45:23.328 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.328 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.328 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T19:45:23.328 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T19:45:23.328 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T19:45:23.328 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.328 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T19:45:23.328 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T19:45:23.328 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T19:45:23.328 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T19:45:23.328 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T19:45:23.328 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T19:45:23.329 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T19:45:23.329 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-11T19:45:23.329 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T19:45:23.329 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T19:45:23.329 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T19:45:23.329 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T19:45:23.329 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T19:45:23.329 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T19:45:23.329 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T19:45:23.329 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.329 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.329 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T19:45:23.330 INFO:teuthology.orchestra.run.smithi161.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T19:45:23.330 INFO:teuthology.orchestra.run.smithi161.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T19:45:23.330 INFO:teuthology.orchestra.run.smithi161.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T19:45:23.330 INFO:teuthology.orchestra.run.smithi161.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-11T19:45:23.330 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-11T19:45:23.330 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T19:45:23.331 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-11T19:45:23.331 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:45:23.331 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-11T19:45:23.796 DEBUG:teuthology.parallel:result is None 2024-03-11T19:45:23.796 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T19:45:23.940 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T19:45:24.012 INFO:teuthology.orchestra.run.smithi097.stdout:19.0.0-1607.g94a53970.el9 2024-03-11T19:45:24.012 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-11T19:45:24.013 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-11T19:45:24.014 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T19:45:24.226 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T19:45:24.252 INFO:teuthology.orchestra.run.smithi114.stdout:19.0.0-1607.g94a53970.el9 2024-03-11T19:45:24.252 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-11T19:45:24.252 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-11T19:45:24.254 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T19:45:24.475 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T19:45:24.500 INFO:teuthology.orchestra.run.smithi161.stdout:19.0.0-1607.g94a53970.el9 2024-03-11T19:45:24.501 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-11T19:45:24.501 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-11T19:45:24.503 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-11T19:45:24.503 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:45:24.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T19:45:24.605 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-11T19:45:24.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T19:45:24.639 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-11T19:45:24.639 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T19:45:24.674 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-11T19:45:24.674 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:45:24.674 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T19:45:24.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T19:45:24.869 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-11T19:45:24.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T19:45:24.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T19:45:24.971 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-11T19:45:24.971 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T19:45:25.003 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T19:45:25.072 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-11T19:45:25.072 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:45:25.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T19:45:25.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T19:45:25.293 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-11T19:45:25.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T19:45:25.321 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T19:45:25.387 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-11T19:45:25.387 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T19:45:25.418 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T19:45:25.487 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-11T19:45:25.488 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:45:25.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T19:45:25.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T19:45:25.703 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-11T19:45:25.703 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T19:45:25.730 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T19:45:25.796 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-11T19:45:25.796 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T19:45:25.827 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T19:45:25.896 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T19:45:25.994 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-11T19:45:25.995 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T19:45:25.996 INFO:tasks.cephadm:Cluster fsid is e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:45:25.996 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T19:45:25.996 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.97', 'mon.b': '172.21.15.114', 'mon.c': '172.21.15.161'} 2024-03-11T19:45:25.996 INFO:tasks.cephadm:First mon is mon.a on smithi097 2024-03-11T19:45:25.996 INFO:tasks.cephadm:First mgr is x 2024-03-11T19:45:25.996 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T19:45:25.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo hostname $(hostname -s) 2024-03-11T19:45:26.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-03-11T19:45:26.116 DEBUG:teuthology.orchestra.run.smithi161:> sudo hostname $(hostname -s) 2024-03-11T19:45:26.147 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T19:45:26.147 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T19:45:26.285 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77431/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:08:19.449750', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77429/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:06:59.696700', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-11T19:45:26.520 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 94a539707733325322964c6a187c6a1b41be1daf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T19:45:26.521 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm 2024-03-11T19:45:26.521 DEBUG:teuthology.orchestra.run.smithi097:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T19:45:27.150 INFO:teuthology.orchestra.run.smithi097.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 19:45 /home/ubuntu/cephtest/cephadm 2024-03-11T19:45:27.151 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T19:45:27.701 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 19:45 /home/ubuntu/cephtest/cephadm 2024-03-11T19:45:27.702 DEBUG:teuthology.orchestra.run.smithi161:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T19:45:28.135 INFO:teuthology.orchestra.run.smithi161.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 19:45 /home/ubuntu/cephtest/cephadm 2024-03-11T19:45:28.136 DEBUG:teuthology.orchestra.run.smithi097:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T19:45:28.195 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T19:45:28.214 DEBUG:teuthology.orchestra.run.smithi161:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T19:45:28.248 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on all hosts... 2024-03-11T19:45:28.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T19:45:28.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T19:45:28.258 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T19:45:29.108 INFO:teuthology.orchestra.run.smithi161.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T19:45:29.160 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T19:45:29.475 INFO:teuthology.orchestra.run.smithi097.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T19:46:03.837 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-11T19:46:03.837 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T19:46:03.837 INFO:teuthology.orchestra.run.smithi161.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T19:46:03.837 INFO:teuthology.orchestra.run.smithi161.stdout: "repo_digests": [ 2024-03-11T19:46:03.838 INFO:teuthology.orchestra.run.smithi161.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T19:46:03.838 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-03-11T19:46:03.838 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-11T19:46:09.802 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-11T19:46:09.802 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T19:46:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T19:46:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-03-11T19:46:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T19:46:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-03-11T19:46:09.803 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-11T19:46:12.104 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T19:46:12.104 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T19:46:12.104 INFO:teuthology.orchestra.run.smithi097.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T19:46:12.104 INFO:teuthology.orchestra.run.smithi097.stdout: "repo_digests": [ 2024-03-11T19:46:12.104 INFO:teuthology.orchestra.run.smithi097.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T19:46:12.105 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T19:46:12.105 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T19:46:12.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph 2024-03-11T19:46:12.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-03-11T19:46:12.287 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph 2024-03-11T19:46:12.323 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /etc/ceph 2024-03-11T19:46:12.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-03-11T19:46:12.451 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 777 /etc/ceph 2024-03-11T19:46:12.485 INFO:tasks.cephadm:Writing seed config... 2024-03-11T19:46:12.486 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-11T19:46:12.486 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-11T19:46:12.486 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-11T19:46:12.486 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-11T19:46:12.486 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-11T19:46:12.487 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-11T19:46:12.487 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-11T19:46:12.487 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-11T19:46:12.487 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-11T19:46:12.487 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-11T19:46:12.487 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-11T19:46:12.487 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-11T19:46:12.487 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-11T19:46:12.487 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-11T19:46:12.487 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-11T19:46:12.487 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-11T19:46:12.487 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-11T19:46:12.487 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-11T19:46:12.488 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-11T19:46:12.488 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-11T19:46:12.488 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-11T19:46:12.488 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-11T19:46:12.488 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T19:46:12.488 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T19:46:12.488 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T19:46:12.488 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T19:46:12.488 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T19:46:12.488 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-11T19:46:12.488 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-11T19:46:12.488 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-11T19:46:12.488 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-11T19:46:12.489 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-11T19:46:12.489 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T19:46:12.489 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T19:46:12.489 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-11T19:46:12.489 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-11T19:46:12.489 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-11T19:46:12.489 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-11T19:46:12.489 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-11T19:46:12.489 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-11T19:46:12.489 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-11T19:46:12.489 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T19:46:12.490 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-11T19:46:12.490 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:46:12.490 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T19:46:12.548 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e81ca0d2-dfdf-11ee-95c8-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-11T19:46:12.548 DEBUG:teuthology.orchestra.run.smithi097:mon.a> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T19:46:12.595 DEBUG:teuthology.orchestra.run.smithi097:mgr.x> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T19:46:12.639 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T19:46:12.640 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf -v bootstrap --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.97 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T19:46:12.911 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T19:46:12.911 INFO:teuthology.orchestra.run.smithi097.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf', '-v', 'bootstrap', '--fsid', 'e81ca0d2-dfdf-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.97', '--skip-admin-label'] 2024-03-11T19:46:12.964 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T19:46:12.965 INFO:teuthology.orchestra.run.smithi097.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T19:46:12.965 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying podman|docker is present... 2024-03-11T19:46:13.014 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T19:46:13.014 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying lvm2 is present... 2024-03-11T19:46:13.014 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying time synchronization is in place... 2024-03-11T19:46:13.030 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T19:46:13.030 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T19:46:13.046 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T19:46:13.047 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout inactive 2024-03-11T19:46:13.068 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout enabled 2024-03-11T19:46:13.086 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout active 2024-03-11T19:46:13.086 INFO:teuthology.orchestra.run.smithi097.stdout:Unit chronyd.service is enabled and running 2024-03-11T19:46:13.086 INFO:teuthology.orchestra.run.smithi097.stdout:Repeating the final host check... 2024-03-11T19:46:13.139 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T19:46:13.140 INFO:teuthology.orchestra.run.smithi097.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T19:46:13.140 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl is present 2024-03-11T19:46:13.140 INFO:teuthology.orchestra.run.smithi097.stdout:lvcreate is present 2024-03-11T19:46:13.155 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T19:46:13.155 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T19:46:13.170 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T19:46:13.171 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout inactive 2024-03-11T19:46:13.194 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout enabled 2024-03-11T19:46:13.214 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout active 2024-03-11T19:46:13.215 INFO:teuthology.orchestra.run.smithi097.stdout:Unit chronyd.service is enabled and running 2024-03-11T19:46:13.215 INFO:teuthology.orchestra.run.smithi097.stdout:Host looks OK 2024-03-11T19:46:13.215 INFO:teuthology.orchestra.run.smithi097.stdout:Cluster fsid: e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:46:13.216 INFO:teuthology.orchestra.run.smithi097.stdout:Acquiring lock 139990094516176 on /run/cephadm/e81ca0d2-dfdf-11ee-95c8-87774f69a715.lock 2024-03-11T19:46:13.216 INFO:teuthology.orchestra.run.smithi097.stdout:Lock 139990094516176 acquired on /run/cephadm/e81ca0d2-dfdf-11ee-95c8-87774f69a715.lock 2024-03-11T19:46:13.217 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying IP 172.21.15.97 port 3300 ... 2024-03-11T19:46:13.218 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying IP 172.21.15.97 port 6789 ... 2024-03-11T19:46:13.219 INFO:teuthology.orchestra.run.smithi097.stdout:Base mon IP(s) is [172.21.15.97:3300, 172.21.15.97:6789], mon addrv is [v2:172.21.15.97:3300,v1:172.21.15.97:6789] 2024-03-11T19:46:13.233 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.97 metric 100 2024-03-11T19:46:13.233 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.97 metric 100 2024-03-11T19:46:13.245 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T19:46:13.246 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-11T19:46:13.246 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1793sec hoplimit 64 pref medium 2024-03-11T19:46:13.257 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T19:46:13.257 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T19:46:13.257 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T19:46:13.258 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-11T19:46:13.258 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d805/64 scope link 2024-03-11T19:46:13.258 INFO:teuthology.orchestra.run.smithi097.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T19:46:13.260 INFO:teuthology.orchestra.run.smithi097.stdout:Mon IP `172.21.15.97` is in CIDR network `172.21.0.0/20` 2024-03-11T19:46:13.260 INFO:teuthology.orchestra.run.smithi097.stdout:Mon IP `172.21.15.97` is in CIDR network `172.21.0.0/20` 2024-03-11T19:46:13.260 INFO:teuthology.orchestra.run.smithi097.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T19:46:13.263 INFO:teuthology.orchestra.run.smithi097.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T19:46:13.265 INFO:teuthology.orchestra.run.smithi097.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T19:46:13.934 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stdout a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T19:46:13.935 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T19:46:13.935 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T19:46:13.935 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stderr Copying blob sha256:0b3c7bee4e66dab4b5376aaf0baf8ba5fc1c3e66c27dc601157782d617bd9dc6 2024-03-11T19:46:13.935 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T19:46:13.935 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stderr Copying config sha256:a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T19:46:13.935 INFO:teuthology.orchestra.run.smithi097.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T19:46:15.200 INFO:teuthology.orchestra.run.smithi097.stdout:ceph: stdout ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T19:46:15.203 INFO:teuthology.orchestra.run.smithi097.stdout:Ceph version: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T19:46:15.204 INFO:teuthology.orchestra.run.smithi097.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T19:46:26.783 INFO:teuthology.orchestra.run.smithi097.stdout:stat: stdout 167 167 2024-03-11T19:46:26.783 INFO:teuthology.orchestra.run.smithi097.stdout:Creating initial keys... 2024-03-11T19:46:28.063 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph-authtool: stdout AQCTX+9lkahMExAALpkGfpB+LT+7D5W6ItNg7w== 2024-03-11T19:46:29.311 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph-authtool: stdout AQCUX+9liOd8IRAABBGbTd8mou7OcQMifOT3SA== 2024-03-11T19:46:30.495 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph-authtool: stdout AQCVX+9laegXMRAAxhSTjUWu6umklopciQ1woA== 2024-03-11T19:46:30.497 INFO:teuthology.orchestra.run.smithi097.stdout:Creating initial monmap... 2024-03-11T19:46:32.039 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T19:46:32.039 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T19:46:32.040 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:46:32.040 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T19:46:32.040 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool for a [v2:172.21.15.97:3300,v1:172.21.15.97:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T19:46:32.040 INFO:teuthology.orchestra.run.smithi097.stdout:setting min_mon_release = quincy 2024-03-11T19:46:32.040 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/monmaptool: set fsid to e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:46:32.040 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T19:46:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:46:32.040 INFO:teuthology.orchestra.run.smithi097.stdout:Creating mon... 2024-03-11T19:46:33.291 INFO:teuthology.orchestra.run.smithi097.stdout:create mon.a on 2024-03-11T19:46:33.669 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-11T19:46:33.914 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T19:46:34.194 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715.target. 2024-03-11T19:46:34.194 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715.target. 2024-03-11T19:46:34.564 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.a 2024-03-11T19:46:34.564 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T19:46:34.850 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715.target.wants/ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@.service. 2024-03-11T19:46:35.564 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:35 smithi097 podman[31506]: 2024-03-11 19:46:35.496836993 +0000 UTC m=+0.379389042 container start 3a81c4727f361fd7a7500acd3fd8cf89284e3685a7f6861daefef411161b96d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-11T19:46:35.564 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:35 smithi097 bash[31506]: 3a81c4727f361fd7a7500acd3fd8cf89284e3685a7f6861daefef411161b96d2 2024-03-11T19:46:35.593 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T19:46:35.593 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout disabled 2024-03-11T19:46:35.608 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T19:46:35.608 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout inactive 2024-03-11T19:46:35.608 INFO:teuthology.orchestra.run.smithi097.stdout:firewalld.service is not enabled 2024-03-11T19:46:35.608 INFO:teuthology.orchestra.run.smithi097.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T19:46:35.609 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for mon to start... 2024-03-11T19:46:35.609 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for mon... 2024-03-11T19:46:35.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:35 smithi097 systemd[1]: Started Ceph mon.a for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:46:35.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:35 smithi097 ceph-mon[31525]: mkfs e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:46:35.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:35 smithi097 ceph-mon[31525]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T19:46:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:36 smithi097 ceph-mon[31525]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T19:46:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:36 smithi097 ceph-mon[31525]: monmap e1: 1 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:46:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:36 smithi097 ceph-mon[31525]: fsmap 2024-03-11T19:46:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:36 smithi097 ceph-mon[31525]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T19:46:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:36 smithi097 ceph-mon[31525]: mgrmap e1: no daemons active 2024-03-11T19:46:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:36 smithi097 ceph-mon[31525]: from='client.? 172.21.15.97:0/303946005' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T19:46:36.955 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T19:46:36.956 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout id: e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:46:36.956 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T19:46:36.956 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-03-11T19:46:36.960 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout services: 2024-03-11T19:46:36.960 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.753778s) 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout data: 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.235+0000 7fc14e400700 1 Processor -- start 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.235+0000 7fc14e400700 1 -- start start 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.235+0000 7fc14e400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 0x7fc148109bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.235+0000 7fc14e400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14810a180 con 0x7fc148109790 2024-03-11T19:46:36.961 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.236+0000 7fc147400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 0x7fc148109bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.236+0000 7fc147400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 0x7fc148109bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35222/0 (socket says 172.21.15.97:35222) 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.236+0000 7fc147400700 1 -- 172.21.15.97:0/2284157764 learned_addr learned my addr 172.21.15.97:0/2284157764 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.237+0000 7fc147400700 1 -- 172.21.15.97:0/2284157764 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14810a940 con 0x7fc148109790 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.237+0000 7fc147400700 1 --2- 172.21.15.97:0/2284157764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 0x7fc148109bb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc1380098a0 tx=0x7fc138009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436ddd63360cf361 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.238+0000 7fc146a00700 1 -- 172.21.15.97:0/2284157764 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc138004160 con 0x7fc148109790 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.238+0000 7fc146a00700 1 -- 172.21.15.97:0/2284157764 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1380042c0 con 0x7fc148109790 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.238+0000 7fc146a00700 1 -- 172.21.15.97:0/2284157764 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1380045a0 con 0x7fc148109790 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.238+0000 7fc14e400700 1 -- 172.21.15.97:0/2284157764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 msgr2=0x7fc148109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.238+0000 7fc14e400700 1 --2- 172.21.15.97:0/2284157764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 0x7fc148109bb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc1380098a0 tx=0x7fc138009bb0 comp rx=0 tx=0).stop 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.238+0000 7fc14e400700 1 -- 172.21.15.97:0/2284157764 shutdown_connections 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.238+0000 7fc14e400700 1 --2- 172.21.15.97:0/2284157764 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 0x7fc148109bb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.238+0000 7fc14e400700 1 -- 172.21.15.97:0/2284157764 >> 172.21.15.97:0/2284157764 conn(0x7fc14807cdf0 msgr2=0x7fc1481074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.239+0000 7fc14e400700 1 -- 172.21.15.97:0/2284157764 shutdown_connections 2024-03-11T19:46:36.962 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.239+0000 7fc14e400700 1 -- 172.21.15.97:0/2284157764 wait complete. 2024-03-11T19:46:36.963 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.240+0000 7fc14e400700 1 Processor -- start 2024-03-11T19:46:36.963 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.240+0000 7fc14e400700 1 -- start start 2024-03-11T19:46:36.963 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.240+0000 7fc14e400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 0x7fc14819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:36.963 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.240+0000 7fc14e400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14819d110 con 0x7fc148109790 2024-03-11T19:46:36.963 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.241+0000 7fc147400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 0x7fc14819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:36.963 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.241+0000 7fc147400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 0x7fc14819cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35228/0 (socket says 172.21.15.97:35228) 2024-03-11T19:46:36.963 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.241+0000 7fc147400700 1 -- 172.21.15.97:0/303946005 learned_addr learned my addr 172.21.15.97:0/303946005 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:36.963 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.241+0000 7fc147400700 1 -- 172.21.15.97:0/303946005 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc138009550 con 0x7fc148109790 2024-03-11T19:46:36.965 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.241+0000 7fc147400700 1 --2- 172.21.15.97:0/303946005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 0x7fc14819cbd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc138004000 tx=0x7fc138004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:36.965 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.242+0000 7fc144c00700 1 -- 172.21.15.97:0/303946005 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc138011630 con 0x7fc148109790 2024-03-11T19:46:36.965 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.242+0000 7fc144c00700 1 -- 172.21.15.97:0/303946005 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc138011790 con 0x7fc148109790 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.242+0000 7fc14e400700 1 -- 172.21.15.97:0/303946005 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14819d310 con 0x7fc148109790 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.242+0000 7fc144c00700 1 -- 172.21.15.97:0/303946005 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc138011a30 con 0x7fc148109790 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.242+0000 7fc14e400700 1 -- 172.21.15.97:0/303946005 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14819d7b0 con 0x7fc148109790 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.243+0000 7fc144c00700 1 -- 172.21.15.97:0/303946005 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc138022070 con 0x7fc148109790 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.243+0000 7fc144c00700 1 -- 172.21.15.97:0/303946005 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc13801c450 con 0x7fc148109790 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.243+0000 7fc14e400700 1 -- 172.21.15.97:0/303946005 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc124005360 con 0x7fc148109790 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.246+0000 7fc144c00700 1 -- 172.21.15.97:0/303946005 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc138011b90 con 0x7fc148109790 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.324+0000 7fc14e400700 1 -- 172.21.15.97:0/303946005 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fc1240058e0 con 0x7fc148109790 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.325+0000 7fc144c00700 1 -- 172.21.15.97:0/303946005 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fc138011d30 con 0x7fc148109790 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.326+0000 7fc14e400700 1 -- 172.21.15.97:0/303946005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 msgr2=0x7fc14819cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.326+0000 7fc14e400700 1 --2- 172.21.15.97:0/303946005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 0x7fc14819cbd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc138004000 tx=0x7fc138004750 comp rx=0 tx=0).stop 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.326+0000 7fc14e400700 1 -- 172.21.15.97:0/303946005 shutdown_connections 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.326+0000 7fc14e400700 1 --2- 172.21.15.97:0/303946005 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc148109790 0x7fc14819cbd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:36.966 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.326+0000 7fc14e400700 1 -- 172.21.15.97:0/303946005 >> 172.21.15.97:0/303946005 conn(0x7fc14807cdf0 msgr2=0x7fc148193a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:36.967 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.326+0000 7fc14e400700 1 -- 172.21.15.97:0/303946005 shutdown_connections 2024-03-11T19:46:36.967 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:36.326+0000 7fc14e400700 1 -- 172.21.15.97:0/303946005 wait complete. 2024-03-11T19:46:36.967 INFO:teuthology.orchestra.run.smithi097.stdout:mon is available 2024-03-11T19:46:36.967 INFO:teuthology.orchestra.run.smithi097.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T19:46:37.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:37 smithi097 ceph-mon[31525]: from='client.? 172.21.15.97:0/3830687891' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T19:46:37.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:37 smithi097 ceph-mon[31525]: from='client.? 172.21.15.97:0/3830687891' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T19:46:38.403 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-03-11T19:46:38.403 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T19:46:38.403 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout fsid = e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:46:38.403 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.97:3300,v1:172.21.15.97:6789] 2024-03-11T19:46:38.403 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T19:46:38.403 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T19:46:38.403 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T19:46:38.403 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.542+0000 7fe411000700 1 Processor -- start 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.543+0000 7fe411000700 1 -- start start 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.543+0000 7fe411000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 0x7fe40c10e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:38.404 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.543+0000 7fe411000700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c10ea10 con 0x7fe40c10e020 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.543+0000 7fe409e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 0x7fe40c10e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.543+0000 7fe409e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 0x7fe40c10e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35234/0 (socket says 172.21.15.97:35234) 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.543+0000 7fe409e00700 1 -- 172.21.15.97:0/269967682 learned_addr learned my addr 172.21.15.97:0/269967682 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.544+0000 7fe409e00700 1 -- 172.21.15.97:0/269967682 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c10eb50 con 0x7fe40c10e020 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.544+0000 7fe409e00700 1 --2- 172.21.15.97:0/269967682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 0x7fe40c10e440 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0098a0 tx=0x7fe3fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f65535eccf50e80 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.544+0000 7fe409400700 1 -- 172.21.15.97:0/269967682 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3fc004160 con 0x7fe40c10e020 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.544+0000 7fe409400700 1 -- 172.21.15.97:0/269967682 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc0042c0 con 0x7fe40c10e020 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.544+0000 7fe409400700 1 -- 172.21.15.97:0/269967682 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3fc0045a0 con 0x7fe40c10e020 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.545+0000 7fe411000700 1 -- 172.21.15.97:0/269967682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 msgr2=0x7fe40c10e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.545+0000 7fe411000700 1 --2- 172.21.15.97:0/269967682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 0x7fe40c10e440 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0098a0 tx=0x7fe3fc009bb0 comp rx=0 tx=0).stop 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.545+0000 7fe411000700 1 -- 172.21.15.97:0/269967682 shutdown_connections 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.545+0000 7fe411000700 1 --2- 172.21.15.97:0/269967682 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 0x7fe40c10e440 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.545+0000 7fe411000700 1 -- 172.21.15.97:0/269967682 >> 172.21.15.97:0/269967682 conn(0x7fe40c07ce20 msgr2=0x7fe40c07f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:38.405 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.545+0000 7fe411000700 1 -- 172.21.15.97:0/269967682 shutdown_connections 2024-03-11T19:46:38.406 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.545+0000 7fe411000700 1 -- 172.21.15.97:0/269967682 wait complete. 2024-03-11T19:46:38.406 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.546+0000 7fe411000700 1 Processor -- start 2024-03-11T19:46:38.406 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.546+0000 7fe411000700 1 -- start start 2024-03-11T19:46:38.406 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.547+0000 7fe411000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 0x7fe40c1a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:38.406 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.547+0000 7fe411000700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c1a14c0 con 0x7fe40c10e020 2024-03-11T19:46:38.406 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.547+0000 7fe409e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 0x7fe40c1a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:38.406 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.547+0000 7fe409e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 0x7fe40c1a0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35244/0 (socket says 172.21.15.97:35244) 2024-03-11T19:46:38.406 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.547+0000 7fe409e00700 1 -- 172.21.15.97:0/3830687891 learned_addr learned my addr 172.21.15.97:0/3830687891 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:38.406 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.547+0000 7fe409e00700 1 -- 172.21.15.97:0/3830687891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc009550 con 0x7fe40c10e020 2024-03-11T19:46:38.406 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.547+0000 7fe409e00700 1 --2- 172.21.15.97:0/3830687891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 0x7fe40c1a0f80 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe40c068670 tx=0x7fe3fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:38.406 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.548+0000 7fe402a00700 1 -- 172.21.15.97:0/3830687891 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3fc011630 con 0x7fe40c10e020 2024-03-11T19:46:38.406 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.548+0000 7fe402a00700 1 -- 172.21.15.97:0/3830687891 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc011790 con 0x7fe40c10e020 2024-03-11T19:46:38.406 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.548+0000 7fe411000700 1 -- 172.21.15.97:0/3830687891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40c1a16c0 con 0x7fe40c10e020 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.548+0000 7fe402a00700 1 -- 172.21.15.97:0/3830687891 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3fc011a30 con 0x7fe40c10e020 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.548+0000 7fe411000700 1 -- 172.21.15.97:0/3830687891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40c1a1b60 con 0x7fe40c10e020 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.548+0000 7fe402a00700 1 -- 172.21.15.97:0/3830687891 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe3fc022070 con 0x7fe40c10e020 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.548+0000 7fe402a00700 1 -- 172.21.15.97:0/3830687891 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe3fc01c450 con 0x7fe40c10e020 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.549+0000 7fe411000700 1 -- 172.21.15.97:0/3830687891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40c19ac70 con 0x7fe40c10e020 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.552+0000 7fe402a00700 1 -- 172.21.15.97:0/3830687891 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe40c10eb50 con 0x7fe40c10e020 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.627+0000 7fe411000700 1 -- 172.21.15.97:0/3830687891 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fe40c10ef70 con 0x7fe40c10e020 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.632+0000 7fe402a00700 1 -- 172.21.15.97:0/3830687891 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fe3fc011c40 con 0x7fe40c10e020 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.632+0000 7fe402a00700 1 -- 172.21.15.97:0/3830687891 <== mon.0 v2:172.21.15.97:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3fc011de0 con 0x7fe40c10e020 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.633+0000 7fe411000700 1 -- 172.21.15.97:0/3830687891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 msgr2=0x7fe40c1a0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.633+0000 7fe411000700 1 --2- 172.21.15.97:0/3830687891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 0x7fe40c1a0f80 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe40c068670 tx=0x7fe3fc004750 comp rx=0 tx=0).stop 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.633+0000 7fe411000700 1 -- 172.21.15.97:0/3830687891 shutdown_connections 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.633+0000 7fe411000700 1 --2- 172.21.15.97:0/3830687891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe40c10e020 0x7fe40c1a0f80 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:38.407 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.633+0000 7fe411000700 1 -- 172.21.15.97:0/3830687891 >> 172.21.15.97:0/3830687891 conn(0x7fe40c07ce20 msgr2=0x7fe40c197db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:38.408 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.634+0000 7fe411000700 1 -- 172.21.15.97:0/3830687891 shutdown_connections 2024-03-11T19:46:38.408 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:37.634+0000 7fe411000700 1 -- 172.21.15.97:0/3830687891 wait complete. 2024-03-11T19:46:38.408 INFO:teuthology.orchestra.run.smithi097.stdout:Generating new minimal ceph.conf... 2024-03-11T19:46:39.402 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:39 smithi097 ceph-mon[31525]: from='client.? 172.21.15.97:0/3451655475' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:46:39.727 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.985+0000 7f8478e00700 1 Processor -- start 2024-03-11T19:46:39.732 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.985+0000 7f8478e00700 1 -- start start 2024-03-11T19:46:39.732 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.986+0000 7f8478e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 0x7f847407b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:39.732 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.986+0000 7f8478e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847407b850 con 0x7f847407ce10 2024-03-11T19:46:39.732 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.986+0000 7f8471e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 0x7f847407b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:39.732 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.986+0000 7f8471e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 0x7f847407b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35248/0 (socket says 172.21.15.97:35248) 2024-03-11T19:46:39.732 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.986+0000 7f8471e00700 1 -- 172.21.15.97:0/165664449 learned_addr learned my addr 172.21.15.97:0/165664449 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:39.732 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.987+0000 7f8471e00700 1 -- 172.21.15.97:0/165664449 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847407b990 con 0x7f847407ce10 2024-03-11T19:46:39.732 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.987+0000 7f8471e00700 1 --2- 172.21.15.97:0/165664449 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 0x7f847407b310 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8464009e80 tx=0x7f846400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=996f420ac8f7211c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:39.732 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.987+0000 7f8470a00700 1 -- 172.21.15.97:0/165664449 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f846400e070 con 0x7f847407ce10 2024-03-11T19:46:39.732 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.987+0000 7f8470a00700 1 -- 172.21.15.97:0/165664449 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f846400b8c0 con 0x7f847407ce10 2024-03-11T19:46:39.732 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.987+0000 7f8470a00700 1 -- 172.21.15.97:0/165664449 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8464004550 con 0x7f847407ce10 2024-03-11T19:46:39.732 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.988+0000 7f8478e00700 1 -- 172.21.15.97:0/165664449 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 msgr2=0x7f847407b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.988+0000 7f8478e00700 1 --2- 172.21.15.97:0/165664449 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 0x7f847407b310 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8464009e80 tx=0x7f846400b330 comp rx=0 tx=0).stop 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.988+0000 7f8478e00700 1 -- 172.21.15.97:0/165664449 shutdown_connections 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.988+0000 7f8478e00700 1 --2- 172.21.15.97:0/165664449 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 0x7f847407b310 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.988+0000 7f8478e00700 1 -- 172.21.15.97:0/165664449 >> 172.21.15.97:0/165664449 conn(0x7f8474102bd0 msgr2=0x7f8474104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.988+0000 7f8478e00700 1 -- 172.21.15.97:0/165664449 shutdown_connections 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.988+0000 7f8478e00700 1 -- 172.21.15.97:0/165664449 wait complete. 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.989+0000 7f8478e00700 1 Processor -- start 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.989+0000 7f8478e00700 1 -- start start 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.990+0000 7f8478e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 0x7f847419d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.990+0000 7f8478e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847419d930 con 0x7f847407ce10 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.990+0000 7f8471e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 0x7f847419d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.990+0000 7f8471e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 0x7f847419d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35264/0 (socket says 172.21.15.97:35264) 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.990+0000 7f8471e00700 1 -- 172.21.15.97:0/3451655475 learned_addr learned my addr 172.21.15.97:0/3451655475 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.990+0000 7f8471e00700 1 -- 172.21.15.97:0/3451655475 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8464009b30 con 0x7f847407ce10 2024-03-11T19:46:39.733 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.991+0000 7f8471e00700 1 --2- 172.21.15.97:0/3451655475 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 0x7f847419d3f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8464004070 tx=0x7f8464003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.991+0000 7f846aa00700 1 -- 172.21.15.97:0/3451655475 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f846400e040 con 0x7f847407ce10 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.991+0000 7f846aa00700 1 -- 172.21.15.97:0/3451655475 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f846400bdf0 con 0x7f847407ce10 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.991+0000 7f846aa00700 1 -- 172.21.15.97:0/3451655475 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f846401c840 con 0x7f847407ce10 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.991+0000 7f8478e00700 1 -- 172.21.15.97:0/3451655475 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847419db30 con 0x7f847407ce10 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.991+0000 7f8478e00700 1 -- 172.21.15.97:0/3451655475 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847419dfb0 con 0x7f847407ce10 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.991+0000 7f846aa00700 1 -- 172.21.15.97:0/3451655475 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f846401cab0 con 0x7f847407ce10 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.991+0000 7f846aa00700 1 -- 172.21.15.97:0/3451655475 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f846401b850 con 0x7f847407ce10 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.992+0000 7f8478e00700 1 -- 172.21.15.97:0/3451655475 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847410d8c0 con 0x7f847407ce10 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:38.995+0000 7f846aa00700 1 -- 172.21.15.97:0/3451655475 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f846402a060 con 0x7f847407ce10 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:39.071+0000 7f8478e00700 1 -- 172.21.15.97:0/3451655475 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f847410dad0 con 0x7f847407ce10 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:39.071+0000 7f846aa00700 1 -- 172.21.15.97:0/3451655475 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f8464020070 con 0x7f847407ce10 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:39.073+0000 7f8478e00700 1 -- 172.21.15.97:0/3451655475 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 msgr2=0x7f847419d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:39.073+0000 7f8478e00700 1 --2- 172.21.15.97:0/3451655475 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 0x7f847419d3f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8464004070 tx=0x7f8464003d00 comp rx=0 tx=0).stop 2024-03-11T19:46:39.734 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:39.073+0000 7f8478e00700 1 -- 172.21.15.97:0/3451655475 shutdown_connections 2024-03-11T19:46:39.735 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:39.073+0000 7f8478e00700 1 --2- 172.21.15.97:0/3451655475 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f847407ce10 0x7f847419d3f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:39.735 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:39.073+0000 7f8478e00700 1 -- 172.21.15.97:0/3451655475 >> 172.21.15.97:0/3451655475 conn(0x7f8474102bd0 msgr2=0x7f8474103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:39.735 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:39.074+0000 7f8478e00700 1 -- 172.21.15.97:0/3451655475 shutdown_connections 2024-03-11T19:46:39.735 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:39.074+0000 7f8478e00700 1 -- 172.21.15.97:0/3451655475 wait complete. 2024-03-11T19:46:39.735 INFO:teuthology.orchestra.run.smithi097.stdout:Restarting the monitor... 2024-03-11T19:46:40.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:39 smithi097 systemd[1]: Stopping Ceph mon.a for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T19:46:40.605 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:40 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a[31521]: 2024-03-11T19:46:40.229+0000 7ff467200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T19:46:40.605 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:40 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a[31521]: 2024-03-11T19:46:40.229+0000 7ff467200700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T19:46:40.605 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:40 smithi097 podman[31766]: 2024-03-11 19:46:40.302867382 +0000 UTC m=+0.232529368 container died 3a81c4727f361fd7a7500acd3fd8cf89284e3685a7f6861daefef411161b96d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T19:46:40.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:40 smithi097 podman[31766]: 2024-03-11 19:46:40.714881924 +0000 UTC m=+0.644543943 container cleanup 3a81c4727f361fd7a7500acd3fd8cf89284e3685a7f6861daefef411161b96d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:46:40.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:40 smithi097 bash[31766]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a 2024-03-11T19:46:41.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:41 smithi097 podman[31780]: 2024-03-11 19:46:41.048954177 +0000 UTC m=+0.729918536 container remove 3a81c4727f361fd7a7500acd3fd8cf89284e3685a7f6861daefef411161b96d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-94a5397, ceph=True, org.label-schema.vendor=CentOS) 2024-03-11T19:46:41.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:41 smithi097 systemd[1]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T19:46:41.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:41 smithi097 systemd[1]: Stopped Ceph mon.a for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:46:41.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:41 smithi097 systemd[1]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.a.service: Consumed 1.889s CPU time. 2024-03-11T19:46:41.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:41 smithi097 systemd[1]: Starting Ceph mon.a for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T19:46:42.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:41 smithi097 podman[31870]: 2024-03-11 19:46:41.902917532 +0000 UTC m=+0.041161351 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T19:46:42.743 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 podman[31870]: 2024-03-11 19:46:42.466569059 +0000 UTC m=+0.604812932 container create 3c2b428eeaee03d17e0bfd22c392aa61620169f3d699922426710cba08a3a226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, ceph=True) 2024-03-11T19:46:42.743 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 podman[31870]: 2024-03-11 19:46:42.686002743 +0000 UTC m=+0.824246576 container init 3c2b428eeaee03d17e0bfd22c392aa61620169f3d699922426710cba08a3a226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:46:42.743 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 podman[31870]: 2024-03-11 19:46:42.697189893 +0000 UTC m=+0.835433722 container start 3c2b428eeaee03d17e0bfd22c392aa61620169f3d699922426710cba08a3a226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-94a5397) 2024-03-11T19:46:42.828 INFO:teuthology.orchestra.run.smithi097.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T19:46:43.088 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T19:46:43.088 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-11T19:46:43.088 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: pidfile_write: ignore empty --pid-file 2024-03-11T19:46:43.088 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: load: jerasure load: lrc 2024-03-11T19:46:43.088 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: RocksDB version: 7.9.2 2024-03-11T19:46:43.088 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Git sha 0 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: DB SUMMARY 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: DB Session ID: 9PMN9WOHJ8ZDL8WAYDG3 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: CURRENT file: CURRENT 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99150 ; 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.error_if_exists: 0 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.create_if_missing: 0 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.paranoid_checks: 1 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T19:46:43.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.env: 0x55dc53f13ae0 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.info_log: 0x55dc55e8b900 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.statistics: (nil) 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.use_fsync: 0 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_log_file_size: 0 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.allow_fallocate: 1 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T19:46:43.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T19:46:43.091 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.use_direct_reads: 0 2024-03-11T19:46:43.091 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T19:46:43.091 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T19:46:43.091 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.db_log_dir: 2024-03-11T19:46:43.091 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.wal_dir: 2024-03-11T19:46:43.091 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T19:46:43.091 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T19:46:43.092 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T19:46:43.092 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T19:46:43.092 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T19:46:43.092 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T19:46:43.092 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T19:46:43.092 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T19:46:43.092 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.write_buffer_manager: 0x55dc55e5f400 2024-03-11T19:46:43.092 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T19:46:43.092 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T19:46:43.092 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T19:46:43.092 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T19:46:43.092 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T19:46:43.092 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.unordered_write: 0 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.row_cache: None 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.wal_filter: None 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.two_write_queues: 0 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.wal_compression: 0 2024-03-11T19:46:43.093 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.atomic_flush: 0 2024-03-11T19:46:43.094 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T19:46:43.094 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T19:46:43.094 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T19:46:43.094 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.log_readahead_size: 0 2024-03-11T19:46:43.094 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T19:46:43.094 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T19:46:43.094 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T19:46:43.094 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T19:46:43.094 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T19:46:43.094 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T19:46:43.094 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T19:46:43.095 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_background_jobs: 2 2024-03-11T19:46:43.095 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_background_compactions: -1 2024-03-11T19:46:43.095 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_subcompactions: 1 2024-03-11T19:46:43.095 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T19:46:43.095 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T19:46:43.095 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T19:46:43.095 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T19:46:43.095 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T19:46:43.095 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T19:46:43.095 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T19:46:43.095 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T19:46:43.095 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_open_files: -1 2024-03-11T19:46:43.096 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T19:46:43.096 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T19:46:43.096 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T19:46:43.096 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_background_flushes: -1 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Compression algorithms supported: 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: kZSTD supported: 0 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: kXpressCompression supported: 0 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: kZlibCompression supported: 1 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: kSnappyCompression supported: 1 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: kLZ4Compression supported: 1 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: kBZip2Compression supported: 0 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T19:46:43.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.merge_operator: 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compaction_filter: None 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dc55e8ba00) 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout: cache_index_and_filter_blocks: 1 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout: pin_top_level_index_and_filter: 1 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout: index_type: 0 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout: data_block_index_type: 0 2024-03-11T19:46:43.098 INFO:journalctl@ceph.mon.a.smithi097.stdout: index_shortening: 1 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: checksum: 4 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: no_block_cache: 0 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache: 0x55dc55e8f090 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache_name: BinnedLRUCache 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache_options: 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: capacity : 536870912 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: num_shard_bits : 4 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: strict_capacity_limit : 0 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: high_pri_pool_ratio: 0.000 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache_compressed: (nil) 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: persistent_cache: (nil) 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_size: 4096 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_size_deviation: 10 2024-03-11T19:46:43.099 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_restart_interval: 16 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: index_block_restart_interval: 1 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: metadata_block_size: 4096 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: partition_filters: 0 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: use_delta_encoding: 1 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: filter_policy: bloomfilter 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: whole_key_filtering: 1 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: verify_compression: 0 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: read_amp_bytes_per_bit: 0 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: format_version: 5 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: enable_index_compression: 1 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_align: 0 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: max_auto_readahead_size: 262144 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: prepopulate_block_cache: 0 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: initial_auto_readahead_size: 8192 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T19:46:43.100 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compression: NoCompression 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.num_levels: 7 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T19:46:43.101 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T19:46:43.102 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T19:46:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T19:46:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T19:46:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T19:46:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T19:46:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T19:46:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T19:46:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T19:46:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T19:46:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T19:46:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T19:46:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T19:46:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T19:46:43.103 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T19:46:43.104 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T19:46:43.104 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T19:46:43.104 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T19:46:43.104 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T19:46:43.104 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T19:46:43.104 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T19:46:43.104 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T19:46:43.104 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T19:46:43.104 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T19:46:43.104 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T19:46:43.104 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T19:46:43.104 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T19:46:43.104 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T19:46:43.105 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.table_properties_collectors: 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.inplace_update_support: 0 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.bloom_locality: 0 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.max_successive_merges: 0 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T19:46:43.106 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.ttl: 2592000 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.enable_blob_files: false 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.min_blob_size: 0 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T19:46:43.107 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7afefc3f-497f-4935-8787-7b890f1a3a75 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710186402750771, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710186402753779, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94424, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 92424, "index_size": 282, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13640, "raw_average_key_size": 49, "raw_value_size": 84961, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710186402, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7afefc3f-497f-4935-8787-7b890f1a3a75", "db_session_id": "9PMN9WOHJ8ZDL8WAYDG3", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710186402753980, "job": 1, "event": "recovery_finished"} 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55dc55f80000 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: DB pointer 0x55dc55f6a000 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** DB Stats ** 2024-03-11T19:46:43.108 INFO:journalctl@ceph.mon.a.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** Compaction Stats [default] ** 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: L0 2/0 94.05 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 45.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: Sum 2/0 94.05 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 45.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 45.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-03-11T19:46:43.109 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** Compaction Stats [default] ** 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 45.8 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative compaction: 0.00 GB write, 9.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval compaction: 0.00 GB write, 9.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T19:46:43.110 INFO:journalctl@ceph.mon.a.smithi097.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout: Block cache BinnedLRUCache@0x55dc55e8f090#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: starting mon.a rank 0 at public addrs [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] at bind addrs [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: mon.a@-1(???) e1 preinit fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: mon.a@-1(???).mds e1 new map 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: mon.a@-1(???).mds e1 print_map 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout: e1 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout: legacy client fscid: -1 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-03-11T19:46:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout: No filesystems configured 2024-03-11T19:46:43.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T19:46:43.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T19:46:43.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T19:46:43.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T19:46:43.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T19:46:43.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 bash[31870]: 3c2b428eeaee03d17e0bfd22c392aa61620169f3d699922426710cba08a3a226 2024-03-11T19:46:43.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 systemd[1]: Started Ceph mon.a for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:46:43.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T19:46:43.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: monmap e1: 1 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:46:43.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: fsmap 2024-03-11T19:46:43.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T19:46:43.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:42 smithi097 ceph-mon[31898]: mgrmap e1: no daemons active 2024-03-11T19:46:44.281 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.439+0000 7f309f000700 1 Processor -- start 2024-03-11T19:46:44.281 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.440+0000 7f309f000700 1 -- start start 2024-03-11T19:46:44.281 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.440+0000 7f309f000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 0x7f309810e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:44.281 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.440+0000 7f309f000700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309810ea10 con 0x7f309810e020 2024-03-11T19:46:44.281 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.441+0000 7f3097e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 0x7f309810e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:44.281 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.441+0000 7f3097e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 0x7f309810e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35268/0 (socket says 172.21.15.97:35268) 2024-03-11T19:46:44.281 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.441+0000 7f3097e00700 1 -- 172.21.15.97:0/3574324342 learned_addr learned my addr 172.21.15.97:0/3574324342 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:44.281 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.441+0000 7f3097e00700 1 -- 172.21.15.97:0/3574324342 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309810eb50 con 0x7f309810e020 2024-03-11T19:46:44.281 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.442+0000 7f3097e00700 1 --2- 172.21.15.97:0/3574324342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 0x7f309810e440 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f30880098a0 tx=0x7f3088009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e03ff2c78bbbfc54 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:44.282 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.442+0000 7f3096a00700 1 -- 172.21.15.97:0/3574324342 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3088004160 con 0x7f309810e020 2024-03-11T19:46:44.282 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.442+0000 7f3096a00700 1 -- 172.21.15.97:0/3574324342 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30880042c0 con 0x7f309810e020 2024-03-11T19:46:44.282 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.442+0000 7f3096a00700 1 -- 172.21.15.97:0/3574324342 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3088011ac0 con 0x7f309810e020 2024-03-11T19:46:44.282 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.443+0000 7f309f000700 1 -- 172.21.15.97:0/3574324342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 msgr2=0x7f309810e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:44.282 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.443+0000 7f309f000700 1 --2- 172.21.15.97:0/3574324342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 0x7f309810e440 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f30880098a0 tx=0x7f3088009bb0 comp rx=0 tx=0).stop 2024-03-11T19:46:44.282 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.443+0000 7f309f000700 1 -- 172.21.15.97:0/3574324342 shutdown_connections 2024-03-11T19:46:44.282 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.443+0000 7f309f000700 1 --2- 172.21.15.97:0/3574324342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 0x7f309810e440 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:44.285 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.443+0000 7f309f000700 1 -- 172.21.15.97:0/3574324342 >> 172.21.15.97:0/3574324342 conn(0x7f309807ce20 msgr2=0x7f309807f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:44.285 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.443+0000 7f309f000700 1 -- 172.21.15.97:0/3574324342 shutdown_connections 2024-03-11T19:46:44.285 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.443+0000 7f309f000700 1 -- 172.21.15.97:0/3574324342 wait complete. 2024-03-11T19:46:44.285 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.444+0000 7f309f000700 1 Processor -- start 2024-03-11T19:46:44.285 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.444+0000 7f309f000700 1 -- start start 2024-03-11T19:46:44.285 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.444+0000 7f309f000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 0x7f30981a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:44.285 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.444+0000 7f309f000700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30981a1d50 con 0x7f309810e020 2024-03-11T19:46:44.285 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.445+0000 7f3097e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 0x7f30981a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.445+0000 7f3097e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 0x7f30981a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:35280/0 (socket says 172.21.15.97:35280) 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.445+0000 7f3097e00700 1 -- 172.21.15.97:0/1572774730 learned_addr learned my addr 172.21.15.97:0/1572774730 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.445+0000 7f3097e00700 1 -- 172.21.15.97:0/1572774730 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3088009550 con 0x7f309810e020 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.445+0000 7f3097e00700 1 --2- 172.21.15.97:0/1572774730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 0x7f30981a1810 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3088009130 tx=0x7f3088011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.446+0000 7f3094c00700 1 -- 172.21.15.97:0/1572774730 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3088003aa0 con 0x7f309810e020 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.446+0000 7f3094c00700 1 -- 172.21.15.97:0/1572774730 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3088003c00 con 0x7f309810e020 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.446+0000 7f309f000700 1 -- 172.21.15.97:0/1572774730 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30981a1f50 con 0x7f309810e020 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.446+0000 7f3094c00700 1 -- 172.21.15.97:0/1572774730 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3088024800 con 0x7f309810e020 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.446+0000 7f309f000700 1 -- 172.21.15.97:0/1572774730 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30981a23f0 con 0x7f309810e020 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.446+0000 7f3094c00700 1 -- 172.21.15.97:0/1572774730 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f308802d3f0 con 0x7f309810e020 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.446+0000 7f3094c00700 1 -- 172.21.15.97:0/1572774730 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3088019890 con 0x7f309810e020 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.447+0000 7f309f000700 1 -- 172.21.15.97:0/1572774730 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30980632a0 con 0x7f309810e020 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.450+0000 7f3094c00700 1 -- 172.21.15.97:0/1572774730 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f309810eb50 con 0x7f309810e020 2024-03-11T19:46:44.286 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.525+0000 7f309f000700 1 -- 172.21.15.97:0/1572774730 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f309810fa60 con 0x7f309810e020 2024-03-11T19:46:44.287 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.528+0000 7f3094c00700 1 -- 172.21.15.97:0/1572774730 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3088024e50 con 0x7f309810e020 2024-03-11T19:46:44.287 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.529+0000 7f309f000700 1 -- 172.21.15.97:0/1572774730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 msgr2=0x7f30981a1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:44.287 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.529+0000 7f309f000700 1 --2- 172.21.15.97:0/1572774730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 0x7f30981a1810 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3088009130 tx=0x7f3088011b40 comp rx=0 tx=0).stop 2024-03-11T19:46:44.287 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.529+0000 7f309f000700 1 -- 172.21.15.97:0/1572774730 shutdown_connections 2024-03-11T19:46:44.287 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.529+0000 7f309f000700 1 --2- 172.21.15.97:0/1572774730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f309810e020 0x7f30981a1810 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:44.287 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.529+0000 7f309f000700 1 -- 172.21.15.97:0/1572774730 >> 172.21.15.97:0/1572774730 conn(0x7f309807ce20 msgr2=0x7f309807e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:44.287 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.529+0000 7f309f000700 1 -- 172.21.15.97:0/1572774730 shutdown_connections 2024-03-11T19:46:44.287 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:43.530+0000 7f309f000700 1 -- 172.21.15.97:0/1572774730 wait complete. 2024-03-11T19:46:44.287 INFO:teuthology.orchestra.run.smithi097.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T19:46:44.303 INFO:teuthology.orchestra.run.smithi097.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T19:46:44.303 INFO:teuthology.orchestra.run.smithi097.stdout:Creating mgr... 2024-03-11T19:46:44.303 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T19:46:44.304 INFO:teuthology.orchestra.run.smithi097.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T19:46:44.743 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.x 2024-03-11T19:46:44.743 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.x.service: Unit ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.x.service not loaded. 2024-03-11T19:46:44.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:44 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1572774730' entity='client.admin' 2024-03-11T19:46:44.979 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715.target.wants/ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@.service. 2024-03-11T19:46:45.350 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:45 smithi097 systemd[1]: Starting Ceph mgr.x for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T19:46:45.659 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:45 smithi097 podman[32111]: 2024-03-11 19:46:45.251095438 +0000 UTC m=+0.034758916 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T19:46:45.659 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:45 smithi097 podman[32111]: 2024-03-11 19:46:45.370828004 +0000 UTC m=+0.154491516 container create 69fd8238897fc4c56eccee52ceb85a235f2a09dd7c33bf2cf802e2f5ff7b6ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-11T19:46:45.659 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:45 smithi097 podman[32111]: 2024-03-11 19:46:45.654900058 +0000 UTC m=+0.438563540 container init 69fd8238897fc4c56eccee52ceb85a235f2a09dd7c33bf2cf802e2f5ff7b6ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:46:45.774 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T19:46:45.775 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout disabled 2024-03-11T19:46:45.792 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T19:46:45.792 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout inactive 2024-03-11T19:46:45.792 INFO:teuthology.orchestra.run.smithi097.stdout:firewalld.service is not enabled 2024-03-11T19:46:45.792 INFO:teuthology.orchestra.run.smithi097.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T19:46:45.812 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T19:46:45.812 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout disabled 2024-03-11T19:46:45.828 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T19:46:45.828 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout inactive 2024-03-11T19:46:45.829 INFO:teuthology.orchestra.run.smithi097.stdout:firewalld.service is not enabled 2024-03-11T19:46:45.829 INFO:teuthology.orchestra.run.smithi097.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T19:46:45.829 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for mgr to start... 2024-03-11T19:46:45.829 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for mgr... 2024-03-11T19:46:45.922 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:45 smithi097 podman[32111]: 2024-03-11 19:46:45.664757597 +0000 UTC m=+0.448421091 container start 69fd8238897fc4c56eccee52ceb85a235f2a09dd7c33bf2cf802e2f5ff7b6ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x, RELEASE=squid-94a5397, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:46:45.922 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:45 smithi097 bash[32111]: 69fd8238897fc4c56eccee52ceb85a235f2a09dd7c33bf2cf802e2f5ff7b6ac4 2024-03-11T19:46:45.922 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:45 smithi097 systemd[1]: Started Ceph mgr.x for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:46:46.545 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:46 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:46.429+0000 7f05eaaa4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T19:46:46.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:46 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1594571880' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T19:46:46.922 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:46 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:46.555+0000 7f05eaaa4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T19:46:47.267 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-03-11T19:46:47.268 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-03-11T19:46:47.268 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:46:47.268 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T19:46:47.268 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T19:46:47.268 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T19:46:47.268 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T19:46:47.268 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:47.268 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T19:46:47.268 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T19:46:47.268 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 0 2024-03-11T19:46:47.268 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:46:47.268 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T19:46:47.269 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T19:46:47.270 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:47.271 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T19:46:47.272 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:47.272 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T19:46:35.573902+0000", 2024-03-11T19:46:47.272 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T19:46:47.272 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:47.272 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T19:46:47.272 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-03-11T19:46:47.272 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.434+0000 7fc4f8600700 1 Processor -- start 2024-03-11T19:46:47.272 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.435+0000 7fc4f8600700 1 -- start start 2024-03-11T19:46:47.272 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.435+0000 7fc4f8600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 0x7fc4f007b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:47.272 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.435+0000 7fc4f8600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f007b830 con 0x7fc4f007ce40 2024-03-11T19:46:47.272 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.435+0000 7fc4f5c00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 0x7fc4f007b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:47.272 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.436+0000 7fc4f5c00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 0x7fc4f007b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39930/0 (socket says 172.21.15.97:39930) 2024-03-11T19:46:47.272 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.436+0000 7fc4f5c00700 1 -- 172.21.15.97:0/1116870635 learned_addr learned my addr 172.21.15.97:0/1116870635 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.436+0000 7fc4f5c00700 1 -- 172.21.15.97:0/1116870635 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f007b970 con 0x7fc4f007ce40 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.436+0000 7fc4f5c00700 1 --2- 172.21.15.97:0/1116870635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 0x7fc4f007b2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc4e40098a0 tx=0x7fc4e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a360339074348e0b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.436+0000 7fc4efe00700 1 -- 172.21.15.97:0/1116870635 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc4e4004160 con 0x7fc4f007ce40 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.437+0000 7fc4efe00700 1 -- 172.21.15.97:0/1116870635 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4e40042c0 con 0x7fc4f007ce40 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.437+0000 7fc4efe00700 1 -- 172.21.15.97:0/1116870635 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc4e4011ac0 con 0x7fc4f007ce40 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.437+0000 7fc4f8600700 1 -- 172.21.15.97:0/1116870635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 msgr2=0x7fc4f007b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.437+0000 7fc4f8600700 1 --2- 172.21.15.97:0/1116870635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 0x7fc4f007b2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc4e40098a0 tx=0x7fc4e4009bb0 comp rx=0 tx=0).stop 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.437+0000 7fc4f8600700 1 -- 172.21.15.97:0/1116870635 shutdown_connections 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.437+0000 7fc4f8600700 1 --2- 172.21.15.97:0/1116870635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 0x7fc4f007b2f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.437+0000 7fc4f8600700 1 -- 172.21.15.97:0/1116870635 >> 172.21.15.97:0/1116870635 conn(0x7fc4f0102bd0 msgr2=0x7fc4f0104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.438+0000 7fc4f8600700 1 -- 172.21.15.97:0/1116870635 shutdown_connections 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.438+0000 7fc4f8600700 1 -- 172.21.15.97:0/1116870635 wait complete. 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.439+0000 7fc4f8600700 1 Processor -- start 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.439+0000 7fc4f8600700 1 -- start start 2024-03-11T19:46:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.439+0000 7fc4f8600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 0x7fc4f019d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.439+0000 7fc4f8600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f019da00 con 0x7fc4f007ce40 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.439+0000 7fc4f5c00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 0x7fc4f019d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.439+0000 7fc4f5c00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 0x7fc4f019d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39942/0 (socket says 172.21.15.97:39942) 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.439+0000 7fc4f5c00700 1 -- 172.21.15.97:0/1594571880 learned_addr learned my addr 172.21.15.97:0/1594571880 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.440+0000 7fc4f5c00700 1 -- 172.21.15.97:0/1594571880 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e4009550 con 0x7fc4f007ce40 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.440+0000 7fc4f5c00700 1 --2- 172.21.15.97:0/1594571880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 0x7fc4f019d4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4011a70 tx=0x7fc4e4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.440+0000 7fc4ee000700 1 -- 172.21.15.97:0/1594571880 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc4e40039c0 con 0x7fc4f007ce40 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.440+0000 7fc4ee000700 1 -- 172.21.15.97:0/1594571880 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4e4003b20 con 0x7fc4f007ce40 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.440+0000 7fc4ee000700 1 -- 172.21.15.97:0/1594571880 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc4e4024610 con 0x7fc4f007ce40 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.440+0000 7fc4f8600700 1 -- 172.21.15.97:0/1594571880 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f019dc00 con 0x7fc4f007ce40 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.440+0000 7fc4f8600700 1 -- 172.21.15.97:0/1594571880 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f019e0a0 con 0x7fc4f007ce40 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.441+0000 7fc4ee000700 1 -- 172.21.15.97:0/1594571880 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc4e4024b90 con 0x7fc4f007ce40 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.441+0000 7fc4ee000700 1 -- 172.21.15.97:0/1594571880 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc4e4019910 con 0x7fc4f007ce40 2024-03-11T19:46:47.274 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.442+0000 7fc4f8600700 1 -- 172.21.15.97:0/1594571880 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f010d880 con 0x7fc4f007ce40 2024-03-11T19:46:47.275 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.444+0000 7fc4ee000700 1 -- 172.21.15.97:0/1594571880 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc4e401aa10 con 0x7fc4f007ce40 2024-03-11T19:46:47.275 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.523+0000 7fc4f8600700 1 -- 172.21.15.97:0/1594571880 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc4f01a0c70 con 0x7fc4f007ce40 2024-03-11T19:46:47.275 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.524+0000 7fc4ee000700 1 -- 172.21.15.97:0/1594571880 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fc4e401abb0 con 0x7fc4f007ce40 2024-03-11T19:46:47.275 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.525+0000 7fc4f8600700 1 -- 172.21.15.97:0/1594571880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 msgr2=0x7fc4f019d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:47.275 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.525+0000 7fc4f8600700 1 --2- 172.21.15.97:0/1594571880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 0x7fc4f019d4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4011a70 tx=0x7fc4e4011b50 comp rx=0 tx=0).stop 2024-03-11T19:46:47.275 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.525+0000 7fc4f8600700 1 -- 172.21.15.97:0/1594571880 shutdown_connections 2024-03-11T19:46:47.275 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.525+0000 7fc4f8600700 1 --2- 172.21.15.97:0/1594571880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4f007ce40 0x7fc4f019d4c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:47.275 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.525+0000 7fc4f8600700 1 -- 172.21.15.97:0/1594571880 >> 172.21.15.97:0/1594571880 conn(0x7fc4f0102bd0 msgr2=0x7fc4f0103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:47.275 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.526+0000 7fc4f8600700 1 -- 172.21.15.97:0/1594571880 shutdown_connections 2024-03-11T19:46:47.275 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:46.526+0000 7fc4f8600700 1 -- 172.21.15.97:0/1594571880 wait complete. 2024-03-11T19:46:47.275 INFO:teuthology.orchestra.run.smithi097.stdout:mgr not available, waiting (1/15)... 2024-03-11T19:46:47.521 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:47 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:47.396+0000 7f05eaaa4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T19:46:47.817 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:47 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:47.516+0000 7f05eaaa4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T19:46:47.817 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:47 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:47.690+0000 7f05eaaa4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T19:46:47.817 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:47 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:47.812+0000 7f05eaaa4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T19:46:48.171 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:47 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:47.921+0000 7f05eaaa4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T19:46:48.628 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:48 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:48.251+0000 7f05eaaa4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T19:46:48.628 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:48 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:48.389+0000 7f05eaaa4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T19:46:48.628 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:48 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:48.499+0000 7f05eaaa4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T19:46:48.922 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:48 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:48.622+0000 7f05eaaa4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T19:46:48.922 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:48 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:48.839+0000 7f05eaaa4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T19:46:50.339 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:50 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3493649036' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T19:46:50.339 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:50 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:50.156+0000 7f05eaaa4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T19:46:50.339 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:50 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:50.334+0000 7f05eaaa4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T19:46:50.736 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-03-11T19:46:50.736 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-03-11T19:46:50.736 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:46:50.736 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T19:46:50.736 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T19:46:50.736 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T19:46:50.736 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 0 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T19:46:50.737 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:50.738 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T19:46:50.738 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T19:46:50.738 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T19:46:50.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T19:46:50.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T19:46:50.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T19:46:50.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T19:46:50.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T19:46:35.573902+0000", 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T19:46:50.747 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.913+0000 7f598ae00700 1 Processor -- start 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.913+0000 7f598ae00700 1 -- start start 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.914+0000 7f598ae00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 0x7f5984106870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.914+0000 7f598ae00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5984106e40 con 0x7f5984106450 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.914+0000 7f5983e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 0x7f5984106870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.915+0000 7f5983e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 0x7f5984106870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39958/0 (socket says 172.21.15.97:39958) 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.915+0000 7f5983e00700 1 -- 172.21.15.97:0/921501381 learned_addr learned my addr 172.21.15.97:0/921501381 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.915+0000 7f5983e00700 1 -- 172.21.15.97:0/921501381 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5984107650 con 0x7f5984106450 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.915+0000 7f5983e00700 1 --2- 172.21.15.97:0/921501381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 0x7f5984106870 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f59740098a0 tx=0x7f5974009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512226f5edd07eee server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.915+0000 7f5982a00700 1 -- 172.21.15.97:0/921501381 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5974004160 con 0x7f5984106450 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.916+0000 7f5982a00700 1 -- 172.21.15.97:0/921501381 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59740042c0 con 0x7f5984106450 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.916+0000 7f5982a00700 1 -- 172.21.15.97:0/921501381 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5974011ac0 con 0x7f5984106450 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.916+0000 7f598ae00700 1 -- 172.21.15.97:0/921501381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 msgr2=0x7f5984106870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:50.748 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.916+0000 7f598ae00700 1 --2- 172.21.15.97:0/921501381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 0x7f5984106870 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f59740098a0 tx=0x7f5974009bb0 comp rx=0 tx=0).stop 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.916+0000 7f598ae00700 1 -- 172.21.15.97:0/921501381 shutdown_connections 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.916+0000 7f598ae00700 1 --2- 172.21.15.97:0/921501381 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 0x7f5984106870 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.916+0000 7f598ae00700 1 -- 172.21.15.97:0/921501381 >> 172.21.15.97:0/921501381 conn(0x7f5984101a30 msgr2=0x7f5984103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.917+0000 7f598ae00700 1 -- 172.21.15.97:0/921501381 shutdown_connections 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.917+0000 7f598ae00700 1 -- 172.21.15.97:0/921501381 wait complete. 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.918+0000 7f598ae00700 1 Processor -- start 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.918+0000 7f598ae00700 1 -- start start 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.918+0000 7f598ae00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 0x7f598419d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.918+0000 7f598ae00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598419daa0 con 0x7f5984106450 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.918+0000 7f5983e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 0x7f598419d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.918+0000 7f5983e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 0x7f598419d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39960/0 (socket says 172.21.15.97:39960) 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.918+0000 7f5983e00700 1 -- 172.21.15.97:0/3493649036 learned_addr learned my addr 172.21.15.97:0/3493649036 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.919+0000 7f5983e00700 1 -- 172.21.15.97:0/3493649036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5974009550 con 0x7f5984106450 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.919+0000 7f5983e00700 1 --2- 172.21.15.97:0/3493649036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 0x7f598419d560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5974009fb0 tx=0x7f5974011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:50.749 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.919+0000 7f5980c00700 1 -- 172.21.15.97:0/3493649036 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59740038b0 con 0x7f5984106450 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.919+0000 7f5980c00700 1 -- 172.21.15.97:0/3493649036 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5974003a10 con 0x7f5984106450 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.919+0000 7f598ae00700 1 -- 172.21.15.97:0/3493649036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598419dca0 con 0x7f5984106450 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.919+0000 7f5980c00700 1 -- 172.21.15.97:0/3493649036 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59740195e0 con 0x7f5984106450 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.919+0000 7f598ae00700 1 -- 172.21.15.97:0/3493649036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598419e140 con 0x7f5984106450 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.920+0000 7f5980c00700 1 -- 172.21.15.97:0/3493649036 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5974019c00 con 0x7f5984106450 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.920+0000 7f5980c00700 1 -- 172.21.15.97:0/3493649036 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5974024990 con 0x7f5984106450 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.920+0000 7f598ae00700 1 -- 172.21.15.97:0/3493649036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5984197820 con 0x7f5984106450 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:49.923+0000 7f5980c00700 1 -- 172.21.15.97:0/3493649036 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5974003b80 con 0x7f5984106450 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:50.000+0000 7f598ae00700 1 -- 172.21.15.97:0/3493649036 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f59840632a0 con 0x7f5984106450 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:50.001+0000 7f5980c00700 1 -- 172.21.15.97:0/3493649036 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f597401aa10 con 0x7f5984106450 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:50.002+0000 7f598ae00700 1 -- 172.21.15.97:0/3493649036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 msgr2=0x7f598419d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:50.002+0000 7f598ae00700 1 --2- 172.21.15.97:0/3493649036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 0x7f598419d560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5974009fb0 tx=0x7f5974011b20 comp rx=0 tx=0).stop 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:50.002+0000 7f598ae00700 1 -- 172.21.15.97:0/3493649036 shutdown_connections 2024-03-11T19:46:50.750 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:50.002+0000 7f598ae00700 1 --2- 172.21.15.97:0/3493649036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5984106450 0x7f598419d560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:50.751 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:50.002+0000 7f598ae00700 1 -- 172.21.15.97:0/3493649036 >> 172.21.15.97:0/3493649036 conn(0x7f5984101a30 msgr2=0x7f5984102990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:50.751 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:50.002+0000 7f598ae00700 1 -- 172.21.15.97:0/3493649036 shutdown_connections 2024-03-11T19:46:50.751 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:50.003+0000 7f598ae00700 1 -- 172.21.15.97:0/3493649036 wait complete. 2024-03-11T19:46:50.751 INFO:teuthology.orchestra.run.smithi097.stdout:mgr not available, waiting (2/15)... 2024-03-11T19:46:51.127 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:50 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:50.752+0000 7f05eaaa4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T19:46:51.127 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:50 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:50.869+0000 7f05eaaa4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T19:46:51.127 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:51 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:51.121+0000 7f05eaaa4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:46:51.421 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:51 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:51.361+0000 7f05eaaa4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T19:46:52.045 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:51 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:51.726+0000 7f05eaaa4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:46:52.045 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:52 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:52.039+0000 7f05eaaa4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T19:46:52.671 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:52 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:52.338+0000 7f05eaaa4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T19:46:53.463 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:53 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:53.407+0000 7f05eaaa4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T19:46:53.828 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:53 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3488513655' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T19:46:54.149 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-03-11T19:46:54.149 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-03-11T19:46:54.149 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:46:54.149 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 0 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T19:46:54.150 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T19:46:54.151 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T19:46:54.152 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T19:46:35.573902+0000", 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T19:46:54.153 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.320+0000 7fd912200700 1 Processor -- start 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.320+0000 7fd912200700 1 -- start start 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.321+0000 7fd912200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 0x7fd90c07b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.321+0000 7fd912200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c07b810 con 0x7fd90c07cdd0 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.321+0000 7fd90b400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 0x7fd90c07b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.321+0000 7fd90b400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 0x7fd90c07b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39976/0 (socket says 172.21.15.97:39976) 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.321+0000 7fd90b400700 1 -- 172.21.15.97:0/1808465880 learned_addr learned my addr 172.21.15.97:0/1808465880 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.322+0000 7fd90b400700 1 -- 172.21.15.97:0/1808465880 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c07b950 con 0x7fd90c07cdd0 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.322+0000 7fd90b400700 1 --2- 172.21.15.97:0/1808465880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 0x7fd90c07b2d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0098a0 tx=0x7fd8fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1c4e1e462da07 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.322+0000 7fd90a000700 1 -- 172.21.15.97:0/1808465880 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd8fc004160 con 0x7fd90c07cdd0 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.322+0000 7fd90a000700 1 -- 172.21.15.97:0/1808465880 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8fc0042c0 con 0x7fd90c07cdd0 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.322+0000 7fd90a000700 1 -- 172.21.15.97:0/1808465880 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd8fc011ac0 con 0x7fd90c07cdd0 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.323+0000 7fd912200700 1 -- 172.21.15.97:0/1808465880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 msgr2=0x7fd90c07b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:54.154 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.323+0000 7fd912200700 1 --2- 172.21.15.97:0/1808465880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 0x7fd90c07b2d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0098a0 tx=0x7fd8fc009bb0 comp rx=0 tx=0).stop 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.323+0000 7fd912200700 1 -- 172.21.15.97:0/1808465880 shutdown_connections 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.323+0000 7fd912200700 1 --2- 172.21.15.97:0/1808465880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 0x7fd90c07b2d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.323+0000 7fd912200700 1 -- 172.21.15.97:0/1808465880 >> 172.21.15.97:0/1808465880 conn(0x7fd90c102bd0 msgr2=0x7fd90c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.323+0000 7fd912200700 1 -- 172.21.15.97:0/1808465880 shutdown_connections 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.323+0000 7fd912200700 1 -- 172.21.15.97:0/1808465880 wait complete. 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.324+0000 7fd912200700 1 Processor -- start 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.324+0000 7fd912200700 1 -- start start 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.325+0000 7fd912200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 0x7fd90c19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.325+0000 7fd912200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c19da00 con 0x7fd90c07cdd0 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.325+0000 7fd90b400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 0x7fd90c19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.325+0000 7fd90b400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 0x7fd90c19d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39990/0 (socket says 172.21.15.97:39990) 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.325+0000 7fd90b400700 1 -- 172.21.15.97:0/3488513655 learned_addr learned my addr 172.21.15.97:0/3488513655 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.325+0000 7fd90b400700 1 -- 172.21.15.97:0/3488513655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc009550 con 0x7fd90c07cdd0 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.326+0000 7fd90b400700 1 --2- 172.21.15.97:0/3488513655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 0x7fd90c19d4c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc011a70 tx=0x7fd8fc011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:54.155 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.326+0000 7fd903e00700 1 -- 172.21.15.97:0/3488513655 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd8fc0039c0 con 0x7fd90c07cdd0 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.326+0000 7fd903e00700 1 -- 172.21.15.97:0/3488513655 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8fc003b20 con 0x7fd90c07cdd0 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.326+0000 7fd912200700 1 -- 172.21.15.97:0/3488513655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c19dc00 con 0x7fd90c07cdd0 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.326+0000 7fd903e00700 1 -- 172.21.15.97:0/3488513655 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd8fc024610 con 0x7fd90c07cdd0 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.326+0000 7fd912200700 1 -- 172.21.15.97:0/3488513655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c19e0a0 con 0x7fd90c07cdd0 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.327+0000 7fd903e00700 1 -- 172.21.15.97:0/3488513655 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd8fc024c00 con 0x7fd90c07cdd0 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.327+0000 7fd903e00700 1 -- 172.21.15.97:0/3488513655 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd8fc019920 con 0x7fd90c07cdd0 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.327+0000 7fd912200700 1 -- 172.21.15.97:0/3488513655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c10d8a0 con 0x7fd90c07cdd0 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.330+0000 7fd903e00700 1 -- 172.21.15.97:0/3488513655 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd8fc01aa10 con 0x7fd90c07cdd0 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.405+0000 7fd912200700 1 -- 172.21.15.97:0/3488513655 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd90c10dab0 con 0x7fd90c07cdd0 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.406+0000 7fd903e00700 1 -- 172.21.15.97:0/3488513655 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7fd8fc01abb0 con 0x7fd90c07cdd0 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.407+0000 7fd912200700 1 -- 172.21.15.97:0/3488513655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 msgr2=0x7fd90c19d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.407+0000 7fd912200700 1 --2- 172.21.15.97:0/3488513655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 0x7fd90c19d4c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc011a70 tx=0x7fd8fc011b50 comp rx=0 tx=0).stop 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.407+0000 7fd912200700 1 -- 172.21.15.97:0/3488513655 shutdown_connections 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.407+0000 7fd912200700 1 --2- 172.21.15.97:0/3488513655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd90c07cdd0 0x7fd90c19d4c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.407+0000 7fd912200700 1 -- 172.21.15.97:0/3488513655 >> 172.21.15.97:0/3488513655 conn(0x7fd90c102bd0 msgr2=0x7fd90c103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:54.156 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.407+0000 7fd912200700 1 -- 172.21.15.97:0/3488513655 shutdown_connections 2024-03-11T19:46:54.157 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:53.407+0000 7fd912200700 1 -- 172.21.15.97:0/3488513655 wait complete. 2024-03-11T19:46:54.157 INFO:teuthology.orchestra.run.smithi097.stdout:mgr not available, waiting (3/15)... 2024-03-11T19:46:54.421 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:54.112+0000 7f05eaaa4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T19:46:54.421 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:54.247+0000 7f05eaaa4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T19:46:54.921 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:46:54.526+0000 7f05eaaa4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T19:46:54.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: Activating manager daemon x 2024-03-11T19:46:54.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: mgrmap e2: x(active, starting, since 0.0112133s) 2024-03-11T19:46:54.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: from='mgr.14100 172.21.15.97:0/1166496113' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T19:46:54.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: from='mgr.14100 172.21.15.97:0/1166496113' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T19:46:54.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: from='mgr.14100 172.21.15.97:0/1166496113' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T19:46:54.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: from='mgr.14100 172.21.15.97:0/1166496113' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T19:46:54.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: from='mgr.14100 172.21.15.97:0/1166496113' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T19:46:54.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: from='mgr.14100 172.21.15.97:0/1166496113' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T19:46:54.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: from='mgr.14100 172.21.15.97:0/1166496113' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:46:54.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: from='mgr.14100 172.21.15.97:0/1166496113' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T19:46:54.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: Manager daemon x is now available 2024-03-11T19:46:54.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: from='mgr.14100 172.21.15.97:0/1166496113' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T19:46:54.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: from='mgr.14100 172.21.15.97:0/1166496113' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T19:46:54.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: from='mgr.14100 172.21.15.97:0/1166496113' entity='mgr.x' 2024-03-11T19:46:54.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: from='mgr.14100 172.21.15.97:0/1166496113' entity='mgr.x' 2024-03-11T19:46:54.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:54 smithi097 ceph-mon[31898]: from='mgr.14100 172.21.15.97:0/1166496113' entity='mgr.x' 2024-03-11T19:46:56.589 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:56 smithi097 ceph-mon[31898]: mgrmap e3: x(active, since 1.02038s) 2024-03-11T19:46:57.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:57 smithi097 ceph-mon[31898]: mgrmap e4: x(active, since 2s) 2024-03-11T19:46:57.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:57 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2037796622' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T19:46:57.773 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-03-11T19:46:57.773 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-03-11T19:46:57.773 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:46:57.773 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T19:46:57.773 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T19:46:57.773 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T19:46:57.773 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T19:46:57.773 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:57.773 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T19:46:57.773 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T19:46:57.773 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 0 2024-03-11T19:46:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:46:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T19:46:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T19:46:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:46:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-11T19:46:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T19:46:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T19:46:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T19:46:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T19:46:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T19:46:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ], 2024-03-11T19:46:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T19:46:57.777 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:57.777 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T19:46:57.777 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T19:46:57.777 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T19:46:35.573902+0000", 2024-03-11T19:46:57.777 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T19:46:57.777 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout }, 2024-03-11T19:46:57.777 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T19:46:57.777 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-03-11T19:46:57.777 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.740+0000 7fed6fa00700 1 Processor -- start 2024-03-11T19:46:57.777 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.740+0000 7fed6fa00700 1 -- start start 2024-03-11T19:46:57.777 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.741+0000 7fed6fa00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 0x7fed68109bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:57.777 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.741+0000 7fed6fa00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6810a1a0 con 0x7fed681097b0 2024-03-11T19:46:57.777 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.741+0000 7fed6d000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 0x7fed68109bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.741+0000 7fed6d000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 0x7fed68109bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45252/0 (socket says 172.21.15.97:45252) 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.741+0000 7fed6d000700 1 -- 172.21.15.97:0/3759086098 learned_addr learned my addr 172.21.15.97:0/3759086098 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.742+0000 7fed6d000700 1 -- 172.21.15.97:0/3759086098 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6810a960 con 0x7fed681097b0 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.742+0000 7fed6d000700 1 --2- 172.21.15.97:0/3759086098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 0x7fed68109bd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0098a0 tx=0x7fed5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5a2ff5429efad3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.742+0000 7fed67400700 1 -- 172.21.15.97:0/3759086098 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed5c004160 con 0x7fed681097b0 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.742+0000 7fed67400700 1 -- 172.21.15.97:0/3759086098 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed5c0042c0 con 0x7fed681097b0 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.742+0000 7fed67400700 1 -- 172.21.15.97:0/3759086098 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed5c011ac0 con 0x7fed681097b0 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.743+0000 7fed6fa00700 1 -- 172.21.15.97:0/3759086098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 msgr2=0x7fed68109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.743+0000 7fed6fa00700 1 --2- 172.21.15.97:0/3759086098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 0x7fed68109bd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0098a0 tx=0x7fed5c009bb0 comp rx=0 tx=0).stop 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.743+0000 7fed6fa00700 1 -- 172.21.15.97:0/3759086098 shutdown_connections 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.743+0000 7fed6fa00700 1 --2- 172.21.15.97:0/3759086098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 0x7fed68109bd0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.743+0000 7fed6fa00700 1 -- 172.21.15.97:0/3759086098 >> 172.21.15.97:0/3759086098 conn(0x7fed6807cbf0 msgr2=0x7fed681074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.743+0000 7fed6fa00700 1 -- 172.21.15.97:0/3759086098 shutdown_connections 2024-03-11T19:46:57.778 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.743+0000 7fed6fa00700 1 -- 172.21.15.97:0/3759086098 wait complete. 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.744+0000 7fed6fa00700 1 Processor -- start 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.745+0000 7fed6fa00700 1 -- start start 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.745+0000 7fed6fa00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 0x7fed6819d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.745+0000 7fed6fa00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6819dae0 con 0x7fed681097b0 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.745+0000 7fed6d000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 0x7fed6819d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.745+0000 7fed6d000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 0x7fed6819d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45256/0 (socket says 172.21.15.97:45256) 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.745+0000 7fed6d000700 1 -- 172.21.15.97:0/2037796622 learned_addr learned my addr 172.21.15.97:0/2037796622 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.745+0000 7fed6d000700 1 -- 172.21.15.97:0/2037796622 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c009550 con 0x7fed681097b0 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.746+0000 7fed6d000700 1 --2- 172.21.15.97:0/2037796622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 0x7fed6819d5a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fed5c011780 tx=0x7fed5c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.746+0000 7fed65600700 1 -- 172.21.15.97:0/2037796622 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed5c003af0 con 0x7fed681097b0 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.746+0000 7fed65600700 1 -- 172.21.15.97:0/2037796622 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed5c003c50 con 0x7fed681097b0 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.746+0000 7fed65600700 1 -- 172.21.15.97:0/2037796622 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed5c0197e0 con 0x7fed681097b0 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.746+0000 7fed6fa00700 1 -- 172.21.15.97:0/2037796622 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6819dce0 con 0x7fed681097b0 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.746+0000 7fed6fa00700 1 -- 172.21.15.97:0/2037796622 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6819e180 con 0x7fed681097b0 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.747+0000 7fed65600700 1 -- 172.21.15.97:0/2037796622 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7fed5c01aba0 con 0x7fed681097b0 2024-03-11T19:46:57.779 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.747+0000 7fed6fa00700 1 -- 172.21.15.97:0/2037796622 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6810db90 con 0x7fed681097b0 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.748+0000 7fed65600700 1 --2- 172.21.15.97:0/2037796622 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fed50038c60 0x7fed5003b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.748+0000 7fed65600700 1 -- 172.21.15.97:0/2037796622 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fed5c04cf80 con 0x7fed681097b0 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.748+0000 7fed67e00700 1 --2- 172.21.15.97:0/2037796622 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fed50038c60 0x7fed5003b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.749+0000 7fed67e00700 1 --2- 172.21.15.97:0/2037796622 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fed50038c60 0x7fed5003b120 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fed54006fd0 tx=0x7fed54006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:56.754+0000 7fed65600700 1 -- 172.21.15.97:0/2037796622 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fed5c01a430 con 0x7fed681097b0 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:57.050+0000 7fed6fa00700 1 -- 172.21.15.97:0/2037796622 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fed681a0ea0 con 0x7fed681097b0 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:57.050+0000 7fed65600700 1 -- 172.21.15.97:0/2037796622 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fed5c02dbe0 con 0x7fed681097b0 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:57.053+0000 7fed6fa00700 1 -- 172.21.15.97:0/2037796622 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fed50038c60 msgr2=0x7fed5003b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:57.053+0000 7fed6fa00700 1 --2- 172.21.15.97:0/2037796622 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fed50038c60 0x7fed5003b120 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fed54006fd0 tx=0x7fed54006e40 comp rx=0 tx=0).stop 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:57.053+0000 7fed6fa00700 1 -- 172.21.15.97:0/2037796622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 msgr2=0x7fed6819d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:57.053+0000 7fed6fa00700 1 --2- 172.21.15.97:0/2037796622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 0x7fed6819d5a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fed5c011780 tx=0x7fed5c011b40 comp rx=0 tx=0).stop 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:57.053+0000 7fed6fa00700 1 -- 172.21.15.97:0/2037796622 shutdown_connections 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:57.053+0000 7fed6fa00700 1 --2- 172.21.15.97:0/2037796622 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fed50038c60 0x7fed5003b120 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:57.053+0000 7fed6fa00700 1 --2- 172.21.15.97:0/2037796622 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed681097b0 0x7fed6819d5a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:57.780 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:57.053+0000 7fed6fa00700 1 -- 172.21.15.97:0/2037796622 >> 172.21.15.97:0/2037796622 conn(0x7fed6807cbf0 msgr2=0x7fed68106c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:57.781 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:57.053+0000 7fed6fa00700 1 -- 172.21.15.97:0/2037796622 shutdown_connections 2024-03-11T19:46:57.781 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:57.053+0000 7fed6fa00700 1 -- 172.21.15.97:0/2037796622 wait complete. 2024-03-11T19:46:57.781 INFO:teuthology.orchestra.run.smithi097.stdout:mgr is available 2024-03-11T19:46:59.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:46:58 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/4106804615' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T19:46:59.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-03-11T19:46:59.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T19:46:59.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout fsid = e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:46:59.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T19:46:59.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T19:46:59.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T19:46:59.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T19:46:59.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-03-11T19:46:59.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T19:46:59.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T19:46:59.636 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 2024-03-11T19:46:59.636 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T19:46:59.636 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-11T19:46:59.636 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-11T19:46:59.636 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-11T19:46:59.636 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T19:46:59.636 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T19:46:59.636 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.663+0000 7fdabb000700 1 Processor -- start 2024-03-11T19:46:59.636 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.663+0000 7fdabb000700 1 -- start start 2024-03-11T19:46:59.636 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.663+0000 7fdabb000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 0x7fdab4106230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:59.636 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.663+0000 7fdabb000700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab407ae60 con 0x7fdab4105e10 2024-03-11T19:46:59.636 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.664+0000 7fdab3e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 0x7fdab4106230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.664+0000 7fdab3e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 0x7fdab4106230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45264/0 (socket says 172.21.15.97:45264) 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.664+0000 7fdab3e00700 1 -- 172.21.15.97:0/193835819 learned_addr learned my addr 172.21.15.97:0/193835819 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.664+0000 7fdab3e00700 1 -- 172.21.15.97:0/193835819 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab4106770 con 0x7fdab4105e10 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.665+0000 7fdab3e00700 1 --2- 172.21.15.97:0/193835819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 0x7fdab4106230 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdaa40098a0 tx=0x7fdaa4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8cf1d18c7c4aced server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.665+0000 7fdab2a00700 1 -- 172.21.15.97:0/193835819 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdaa4004160 con 0x7fdab4105e10 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.665+0000 7fdab2a00700 1 -- 172.21.15.97:0/193835819 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdaa40042c0 con 0x7fdab4105e10 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.665+0000 7fdab2a00700 1 -- 172.21.15.97:0/193835819 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdaa4011ac0 con 0x7fdab4105e10 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.666+0000 7fdabb000700 1 -- 172.21.15.97:0/193835819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 msgr2=0x7fdab4106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.666+0000 7fdabb000700 1 --2- 172.21.15.97:0/193835819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 0x7fdab4106230 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdaa40098a0 tx=0x7fdaa4009bb0 comp rx=0 tx=0).stop 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.666+0000 7fdabb000700 1 -- 172.21.15.97:0/193835819 shutdown_connections 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.666+0000 7fdabb000700 1 --2- 172.21.15.97:0/193835819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 0x7fdab4106230 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.666+0000 7fdabb000700 1 -- 172.21.15.97:0/193835819 >> 172.21.15.97:0/193835819 conn(0x7fdab4101a30 msgr2=0x7fdab4103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.666+0000 7fdabb000700 1 -- 172.21.15.97:0/193835819 shutdown_connections 2024-03-11T19:46:59.637 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.666+0000 7fdabb000700 1 -- 172.21.15.97:0/193835819 wait complete. 2024-03-11T19:46:59.638 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.667+0000 7fdabb000700 1 Processor -- start 2024-03-11T19:46:59.638 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.667+0000 7fdabb000700 1 -- start start 2024-03-11T19:46:59.638 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.667+0000 7fdabb000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 0x7fdab419d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:59.638 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.667+0000 7fdabb000700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab419da30 con 0x7fdab4105e10 2024-03-11T19:46:59.638 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.667+0000 7fdab3e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 0x7fdab419d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:59.638 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.667+0000 7fdab3e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 0x7fdab419d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45266/0 (socket says 172.21.15.97:45266) 2024-03-11T19:46:59.638 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.668+0000 7fdab3e00700 1 -- 172.21.15.97:0/4106804615 learned_addr learned my addr 172.21.15.97:0/4106804615 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:46:59.638 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.668+0000 7fdab3e00700 1 -- 172.21.15.97:0/4106804615 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa4009550 con 0x7fdab4105e10 2024-03-11T19:46:59.638 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.668+0000 7fdab3e00700 1 --2- 172.21.15.97:0/4106804615 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 0x7fdab419d4f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4011a70 tx=0x7fdaa4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:59.638 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.668+0000 7fdab0c00700 1 -- 172.21.15.97:0/4106804615 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdaa40039c0 con 0x7fdab4105e10 2024-03-11T19:46:59.638 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.668+0000 7fdab0c00700 1 -- 172.21.15.97:0/4106804615 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdaa4003b20 con 0x7fdab4105e10 2024-03-11T19:46:59.638 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.668+0000 7fdab0c00700 1 -- 172.21.15.97:0/4106804615 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdaa4024610 con 0x7fdab4105e10 2024-03-11T19:46:59.638 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.668+0000 7fdabb000700 1 -- 172.21.15.97:0/4106804615 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab419dc30 con 0x7fdab4105e10 2024-03-11T19:46:59.639 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.669+0000 7fdabb000700 1 -- 172.21.15.97:0/4106804615 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab419e0d0 con 0x7fdab4105e10 2024-03-11T19:46:59.639 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.670+0000 7fdab0c00700 1 -- 172.21.15.97:0/4106804615 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7fdaa401aa10 con 0x7fdab4105e10 2024-03-11T19:46:59.641 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.670+0000 7fdabb000700 1 -- 172.21.15.97:0/4106804615 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab4197190 con 0x7fdab4105e10 2024-03-11T19:46:59.641 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.670+0000 7fdab0c00700 1 --2- 172.21.15.97:0/4106804615 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fda98038cb0 0x7fda9803b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:46:59.641 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.670+0000 7fdab0c00700 1 -- 172.21.15.97:0/4106804615 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdaa404cbe0 con 0x7fdab4105e10 2024-03-11T19:46:59.641 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.671+0000 7fdab3400700 1 --2- 172.21.15.97:0/4106804615 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fda98038cb0 0x7fda9803b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:46:59.641 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.671+0000 7fdab3400700 1 --2- 172.21.15.97:0/4106804615 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fda98038cb0 0x7fda9803b170 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0006fd0 tx=0x7fdaa0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:46:59.641 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.675+0000 7fdab0c00700 1 -- 172.21.15.97:0/4106804615 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdaa4018dc0 con 0x7fdab4105e10 2024-03-11T19:46:59.641 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.891+0000 7fdabb000700 1 -- 172.21.15.97:0/4106804615 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fdab41075d0 con 0x7fdab4105e10 2024-03-11T19:46:59.641 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.893+0000 7fdab0c00700 1 -- 172.21.15.97:0/4106804615 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fdaa4018be0 con 0x7fdab4105e10 2024-03-11T19:46:59.641 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.896+0000 7fdabb000700 1 -- 172.21.15.97:0/4106804615 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fda98038cb0 msgr2=0x7fda9803b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:59.641 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.896+0000 7fdabb000700 1 --2- 172.21.15.97:0/4106804615 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fda98038cb0 0x7fda9803b170 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0006fd0 tx=0x7fdaa0006e40 comp rx=0 tx=0).stop 2024-03-11T19:46:59.641 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.896+0000 7fdabb000700 1 -- 172.21.15.97:0/4106804615 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 msgr2=0x7fdab419d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:46:59.641 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.896+0000 7fdabb000700 1 --2- 172.21.15.97:0/4106804615 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 0x7fdab419d4f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4011a70 tx=0x7fdaa4011b50 comp rx=0 tx=0).stop 2024-03-11T19:46:59.641 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.897+0000 7fdabb000700 1 -- 172.21.15.97:0/4106804615 shutdown_connections 2024-03-11T19:46:59.642 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.897+0000 7fdabb000700 1 --2- 172.21.15.97:0/4106804615 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fda98038cb0 0x7fda9803b170 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:59.642 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.897+0000 7fdabb000700 1 --2- 172.21.15.97:0/4106804615 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdab4105e10 0x7fdab419d4f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:46:59.642 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.897+0000 7fdabb000700 1 -- 172.21.15.97:0/4106804615 >> 172.21.15.97:0/4106804615 conn(0x7fdab4101a30 msgr2=0x7fdab41942d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:46:59.642 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.897+0000 7fdabb000700 1 -- 172.21.15.97:0/4106804615 shutdown_connections 2024-03-11T19:46:59.642 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:46:58.897+0000 7fdabb000700 1 -- 172.21.15.97:0/4106804615 wait complete. 2024-03-11T19:46:59.642 INFO:teuthology.orchestra.run.smithi097.stdout:Enabling cephadm module... 2024-03-11T19:47:00.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:00 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/438242399' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T19:47:00.922 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:00 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: ignoring --setuser ceph since I am not root 2024-03-11T19:47:00.922 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:00 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: ignoring --setgroup ceph since I am not root 2024-03-11T19:47:01.333 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.249+0000 7fcd3fc00700 1 Processor -- start 2024-03-11T19:47:01.333 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.250+0000 7fcd3fc00700 1 -- start start 2024-03-11T19:47:01.334 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.250+0000 7fcd3fc00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 0x7fcd3807b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:01.334 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.250+0000 7fcd3fc00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3807b920 con 0x7fcd3807cea0 2024-03-11T19:47:01.334 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.250+0000 7fcd3d200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 0x7fcd3807b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:01.334 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.251+0000 7fcd3d200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 0x7fcd3807b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45278/0 (socket says 172.21.15.97:45278) 2024-03-11T19:47:01.334 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.251+0000 7fcd3d200700 1 -- 172.21.15.97:0/2746737468 learned_addr learned my addr 172.21.15.97:0/2746737468 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:01.334 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.251+0000 7fcd3d200700 1 -- 172.21.15.97:0/2746737468 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3807ba60 con 0x7fcd3807cea0 2024-03-11T19:47:01.334 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.251+0000 7fcd3d200700 1 --2- 172.21.15.97:0/2746737468 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 0x7fcd3807b350 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0098a0 tx=0x7fcd2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b820d10c04940d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:01.334 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.251+0000 7fcd37400700 1 -- 172.21.15.97:0/2746737468 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd2c004160 con 0x7fcd3807cea0 2024-03-11T19:47:01.334 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.252+0000 7fcd37400700 1 -- 172.21.15.97:0/2746737468 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd2c0042c0 con 0x7fcd3807cea0 2024-03-11T19:47:01.334 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.252+0000 7fcd37400700 1 -- 172.21.15.97:0/2746737468 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd2c011ac0 con 0x7fcd3807cea0 2024-03-11T19:47:01.334 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.252+0000 7fcd3fc00700 1 -- 172.21.15.97:0/2746737468 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 msgr2=0x7fcd3807b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:01.334 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.252+0000 7fcd3fc00700 1 --2- 172.21.15.97:0/2746737468 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 0x7fcd3807b350 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0098a0 tx=0x7fcd2c009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:01.334 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.252+0000 7fcd3fc00700 1 -- 172.21.15.97:0/2746737468 shutdown_connections 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.252+0000 7fcd3fc00700 1 --2- 172.21.15.97:0/2746737468 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 0x7fcd3807b350 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.252+0000 7fcd3fc00700 1 -- 172.21.15.97:0/2746737468 >> 172.21.15.97:0/2746737468 conn(0x7fcd38102c30 msgr2=0x7fcd38105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.252+0000 7fcd3fc00700 1 -- 172.21.15.97:0/2746737468 shutdown_connections 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.252+0000 7fcd3fc00700 1 -- 172.21.15.97:0/2746737468 wait complete. 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.253+0000 7fcd3fc00700 1 Processor -- start 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.254+0000 7fcd3fc00700 1 -- start start 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.254+0000 7fcd3fc00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 0x7fcd3819d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.254+0000 7fcd3fc00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3819da20 con 0x7fcd3807cea0 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.254+0000 7fcd3d200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 0x7fcd3819d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.254+0000 7fcd3d200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 0x7fcd3819d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45290/0 (socket says 172.21.15.97:45290) 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.254+0000 7fcd3d200700 1 -- 172.21.15.97:0/438242399 learned_addr learned my addr 172.21.15.97:0/438242399 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.255+0000 7fcd3d200700 1 -- 172.21.15.97:0/438242399 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2c009550 con 0x7fcd3807cea0 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.255+0000 7fcd3d200700 1 --2- 172.21.15.97:0/438242399 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 0x7fcd3819d4e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c009130 tx=0x7fcd2c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:01.335 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.255+0000 7fcd35600700 1 -- 172.21.15.97:0/438242399 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd2c003aa0 con 0x7fcd3807cea0 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.255+0000 7fcd35600700 1 -- 172.21.15.97:0/438242399 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd2c003c00 con 0x7fcd3807cea0 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.255+0000 7fcd35600700 1 -- 172.21.15.97:0/438242399 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd2c024800 con 0x7fcd3807cea0 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.255+0000 7fcd3fc00700 1 -- 172.21.15.97:0/438242399 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3819dc20 con 0x7fcd3807cea0 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.255+0000 7fcd3fc00700 1 -- 172.21.15.97:0/438242399 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd3819e0c0 con 0x7fcd3807cea0 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.256+0000 7fcd3fc00700 1 -- 172.21.15.97:0/438242399 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd3810d980 con 0x7fcd3807cea0 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.256+0000 7fcd35600700 1 -- 172.21.15.97:0/438242399 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7fcd2c01abd0 con 0x7fcd3807cea0 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.257+0000 7fcd35600700 1 --2- 172.21.15.97:0/438242399 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fcd20038cb0 0x7fcd2003b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.257+0000 7fcd35600700 1 -- 172.21.15.97:0/438242399 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcd2c04ca10 con 0x7fcd3807cea0 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.257+0000 7fcd37e00700 1 --2- 172.21.15.97:0/438242399 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fcd20038cb0 0x7fcd2003b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.258+0000 7fcd37e00700 1 --2- 172.21.15.97:0/438242399 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fcd20038cb0 0x7fcd2003b170 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcd24006fd0 tx=0x7fcd24006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.262+0000 7fcd35600700 1 -- 172.21.15.97:0/438242399 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcd2c018760 con 0x7fcd3807cea0 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.542+0000 7fcd3fc00700 1 -- 172.21.15.97:0/438242399 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fcd381a0de0 con 0x7fcd3807cea0 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.595+0000 7fcd35600700 1 -- 172.21.15.97:0/438242399 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fcd2c00b3e0 con 0x7fcd3807cea0 2024-03-11T19:47:01.336 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.599+0000 7fcd35600700 1 -- 172.21.15.97:0/438242399 <== mon.0 v2:172.21.15.97:3300/0 8 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fcd2c01a430 con 0x7fcd3807cea0 2024-03-11T19:47:01.337 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.600+0000 7fcd3fc00700 1 -- 172.21.15.97:0/438242399 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fcd20038cb0 msgr2=0x7fcd2003b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:01.337 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.600+0000 7fcd3fc00700 1 --2- 172.21.15.97:0/438242399 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fcd20038cb0 0x7fcd2003b170 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcd24006fd0 tx=0x7fcd24006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:01.337 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.601+0000 7fcd3fc00700 1 -- 172.21.15.97:0/438242399 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 msgr2=0x7fcd3819d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:01.337 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.601+0000 7fcd3fc00700 1 --2- 172.21.15.97:0/438242399 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 0x7fcd3819d4e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c009130 tx=0x7fcd2c011b40 comp rx=0 tx=0).stop 2024-03-11T19:47:01.337 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.601+0000 7fcd3fc00700 1 -- 172.21.15.97:0/438242399 shutdown_connections 2024-03-11T19:47:01.337 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.601+0000 7fcd3fc00700 1 --2- 172.21.15.97:0/438242399 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fcd20038cb0 0x7fcd2003b170 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:01.337 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.601+0000 7fcd3fc00700 1 --2- 172.21.15.97:0/438242399 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd3807cea0 0x7fcd3819d4e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:01.337 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.601+0000 7fcd3fc00700 1 -- 172.21.15.97:0/438242399 >> 172.21.15.97:0/438242399 conn(0x7fcd38102c30 msgr2=0x7fcd38103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:01.337 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.601+0000 7fcd3fc00700 1 -- 172.21.15.97:0/438242399 shutdown_connections 2024-03-11T19:47:01.337 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:00.601+0000 7fcd3fc00700 1 -- 172.21.15.97:0/438242399 wait complete. 2024-03-11T19:47:01.609 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:01 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:01.395+0000 7f3d6857c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T19:47:01.609 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:01 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:01.518+0000 7f3d6857c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T19:47:01.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:01 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/438242399' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T19:47:01.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:01 smithi097 ceph-mon[31898]: mgrmap e5: x(active, since 6s) 2024-03-11T19:47:02.451 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:02 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:02.323+0000 7f3d6857c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T19:47:02.451 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:02 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:02.445+0000 7f3d6857c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T19:47:02.756 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:02 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1225921973' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T19:47:02.756 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:02 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:02.624+0000 7f3d6857c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T19:47:02.756 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:02 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:02.750+0000 7f3d6857c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T19:47:03.010 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:02 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:02.859+0000 7f3d6857c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T19:47:03.017 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-03-11T19:47:03.017 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T19:47:03.018 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T19:47:03.018 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-11T19:47:03.018 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T19:47:03.018 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-03-11T19:47:03.019 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.987+0000 7fd1bd600700 1 Processor -- start 2024-03-11T19:47:03.019 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.987+0000 7fd1bd600700 1 -- start start 2024-03-11T19:47:03.019 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.988+0000 7fd1bd600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 0x7fd1b8106230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:03.019 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.988+0000 7fd1bd600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b807ae60 con 0x7fd1b8105e10 2024-03-11T19:47:03.019 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.988+0000 7fd1b6a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 0x7fd1b8106230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:03.019 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.988+0000 7fd1b6a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 0x7fd1b8106230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45308/0 (socket says 172.21.15.97:45308) 2024-03-11T19:47:03.019 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.988+0000 7fd1b6a00700 1 -- 172.21.15.97:0/2276086844 learned_addr learned my addr 172.21.15.97:0/2276086844 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:03.021 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.988+0000 7fd1b6a00700 1 -- 172.21.15.97:0/2276086844 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b8106770 con 0x7fd1b8105e10 2024-03-11T19:47:03.021 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.989+0000 7fd1b6a00700 1 --2- 172.21.15.97:0/2276086844 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 0x7fd1b8106230 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd1a80098a0 tx=0x7fd1a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1251ce1d285144a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:03.021 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.989+0000 7fd1b5600700 1 -- 172.21.15.97:0/2276086844 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd1a8004160 con 0x7fd1b8105e10 2024-03-11T19:47:03.021 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.989+0000 7fd1b5600700 1 -- 172.21.15.97:0/2276086844 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd1a80042c0 con 0x7fd1b8105e10 2024-03-11T19:47:03.021 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.989+0000 7fd1b5600700 1 -- 172.21.15.97:0/2276086844 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd1a8011ac0 con 0x7fd1b8105e10 2024-03-11T19:47:03.021 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.989+0000 7fd1bd600700 1 -- 172.21.15.97:0/2276086844 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 msgr2=0x7fd1b8106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:03.021 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.989+0000 7fd1bd600700 1 --2- 172.21.15.97:0/2276086844 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 0x7fd1b8106230 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd1a80098a0 tx=0x7fd1a8009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:03.021 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.989+0000 7fd1bd600700 1 -- 172.21.15.97:0/2276086844 shutdown_connections 2024-03-11T19:47:03.021 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.990+0000 7fd1bd600700 1 --2- 172.21.15.97:0/2276086844 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 0x7fd1b8106230 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:03.021 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.990+0000 7fd1bd600700 1 -- 172.21.15.97:0/2276086844 >> 172.21.15.97:0/2276086844 conn(0x7fd1b8101a30 msgr2=0x7fd1b8103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:03.023 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.990+0000 7fd1bd600700 1 -- 172.21.15.97:0/2276086844 shutdown_connections 2024-03-11T19:47:03.023 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.990+0000 7fd1bd600700 1 -- 172.21.15.97:0/2276086844 wait complete. 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.991+0000 7fd1bd600700 1 Processor -- start 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.991+0000 7fd1bd600700 1 -- start start 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.991+0000 7fd1bd600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 0x7fd1b819d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.991+0000 7fd1bd600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b819da30 con 0x7fd1b8105e10 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.991+0000 7fd1b6a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 0x7fd1b819d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.991+0000 7fd1b6a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 0x7fd1b819d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45316/0 (socket says 172.21.15.97:45316) 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.991+0000 7fd1b6a00700 1 -- 172.21.15.97:0/1225921973 learned_addr learned my addr 172.21.15.97:0/1225921973 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.992+0000 7fd1b6a00700 1 -- 172.21.15.97:0/1225921973 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a8009550 con 0x7fd1b8105e10 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.992+0000 7fd1b6a00700 1 --2- 172.21.15.97:0/1225921973 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 0x7fd1b819d4f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8011a70 tx=0x7fd1a8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.992+0000 7fd1af400700 1 -- 172.21.15.97:0/1225921973 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd1a80039c0 con 0x7fd1b8105e10 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.992+0000 7fd1af400700 1 -- 172.21.15.97:0/1225921973 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd1a8003b20 con 0x7fd1b8105e10 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.992+0000 7fd1bd600700 1 -- 172.21.15.97:0/1225921973 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b819dc30 con 0x7fd1b8105e10 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.992+0000 7fd1af400700 1 -- 172.21.15.97:0/1225921973 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd1a8024610 con 0x7fd1b8105e10 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.992+0000 7fd1bd600700 1 -- 172.21.15.97:0/1225921973 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b819e0d0 con 0x7fd1b8105e10 2024-03-11T19:47:03.024 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.993+0000 7fd1af400700 1 -- 172.21.15.97:0/1225921973 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fd1a801aa10 con 0x7fd1b8105e10 2024-03-11T19:47:03.027 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.993+0000 7fd1bd600700 1 -- 172.21.15.97:0/1225921973 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b8197190 con 0x7fd1b8105e10 2024-03-11T19:47:03.027 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.994+0000 7fd1af400700 1 --2- 172.21.15.97:0/1225921973 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fd19c038cb0 0x7fd19c03b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:03.027 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.994+0000 7fd1af400700 1 -- 172.21.15.97:0/1225921973 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd1a804cc60 con 0x7fd1b8105e10 2024-03-11T19:47:03.027 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.994+0000 7fd1b6000700 1 -- 172.21.15.97:0/1225921973 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fd19c038cb0 msgr2=0x7fd19c03b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3216682632 2024-03-11T19:47:03.027 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.994+0000 7fd1b6000700 1 --2- 172.21.15.97:0/1225921973 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fd19c038cb0 0x7fd19c03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:47:03.027 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:01.999+0000 7fd1af400700 1 -- 172.21.15.97:0/1225921973 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd1a8018dc0 con 0x7fd1b8105e10 2024-03-11T19:47:03.027 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.195+0000 7fd1b6000700 1 -- 172.21.15.97:0/1225921973 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fd19c038cb0 msgr2=0x7fd19c03b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3216682632 2024-03-11T19:47:03.027 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.195+0000 7fd1b6000700 1 --2- 172.21.15.97:0/1225921973 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fd19c038cb0 0x7fd19c03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T19:47:03.027 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.273+0000 7fd1bd600700 1 -- 172.21.15.97:0/1225921973 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd1b802da30 con 0x7fd1b8105e10 2024-03-11T19:47:03.027 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.274+0000 7fd1af400700 1 -- 172.21.15.97:0/1225921973 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fd1a8018be0 con 0x7fd1b8105e10 2024-03-11T19:47:03.027 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.277+0000 7fd1bd600700 1 -- 172.21.15.97:0/1225921973 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fd19c038cb0 msgr2=0x7fd19c03b170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:47:03.027 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.277+0000 7fd1bd600700 1 --2- 172.21.15.97:0/1225921973 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fd19c038cb0 0x7fd19c03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:03.028 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.277+0000 7fd1bd600700 1 -- 172.21.15.97:0/1225921973 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 msgr2=0x7fd1b819d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:03.028 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.277+0000 7fd1bd600700 1 --2- 172.21.15.97:0/1225921973 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 0x7fd1b819d4f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8011a70 tx=0x7fd1a8011b50 comp rx=0 tx=0).stop 2024-03-11T19:47:03.028 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.277+0000 7fd1bd600700 1 -- 172.21.15.97:0/1225921973 shutdown_connections 2024-03-11T19:47:03.028 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.277+0000 7fd1bd600700 1 --2- 172.21.15.97:0/1225921973 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fd19c038cb0 0x7fd19c03b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:03.028 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.277+0000 7fd1bd600700 1 --2- 172.21.15.97:0/1225921973 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1b8105e10 0x7fd1b819d4f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:03.028 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.277+0000 7fd1bd600700 1 -- 172.21.15.97:0/1225921973 >> 172.21.15.97:0/1225921973 conn(0x7fd1b8101a30 msgr2=0x7fd1b81942d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:03.028 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.277+0000 7fd1bd600700 1 -- 172.21.15.97:0/1225921973 shutdown_connections 2024-03-11T19:47:03.028 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:02.277+0000 7fd1bd600700 1 -- 172.21.15.97:0/1225921973 wait complete. 2024-03-11T19:47:03.028 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for the mgr to restart... 2024-03-11T19:47:03.028 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for mgr epoch 5... 2024-03-11T19:47:03.297 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:03 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:03.216+0000 7f3d6857c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T19:47:03.563 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:03 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:03.365+0000 7f3d6857c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T19:47:03.563 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:03 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:03.485+0000 7f3d6857c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T19:47:03.825 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:03 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:03.616+0000 7f3d6857c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T19:47:04.171 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:03 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:03.835+0000 7f3d6857c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T19:47:05.421 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:05 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:05.078+0000 7f3d6857c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T19:47:05.422 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:05 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:05.247+0000 7f3d6857c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T19:47:05.921 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:05 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:05.660+0000 7f3d6857c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T19:47:05.921 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:05 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:05.779+0000 7f3d6857c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T19:47:06.421 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:06 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:06.033+0000 7f3d6857c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:47:06.422 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:06 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:06.277+0000 7f3d6857c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T19:47:06.921 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:06 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:06.651+0000 7f3d6857c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:47:07.273 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:06 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:06.964+0000 7f3d6857c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T19:47:07.671 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:07 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:07.267+0000 7f3d6857c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T19:47:08.671 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:08.274+0000 7f3d6857c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T19:47:09.369 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:08.959+0000 7f3d6857c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T19:47:09.370 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:09.088+0000 7f3d6857c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T19:47:09.671 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:09.365+0000 7f3d6857c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T19:47:09.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-mon[31898]: Active manager daemon x restarted 2024-03-11T19:47:09.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-mon[31898]: Activating manager daemon x 2024-03-11T19:47:09.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-mon[31898]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T19:47:09.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-mon[31898]: mgrmap e6: x(active, starting, since 0.00946146s) 2024-03-11T19:47:09.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:47:09.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T19:47:09.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T19:47:09.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T19:47:09.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T19:47:09.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-mon[31898]: Manager daemon x is now available 2024-03-11T19:47:09.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:09.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:09 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:10.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:10 smithi097 ceph-mon[31898]: Found migration_current of "None". Setting to last migration. 2024-03-11T19:47:10.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:10 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:47:10.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:10 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:47:10.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:10 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:47:10.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:10 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T19:47:10.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:10 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T19:47:10.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:10 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:10.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:10 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:10.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:10 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:47:10.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:10 smithi097 ceph-mon[31898]: mgrmap e7: x(active, since 1.01649s) 2024-03-11T19:47:11.117 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-03-11T19:47:11.117 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T19:47:11.117 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T19:47:11.117 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-03-11T19:47:11.117 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.611+0000 7fb02a400700 1 Processor -- start 2024-03-11T19:47:11.117 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.611+0000 7fb02a400700 1 -- start start 2024-03-11T19:47:11.117 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.612+0000 7fb02a400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 0x7fb02407b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:11.117 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.612+0000 7fb02a400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02407b810 con 0x7fb02407cdd0 2024-03-11T19:47:11.117 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.612+0000 7fb023400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 0x7fb02407b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:11.117 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.612+0000 7fb023400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 0x7fb02407b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45318/0 (socket says 172.21.15.97:45318) 2024-03-11T19:47:11.117 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.612+0000 7fb023400700 1 -- 172.21.15.97:0/190024833 learned_addr learned my addr 172.21.15.97:0/190024833 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.613+0000 7fb023400700 1 -- 172.21.15.97:0/190024833 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02407b950 con 0x7fb02407cdd0 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.613+0000 7fb023400700 1 --2- 172.21.15.97:0/190024833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 0x7fb02407b2d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb0140098a0 tx=0x7fb014009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dcbd4f8aae7bdaa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.613+0000 7fb022000700 1 -- 172.21.15.97:0/190024833 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb014004160 con 0x7fb02407cdd0 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.613+0000 7fb022000700 1 -- 172.21.15.97:0/190024833 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0140042c0 con 0x7fb02407cdd0 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.613+0000 7fb022000700 1 -- 172.21.15.97:0/190024833 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb014011ac0 con 0x7fb02407cdd0 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.614+0000 7fb02a400700 1 -- 172.21.15.97:0/190024833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 msgr2=0x7fb02407b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.614+0000 7fb02a400700 1 --2- 172.21.15.97:0/190024833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 0x7fb02407b2d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb0140098a0 tx=0x7fb014009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.614+0000 7fb02a400700 1 -- 172.21.15.97:0/190024833 shutdown_connections 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.614+0000 7fb02a400700 1 --2- 172.21.15.97:0/190024833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 0x7fb02407b2d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.614+0000 7fb02a400700 1 -- 172.21.15.97:0/190024833 >> 172.21.15.97:0/190024833 conn(0x7fb024102bd0 msgr2=0x7fb024104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.615+0000 7fb02a400700 1 -- 172.21.15.97:0/190024833 shutdown_connections 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.615+0000 7fb02a400700 1 -- 172.21.15.97:0/190024833 wait complete. 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.616+0000 7fb02a400700 1 Processor -- start 2024-03-11T19:47:11.118 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.616+0000 7fb02a400700 1 -- start start 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.616+0000 7fb02a400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 0x7fb02419d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.616+0000 7fb02a400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02419da00 con 0x7fb02407cdd0 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.616+0000 7fb023400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 0x7fb02419d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.616+0000 7fb023400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 0x7fb02419d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45324/0 (socket says 172.21.15.97:45324) 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.616+0000 7fb023400700 1 -- 172.21.15.97:0/3550397401 learned_addr learned my addr 172.21.15.97:0/3550397401 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.617+0000 7fb023400700 1 -- 172.21.15.97:0/3550397401 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb014009550 con 0x7fb02407cdd0 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.617+0000 7fb023400700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 0x7fb02419d4c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb014011a70 tx=0x7fb014011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.617+0000 7fb01be00700 1 -- 172.21.15.97:0/3550397401 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0140039c0 con 0x7fb02407cdd0 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.617+0000 7fb01be00700 1 -- 172.21.15.97:0/3550397401 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb014003b20 con 0x7fb02407cdd0 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.617+0000 7fb01be00700 1 -- 172.21.15.97:0/3550397401 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb014024610 con 0x7fb02407cdd0 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.617+0000 7fb02a400700 1 -- 172.21.15.97:0/3550397401 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02419dc00 con 0x7fb02407cdd0 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.618+0000 7fb02a400700 1 -- 172.21.15.97:0/3550397401 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02419e0a0 con 0x7fb02407cdd0 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.619+0000 7fb01be00700 1 -- 172.21.15.97:0/3550397401 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fb01401a430 con 0x7fb02407cdd0 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.619+0000 7fb01be00700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 0x7fb00803b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:11.119 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.619+0000 7fb01be00700 1 -- 172.21.15.97:0/3550397401 --> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb00803b880 con 0x7fb008038cb0 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.619+0000 7fb01be00700 1 -- 172.21.15.97:0/3550397401 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb01404cc90 con 0x7fb02407cdd0 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.619+0000 7fb022a00700 1 -- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 msgr2=0x7fb00803b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3216682632 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.619+0000 7fb022a00700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 0x7fb00803b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.820+0000 7fb022a00700 1 -- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 msgr2=0x7fb00803b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3216682632 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:03.820+0000 7fb022a00700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 0x7fb00803b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:04.221+0000 7fb022a00700 1 -- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 msgr2=0x7fb00803b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3216682632 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:04.221+0000 7fb022a00700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 0x7fb00803b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:05.022+0000 7fb022a00700 1 -- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 msgr2=0x7fb00803b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3216682632 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:05.022+0000 7fb022a00700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 0x7fb00803b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:06.624+0000 7fb022a00700 1 -- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 msgr2=0x7fb00803b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3216682632 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:06.624+0000 7fb022a00700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 0x7fb00803b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:09.375+0000 7fb01be00700 1 -- 172.21.15.97:0/3550397401 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mgrmap(e 6) v1 ==== 45536+0+0 (secure 0 0 0) 0x7fb01401aa10 con 0x7fb02407cdd0 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:09.375+0000 7fb01be00700 1 -- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 msgr2=0x7fb00803b170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:47:11.120 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:09.375+0000 7fb01be00700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 0x7fb00803b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.379+0000 7fb01be00700 1 -- 172.21.15.97:0/3550397401 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mgrmap(e 7) v1 ==== 45663+0+0 (secure 0 0 0) 0x7fb01400fd80 con 0x7fb02407cdd0 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.379+0000 7fb01be00700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fb00803c6c0 0x7fb00803eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.379+0000 7fb01be00700 1 -- 172.21.15.97:0/3550397401 --> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb00803b880 con 0x7fb00803c6c0 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.381+0000 7fb022a00700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fb00803c6c0 0x7fb00803eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.384+0000 7fb022a00700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fb00803c6c0 0x7fb00803eab0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb010003a10 tx=0x7fb010009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.386+0000 7fb01be00700 1 -- 172.21.15.97:0/3550397401 <== mgr.14120 v2:172.21.15.97:6800/1708150324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fb00803b880 con 0x7fb00803c6c0 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.394+0000 7fb02a400700 1 -- 172.21.15.97:0/3550397401 --> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb02410d8a0 con 0x7fb00803c6c0 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.395+0000 7fb01be00700 1 -- 172.21.15.97:0/3550397401 <== mgr.14120 v2:172.21.15.97:6800/1708150324 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb02410d8a0 con 0x7fb00803c6c0 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.396+0000 7fb02a400700 1 -- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fb00803c6c0 msgr2=0x7fb00803eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.396+0000 7fb02a400700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fb00803c6c0 0x7fb00803eab0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb010003a10 tx=0x7fb010009290 comp rx=0 tx=0).stop 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.396+0000 7fb02a400700 1 -- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 msgr2=0x7fb02419d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.396+0000 7fb02a400700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 0x7fb02419d4c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb014011a70 tx=0x7fb014011b50 comp rx=0 tx=0).stop 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.396+0000 7fb02a400700 1 -- 172.21.15.97:0/3550397401 shutdown_connections 2024-03-11T19:47:11.121 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.396+0000 7fb02a400700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb02407cdd0 0x7fb02419d4c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:11.122 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.396+0000 7fb02a400700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fb00803c6c0 0x7fb00803eab0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:11.122 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.396+0000 7fb02a400700 1 --2- 172.21.15.97:0/3550397401 >> [v2:172.21.15.97:6800/3216682632,v1:172.21.15.97:6801/3216682632] conn(0x7fb008038cb0 0x7fb00803b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:11.122 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.396+0000 7fb02a400700 1 -- 172.21.15.97:0/3550397401 >> 172.21.15.97:0/3550397401 conn(0x7fb024102bd0 msgr2=0x7fb024103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:11.122 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.396+0000 7fb02a400700 1 -- 172.21.15.97:0/3550397401 shutdown_connections 2024-03-11T19:47:11.122 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:10.396+0000 7fb02a400700 1 -- 172.21.15.97:0/3550397401 wait complete. 2024-03-11T19:47:11.122 INFO:teuthology.orchestra.run.smithi097.stdout:mgr epoch 5 is available 2024-03-11T19:47:11.122 INFO:teuthology.orchestra.run.smithi097.stdout:Setting orchestrator backend to cephadm... 2024-03-11T19:47:11.695 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:11 smithi097 ceph-mon[31898]: [11/Mar/2024:19:47:09] ENGINE Bus STARTING 2024-03-11T19:47:11.695 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:11 smithi097 ceph-mon[31898]: [11/Mar/2024:19:47:09] ENGINE Serving on https://172.21.15.97:7150 2024-03-11T19:47:11.695 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:11 smithi097 ceph-mon[31898]: [11/Mar/2024:19:47:10] ENGINE Serving on http://172.21.15.97:8765 2024-03-11T19:47:11.695 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:11 smithi097 ceph-mon[31898]: [11/Mar/2024:19:47:10] ENGINE Bus STARTED 2024-03-11T19:47:11.695 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:11 smithi097 ceph-mon[31898]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T19:47:11.695 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:11 smithi097 ceph-mon[31898]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T19:47:12.656 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:12 smithi097 ceph-mon[31898]: mgrmap e8: x(active, since 2s) 2024-03-11T19:47:12.657 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:12 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:12.657 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:12 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:47:12.665 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.721+0000 7f1649a00700 1 Processor -- start 2024-03-11T19:47:12.665 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.721+0000 7f1649a00700 1 -- start start 2024-03-11T19:47:12.665 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.722+0000 7f1649a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 0x7f164410e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:12.665 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.722+0000 7f1649a00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164410e8a0 con 0x7f164410df40 2024-03-11T19:47:12.666 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.722+0000 7f1643e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 0x7f164410e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:12.666 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.722+0000 7f1643e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 0x7f164410e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56636/0 (socket says 172.21.15.97:56636) 2024-03-11T19:47:12.666 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.722+0000 7f1643e00700 1 -- 172.21.15.97:0/1864378271 learned_addr learned my addr 172.21.15.97:0/1864378271 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:12.666 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.723+0000 7f1643e00700 1 -- 172.21.15.97:0/1864378271 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164410e9e0 con 0x7f164410df40 2024-03-11T19:47:12.666 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.723+0000 7f1643e00700 1 --2- 172.21.15.97:0/1864378271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 0x7f164410e360 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f16340098a0 tx=0x7f1634009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f8d06a5494212c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:12.666 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.723+0000 7f1642a00700 1 -- 172.21.15.97:0/1864378271 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1634004160 con 0x7f164410df40 2024-03-11T19:47:12.670 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.723+0000 7f1642a00700 1 -- 172.21.15.97:0/1864378271 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16340042c0 con 0x7f164410df40 2024-03-11T19:47:12.671 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.723+0000 7f1642a00700 1 -- 172.21.15.97:0/1864378271 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1634011ac0 con 0x7f164410df40 2024-03-11T19:47:12.671 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.724+0000 7f1649a00700 1 -- 172.21.15.97:0/1864378271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 msgr2=0x7f164410e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:12.671 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.724+0000 7f1649a00700 1 --2- 172.21.15.97:0/1864378271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 0x7f164410e360 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f16340098a0 tx=0x7f1634009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:12.671 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.724+0000 7f1649a00700 1 -- 172.21.15.97:0/1864378271 shutdown_connections 2024-03-11T19:47:12.671 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.724+0000 7f1649a00700 1 --2- 172.21.15.97:0/1864378271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 0x7f164410e360 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:12.671 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.724+0000 7f1649a00700 1 -- 172.21.15.97:0/1864378271 >> 172.21.15.97:0/1864378271 conn(0x7f164407cc00 msgr2=0x7f164407f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:12.671 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.724+0000 7f1649a00700 1 -- 172.21.15.97:0/1864378271 shutdown_connections 2024-03-11T19:47:12.671 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.725+0000 7f1649a00700 1 -- 172.21.15.97:0/1864378271 wait complete. 2024-03-11T19:47:12.671 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.726+0000 7f1649a00700 1 Processor -- start 2024-03-11T19:47:12.671 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.726+0000 7f1649a00700 1 -- start start 2024-03-11T19:47:12.671 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.726+0000 7f1649a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 0x7f16441a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:12.671 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.726+0000 7f1649a00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16441a1cd0 con 0x7f164410df40 2024-03-11T19:47:12.671 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.726+0000 7f1643e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 0x7f16441a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.726+0000 7f1643e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 0x7f16441a1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56644/0 (socket says 172.21.15.97:56644) 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.726+0000 7f1643e00700 1 -- 172.21.15.97:0/1757273286 learned_addr learned my addr 172.21.15.97:0/1757273286 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.727+0000 7f1643e00700 1 -- 172.21.15.97:0/1757273286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1634009550 con 0x7f164410df40 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.727+0000 7f1643e00700 1 --2- 172.21.15.97:0/1757273286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 0x7f16441a1790 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1634003f70 tx=0x7f1634011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.727+0000 7f1640c00700 1 -- 172.21.15.97:0/1757273286 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16340039f0 con 0x7f164410df40 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.727+0000 7f1640c00700 1 -- 172.21.15.97:0/1757273286 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1634003b50 con 0x7f164410df40 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.727+0000 7f1649a00700 1 -- 172.21.15.97:0/1757273286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16441a1ed0 con 0x7f164410df40 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.727+0000 7f1640c00700 1 -- 172.21.15.97:0/1757273286 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f163401a690 con 0x7f164410df40 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.727+0000 7f1649a00700 1 -- 172.21.15.97:0/1757273286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16441a2370 con 0x7f164410df40 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.728+0000 7f1640c00700 1 -- 172.21.15.97:0/1757273286 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f1634024430 con 0x7f164410df40 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.729+0000 7f1649a00700 1 -- 172.21.15.97:0/1757273286 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164410f8a0 con 0x7f164410df40 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.729+0000 7f1640c00700 1 --2- 172.21.15.97:0/1757273286 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f1628038c60 0x7f162803b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.729+0000 7f1640c00700 1 -- 172.21.15.97:0/1757273286 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f163404cfc0 con 0x7f164410df40 2024-03-11T19:47:12.672 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.729+0000 7f1643400700 1 --2- 172.21.15.97:0/1757273286 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f1628038c60 0x7f162803b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:12.673 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.730+0000 7f1643400700 1 --2- 172.21.15.97:0/1757273286 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f1628038c60 0x7f162803b120 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1630006fd0 tx=0x7f1630006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:12.673 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.735+0000 7f1640c00700 1 -- 172.21.15.97:0/1757273286 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1634018a40 con 0x7f164410df40 2024-03-11T19:47:12.673 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.970+0000 7f1649a00700 1 -- 172.21.15.97:0/1757273286 --> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f164407ee90 con 0x7f1628038c60 2024-03-11T19:47:12.673 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.986+0000 7f1640c00700 1 -- 172.21.15.97:0/1757273286 <== mgr.14120 v2:172.21.15.97:6800/1708150324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f164407ee90 con 0x7f1628038c60 2024-03-11T19:47:12.673 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.989+0000 7f1649a00700 1 -- 172.21.15.97:0/1757273286 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f1628038c60 msgr2=0x7f162803b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:12.673 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.989+0000 7f1649a00700 1 --2- 172.21.15.97:0/1757273286 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f1628038c60 0x7f162803b120 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1630006fd0 tx=0x7f1630006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:12.673 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.989+0000 7f1649a00700 1 -- 172.21.15.97:0/1757273286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 msgr2=0x7f16441a1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:12.673 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.989+0000 7f1649a00700 1 --2- 172.21.15.97:0/1757273286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 0x7f16441a1790 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1634003f70 tx=0x7f1634011b40 comp rx=0 tx=0).stop 2024-03-11T19:47:12.673 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.990+0000 7f1649a00700 1 -- 172.21.15.97:0/1757273286 shutdown_connections 2024-03-11T19:47:12.674 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.990+0000 7f1649a00700 1 --2- 172.21.15.97:0/1757273286 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f1628038c60 0x7f162803b120 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:12.674 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.990+0000 7f1649a00700 1 --2- 172.21.15.97:0/1757273286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f164410df40 0x7f16441a1790 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:12.674 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.990+0000 7f1649a00700 1 -- 172.21.15.97:0/1757273286 >> 172.21.15.97:0/1757273286 conn(0x7f164407cc00 msgr2=0x7f164407e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:12.674 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.990+0000 7f1649a00700 1 -- 172.21.15.97:0/1757273286 shutdown_connections 2024-03-11T19:47:12.674 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:11.990+0000 7f1649a00700 1 -- 172.21.15.97:0/1757273286 wait complete. 2024-03-11T19:47:13.522 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:13 smithi097 ceph-mon[31898]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:47:14.214 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T19:47:14.217 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.263+0000 7fa436800700 1 Processor -- start 2024-03-11T19:47:14.217 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.263+0000 7fa436800700 1 -- start start 2024-03-11T19:47:14.217 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.264+0000 7fa436800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 0x7fa43007b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:14.217 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.264+0000 7fa436800700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43007b850 con 0x7fa43007ce10 2024-03-11T19:47:14.217 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.264+0000 7fa42f400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 0x7fa43007b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:14.217 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.264+0000 7fa42f400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 0x7fa43007b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56656/0 (socket says 172.21.15.97:56656) 2024-03-11T19:47:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.264+0000 7fa42f400700 1 -- 172.21.15.97:0/1125799890 learned_addr learned my addr 172.21.15.97:0/1125799890 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.265+0000 7fa42f400700 1 -- 172.21.15.97:0/1125799890 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43007b990 con 0x7fa43007ce10 2024-03-11T19:47:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.265+0000 7fa42f400700 1 --2- 172.21.15.97:0/1125799890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 0x7fa43007b310 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa4200098a0 tx=0x7fa420009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f058b6b2df26316a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.266+0000 7fa42e000700 1 -- 172.21.15.97:0/1125799890 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa420004160 con 0x7fa43007ce10 2024-03-11T19:47:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.266+0000 7fa42e000700 1 -- 172.21.15.97:0/1125799890 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4200042c0 con 0x7fa43007ce10 2024-03-11T19:47:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.266+0000 7fa42e000700 1 -- 172.21.15.97:0/1125799890 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa420011ac0 con 0x7fa43007ce10 2024-03-11T19:47:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.266+0000 7fa436800700 1 -- 172.21.15.97:0/1125799890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 msgr2=0x7fa43007b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.266+0000 7fa436800700 1 --2- 172.21.15.97:0/1125799890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 0x7fa43007b310 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa4200098a0 tx=0x7fa420009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.266+0000 7fa436800700 1 -- 172.21.15.97:0/1125799890 shutdown_connections 2024-03-11T19:47:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.266+0000 7fa436800700 1 --2- 172.21.15.97:0/1125799890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 0x7fa43007b310 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.266+0000 7fa436800700 1 -- 172.21.15.97:0/1125799890 >> 172.21.15.97:0/1125799890 conn(0x7fa430102bd0 msgr2=0x7fa430104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.267+0000 7fa436800700 1 -- 172.21.15.97:0/1125799890 shutdown_connections 2024-03-11T19:47:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.267+0000 7fa436800700 1 -- 172.21.15.97:0/1125799890 wait complete. 2024-03-11T19:47:14.219 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.268+0000 7fa436800700 1 Processor -- start 2024-03-11T19:47:14.219 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.268+0000 7fa436800700 1 -- start start 2024-03-11T19:47:14.219 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.268+0000 7fa436800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 0x7fa43019d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:14.219 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.268+0000 7fa436800700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43019d9e0 con 0x7fa43007ce10 2024-03-11T19:47:14.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.268+0000 7fa42f400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 0x7fa43019d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:14.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.268+0000 7fa42f400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 0x7fa43019d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56666/0 (socket says 172.21.15.97:56666) 2024-03-11T19:47:14.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.268+0000 7fa42f400700 1 -- 172.21.15.97:0/115301524 learned_addr learned my addr 172.21.15.97:0/115301524 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:14.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.269+0000 7fa42f400700 1 -- 172.21.15.97:0/115301524 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa420009550 con 0x7fa43007ce10 2024-03-11T19:47:14.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.269+0000 7fa42f400700 1 --2- 172.21.15.97:0/115301524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 0x7fa43019d4a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa420011a70 tx=0x7fa420011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:14.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.269+0000 7fa427e00700 1 -- 172.21.15.97:0/115301524 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa4200039c0 con 0x7fa43007ce10 2024-03-11T19:47:14.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.269+0000 7fa427e00700 1 -- 172.21.15.97:0/115301524 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa420003b20 con 0x7fa43007ce10 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.269+0000 7fa436800700 1 -- 172.21.15.97:0/115301524 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43019dbe0 con 0x7fa43007ce10 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.269+0000 7fa427e00700 1 -- 172.21.15.97:0/115301524 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa420024610 con 0x7fa43007ce10 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.270+0000 7fa436800700 1 -- 172.21.15.97:0/115301524 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43019e080 con 0x7fa43007ce10 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.271+0000 7fa436800700 1 -- 172.21.15.97:0/115301524 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa43010d8c0 con 0x7fa43007ce10 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.271+0000 7fa427e00700 1 -- 172.21.15.97:0/115301524 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fa42001aa10 con 0x7fa43007ce10 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.271+0000 7fa427e00700 1 --2- 172.21.15.97:0/115301524 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fa414038d00 0x7fa41403b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.271+0000 7fa427e00700 1 -- 172.21.15.97:0/115301524 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa42004cbe0 con 0x7fa43007ce10 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.272+0000 7fa42ea00700 1 --2- 172.21.15.97:0/115301524 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fa414038d00 0x7fa41403b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.273+0000 7fa42ea00700 1 --2- 172.21.15.97:0/115301524 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fa414038d00 0x7fa41403b1c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa41c006fd0 tx=0x7fa41c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.277+0000 7fa427e00700 1 -- 172.21.15.97:0/115301524 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa420018dc0 con 0x7fa43007ce10 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.497+0000 7fa436800700 1 -- 172.21.15.97:0/115301524 --> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fa430103e50 con 0x7fa414038d00 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.499+0000 7fa427e00700 1 -- 172.21.15.97:0/115301524 <== mgr.14120 v2:172.21.15.97:6800/1708150324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fa430103e50 con 0x7fa414038d00 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.501+0000 7fa436800700 1 -- 172.21.15.97:0/115301524 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fa414038d00 msgr2=0x7fa41403b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.501+0000 7fa436800700 1 --2- 172.21.15.97:0/115301524 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fa414038d00 0x7fa41403b1c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa41c006fd0 tx=0x7fa41c006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:14.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.501+0000 7fa436800700 1 -- 172.21.15.97:0/115301524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 msgr2=0x7fa43019d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:14.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.501+0000 7fa436800700 1 --2- 172.21.15.97:0/115301524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 0x7fa43019d4a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa420011a70 tx=0x7fa420011b50 comp rx=0 tx=0).stop 2024-03-11T19:47:14.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.501+0000 7fa436800700 1 -- 172.21.15.97:0/115301524 shutdown_connections 2024-03-11T19:47:14.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.501+0000 7fa436800700 1 --2- 172.21.15.97:0/115301524 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fa414038d00 0x7fa41403b1c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:14.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.501+0000 7fa436800700 1 --2- 172.21.15.97:0/115301524 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa43007ce10 0x7fa43019d4a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:14.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.501+0000 7fa436800700 1 -- 172.21.15.97:0/115301524 >> 172.21.15.97:0/115301524 conn(0x7fa430102bd0 msgr2=0x7fa430103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:14.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.501+0000 7fa436800700 1 -- 172.21.15.97:0/115301524 shutdown_connections 2024-03-11T19:47:14.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:13.502+0000 7fa436800700 1 -- 172.21.15.97:0/115301524 wait complete. 2024-03-11T19:47:14.224 INFO:teuthology.orchestra.run.smithi097.stdout:Generating ssh key... 2024-03-11T19:47:14.553 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:14 smithi097 ceph-mon[31898]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:47:15.583 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: Generating public/private rsa key pair. 2024-03-11T19:47:15.583 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: Your identification has been saved in /tmp/tmp6xjt10zu/key. 2024-03-11T19:47:15.583 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: Your public key has been saved in /tmp/tmp6xjt10zu/key.pub. 2024-03-11T19:47:15.583 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: The key fingerprint is: 2024-03-11T19:47:15.583 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: SHA256:K/fNO/LQpdoWrH49xPrmaeG8ioNoIIV+j7JyFOktuEQ ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:47:15.583 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: The key's randomart image is: 2024-03-11T19:47:15.583 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: +---[RSA 3072]----+ 2024-03-11T19:47:15.583 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: | | 2024-03-11T19:47:15.583 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: | | 2024-03-11T19:47:15.583 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: | o | 2024-03-11T19:47:15.583 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: | E+ . | 2024-03-11T19:47:15.583 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: |.+ + S . .. | 2024-03-11T19:47:15.584 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: |..* + . .oo+ | 2024-03-11T19:47:15.584 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: |.o + +..o...oB . | 2024-03-11T19:47:15.584 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: |o o . +o..+B= B. | 2024-03-11T19:47:15.584 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: | o.o . .=*BO=o | 2024-03-11T19:47:15.584 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: +----[SHA256]-----+ 2024-03-11T19:47:15.877 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.838+0000 7f8a10600700 1 Processor -- start 2024-03-11T19:47:15.883 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.838+0000 7f8a10600700 1 -- start start 2024-03-11T19:47:15.883 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.839+0000 7f8a10600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 0x7f8a0807b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:15.883 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.839+0000 7f8a10600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0807b830 con 0x7f8a0807cdf0 2024-03-11T19:47:15.883 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.839+0000 7f8a0dc00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 0x7f8a0807b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:15.883 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.839+0000 7f8a0dc00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 0x7f8a0807b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56214/0 (socket says 172.21.15.97:56214) 2024-03-11T19:47:15.883 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.839+0000 7f8a0dc00700 1 -- 172.21.15.97:0/834766561 learned_addr learned my addr 172.21.15.97:0/834766561 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:15.884 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.840+0000 7f8a0dc00700 1 -- 172.21.15.97:0/834766561 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0807b970 con 0x7f8a0807cdf0 2024-03-11T19:47:15.884 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.840+0000 7f8a0dc00700 1 --2- 172.21.15.97:0/834766561 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 0x7f8a0807b2f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f89fc0098a0 tx=0x7f89fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935dd3319c33c383 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:15.884 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.840+0000 7f8a07e00700 1 -- 172.21.15.97:0/834766561 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f89fc004160 con 0x7f8a0807cdf0 2024-03-11T19:47:15.884 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.840+0000 7f8a07e00700 1 -- 172.21.15.97:0/834766561 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f89fc0042c0 con 0x7f8a0807cdf0 2024-03-11T19:47:15.884 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.840+0000 7f8a07e00700 1 -- 172.21.15.97:0/834766561 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f89fc011ac0 con 0x7f8a0807cdf0 2024-03-11T19:47:15.884 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.841+0000 7f8a10600700 1 -- 172.21.15.97:0/834766561 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 msgr2=0x7f8a0807b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:15.884 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.841+0000 7f8a10600700 1 --2- 172.21.15.97:0/834766561 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 0x7f8a0807b2f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f89fc0098a0 tx=0x7f89fc009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:15.884 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.841+0000 7f8a10600700 1 -- 172.21.15.97:0/834766561 shutdown_connections 2024-03-11T19:47:15.884 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.841+0000 7f8a10600700 1 --2- 172.21.15.97:0/834766561 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 0x7f8a0807b2f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:15.884 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.841+0000 7f8a10600700 1 -- 172.21.15.97:0/834766561 >> 172.21.15.97:0/834766561 conn(0x7f8a08102bd0 msgr2=0x7f8a08104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:15.884 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.841+0000 7f8a10600700 1 -- 172.21.15.97:0/834766561 shutdown_connections 2024-03-11T19:47:15.884 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.841+0000 7f8a10600700 1 -- 172.21.15.97:0/834766561 wait complete. 2024-03-11T19:47:15.884 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.842+0000 7f8a10600700 1 Processor -- start 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.843+0000 7f8a10600700 1 -- start start 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.843+0000 7f8a10600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 0x7f8a0819d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.843+0000 7f8a10600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0819da00 con 0x7f8a0807cdf0 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.843+0000 7f8a0dc00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 0x7f8a0819d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.843+0000 7f8a0dc00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 0x7f8a0819d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56222/0 (socket says 172.21.15.97:56222) 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.843+0000 7f8a0dc00700 1 -- 172.21.15.97:0/1034221599 learned_addr learned my addr 172.21.15.97:0/1034221599 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.843+0000 7f8a0dc00700 1 -- 172.21.15.97:0/1034221599 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89fc009550 con 0x7f8a0807cdf0 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.844+0000 7f8a0dc00700 1 --2- 172.21.15.97:0/1034221599 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 0x7f8a0819d4c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f89fc011a70 tx=0x7f89fc011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.844+0000 7f8a06000700 1 -- 172.21.15.97:0/1034221599 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f89fc0039c0 con 0x7f8a0807cdf0 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.844+0000 7f8a06000700 1 -- 172.21.15.97:0/1034221599 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f89fc003b20 con 0x7f8a0807cdf0 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.844+0000 7f8a06000700 1 -- 172.21.15.97:0/1034221599 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f89fc024610 con 0x7f8a0807cdf0 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.844+0000 7f8a10600700 1 -- 172.21.15.97:0/1034221599 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0819dc00 con 0x7f8a0807cdf0 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.844+0000 7f8a10600700 1 -- 172.21.15.97:0/1034221599 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0819e0a0 con 0x7f8a0807cdf0 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.845+0000 7f8a06000700 1 -- 172.21.15.97:0/1034221599 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f89fc01aa10 con 0x7f8a0807cdf0 2024-03-11T19:47:15.885 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.845+0000 7f8a10600700 1 -- 172.21.15.97:0/1034221599 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0810d880 con 0x7f8a0807cdf0 2024-03-11T19:47:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.846+0000 7f8a06000700 1 --2- 172.21.15.97:0/1034221599 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f89f0038d00 0x7f89f003b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.846+0000 7f8a06000700 1 -- 172.21.15.97:0/1034221599 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f89fc04cd10 con 0x7f8a0807cdf0 2024-03-11T19:47:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.846+0000 7f8a0d200700 1 --2- 172.21.15.97:0/1034221599 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f89f0038d00 0x7f89f003b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.847+0000 7f8a0d200700 1 --2- 172.21.15.97:0/1034221599 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f89f0038d00 0x7f89f003b1c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f89f4006fd0 tx=0x7f89f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:14.851+0000 7f8a06000700 1 -- 172.21.15.97:0/1034221599 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f89fc01a430 con 0x7f8a0807cdf0 2024-03-11T19:47:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:15.068+0000 7f8a10600700 1 -- 172.21.15.97:0/1034221599 --> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a08103e60 con 0x7f89f0038d00 2024-03-11T19:47:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:15.224+0000 7f8a06000700 1 -- 172.21.15.97:0/1034221599 <== mgr.14120 v2:172.21.15.97:6800/1708150324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8a08103e60 con 0x7f89f0038d00 2024-03-11T19:47:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:15.226+0000 7f8a10600700 1 -- 172.21.15.97:0/1034221599 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f89f0038d00 msgr2=0x7f89f003b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:15.226+0000 7f8a10600700 1 --2- 172.21.15.97:0/1034221599 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f89f0038d00 0x7f89f003b1c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f89f4006fd0 tx=0x7f89f4006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:15.226+0000 7f8a10600700 1 -- 172.21.15.97:0/1034221599 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 msgr2=0x7f8a0819d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:15.226+0000 7f8a10600700 1 --2- 172.21.15.97:0/1034221599 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 0x7f8a0819d4c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f89fc011a70 tx=0x7f89fc011b50 comp rx=0 tx=0).stop 2024-03-11T19:47:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:15.226+0000 7f8a10600700 1 -- 172.21.15.97:0/1034221599 shutdown_connections 2024-03-11T19:47:15.886 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:15.226+0000 7f8a10600700 1 --2- 172.21.15.97:0/1034221599 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f89f0038d00 0x7f89f003b1c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:15.887 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:15.226+0000 7f8a10600700 1 --2- 172.21.15.97:0/1034221599 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8a0807cdf0 0x7f8a0819d4c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:15.887 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:15.226+0000 7f8a10600700 1 -- 172.21.15.97:0/1034221599 >> 172.21.15.97:0/1034221599 conn(0x7f8a08102bd0 msgr2=0x7f8a08103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:15.887 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:15.227+0000 7f8a10600700 1 -- 172.21.15.97:0/1034221599 shutdown_connections 2024-03-11T19:47:15.887 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:15.227+0000 7f8a10600700 1 -- 172.21.15.97:0/1034221599 wait complete. 2024-03-11T19:47:16.499 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:16 smithi097 ceph-mon[31898]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:47:16.499 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:16 smithi097 ceph-mon[31898]: Generating ssh key... 2024-03-11T19:47:16.499 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:16 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:16.499 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:16 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:17.490 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDunaBTS23ZpNBCk9u6s5TwTfwvmYdlHlINjgLXa/dM4yKbv1/qlv/AZ3waIglSfww/e6nN/aoPEM6FZrN0Qk+YGnIs4qdo73JYkknG//UZaO1lnJdIGaFSRyVdPkbj3pm8K8h1btdUWqTCJI5utQpRrjDprRq6oaS6/aVCquvvXOC8O4/aqc0Lkzj+hmA0MRMX5aJuOKjPnUBPjiFk7hZHQKp+WIg+SLW67SB7h1oiiJrL4IYS0EPM7iITiHM9BBDnq4IP9K0bIDX50Zmh9MVceR92BfMK/waZTEA9ks39LwktWhY0e3mMujW9lypb7rkrZOT69NoL/oTwT6Dk0LOUBJZ7yVWX/C+IgcwWAGv7uslecqJPC899K1U0xnbPvYZsK1LdS+W365Uhv/bK3qWuVGgacpIUZ+pB6p9cBkIQfuZWEe5Wi+D7oL5nQi85ogxgmjLYUaQUr7USyoWEvAXgaX62aGOle9NCSzd/+ftkP/8PIpMg7+8OeTBEpXfdGxs= ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:47:17.491 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.494+0000 7f34ed400700 1 Processor -- start 2024-03-11T19:47:17.491 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.494+0000 7f34ed400700 1 -- start start 2024-03-11T19:47:17.491 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.494+0000 7f34ed400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 0x7f34e810a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:17.491 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.494+0000 7f34ed400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e807ae60 con 0x7f34e81080e0 2024-03-11T19:47:17.491 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.495+0000 7f34e6800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 0x7f34e810a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:17.491 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.495+0000 7f34e6800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 0x7f34e810a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56232/0 (socket says 172.21.15.97:56232) 2024-03-11T19:47:17.491 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.495+0000 7f34e6800700 1 -- 172.21.15.97:0/3851571954 learned_addr learned my addr 172.21.15.97:0/3851571954 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:17.491 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.495+0000 7f34e6800700 1 -- 172.21.15.97:0/3851571954 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e810aa10 con 0x7f34e81080e0 2024-03-11T19:47:17.491 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.496+0000 7f34e6800700 1 --2- 172.21.15.97:0/3851571954 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 0x7f34e810a4d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009e80 tx=0x7f34d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db11283f1aaaf54 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:17.491 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.496+0000 7f34e5e00700 1 -- 172.21.15.97:0/3851571954 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34d800e070 con 0x7f34e81080e0 2024-03-11T19:47:17.491 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.496+0000 7f34e5e00700 1 -- 172.21.15.97:0/3851571954 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34d800b8c0 con 0x7f34e81080e0 2024-03-11T19:47:17.491 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.496+0000 7f34e5e00700 1 -- 172.21.15.97:0/3851571954 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34d8004550 con 0x7f34e81080e0 2024-03-11T19:47:17.491 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.496+0000 7f34ed400700 1 -- 172.21.15.97:0/3851571954 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 msgr2=0x7f34e810a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.496+0000 7f34ed400700 1 --2- 172.21.15.97:0/3851571954 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 0x7f34e810a4d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009e80 tx=0x7f34d800b330 comp rx=0 tx=0).stop 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.497+0000 7f34ed400700 1 -- 172.21.15.97:0/3851571954 shutdown_connections 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.497+0000 7f34ed400700 1 --2- 172.21.15.97:0/3851571954 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 0x7f34e810a4d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.497+0000 7f34ed400700 1 -- 172.21.15.97:0/3851571954 >> 172.21.15.97:0/3851571954 conn(0x7f34e8101b10 msgr2=0x7f34e8103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.497+0000 7f34ed400700 1 -- 172.21.15.97:0/3851571954 shutdown_connections 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.497+0000 7f34ed400700 1 -- 172.21.15.97:0/3851571954 wait complete. 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.498+0000 7f34ed400700 1 Processor -- start 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.498+0000 7f34ed400700 1 -- start start 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.498+0000 7f34ed400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 0x7f34e819d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.498+0000 7f34ed400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e819d960 con 0x7f34e81080e0 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.499+0000 7f34e6800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 0x7f34e819d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.499+0000 7f34e6800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 0x7f34e819d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56234/0 (socket says 172.21.15.97:56234) 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.499+0000 7f34e6800700 1 -- 172.21.15.97:0/2656657199 learned_addr learned my addr 172.21.15.97:0/2656657199 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:17.492 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.499+0000 7f34e6800700 1 -- 172.21.15.97:0/2656657199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d8009b30 con 0x7f34e81080e0 2024-03-11T19:47:17.493 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.499+0000 7f34e6800700 1 --2- 172.21.15.97:0/2656657199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 0x7f34e819d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009fd0 tx=0x7f34d8003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:17.493 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.500+0000 7f34dfe00700 1 -- 172.21.15.97:0/2656657199 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34d800e040 con 0x7f34e81080e0 2024-03-11T19:47:17.493 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.500+0000 7f34dfe00700 1 -- 172.21.15.97:0/2656657199 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34d80128f0 con 0x7f34e81080e0 2024-03-11T19:47:17.493 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.500+0000 7f34ed400700 1 -- 172.21.15.97:0/2656657199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e819db60 con 0x7f34e81080e0 2024-03-11T19:47:17.493 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.500+0000 7f34dfe00700 1 -- 172.21.15.97:0/2656657199 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34d801c890 con 0x7f34e81080e0 2024-03-11T19:47:17.493 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.500+0000 7f34ed400700 1 -- 172.21.15.97:0/2656657199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e819dfe0 con 0x7f34e81080e0 2024-03-11T19:47:17.493 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.501+0000 7f34dfe00700 1 -- 172.21.15.97:0/2656657199 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f34d801c9f0 con 0x7f34e81080e0 2024-03-11T19:47:17.493 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.501+0000 7f34dca00700 1 -- 172.21.15.97:0/2656657199 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c0005330 con 0x7f34e81080e0 2024-03-11T19:47:17.493 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.502+0000 7f34dfe00700 1 --2- 172.21.15.97:0/2656657199 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f34cc038d00 0x7f34cc03b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:17.493 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.502+0000 7f34dfe00700 1 -- 172.21.15.97:0/2656657199 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f34d804cf30 con 0x7f34e81080e0 2024-03-11T19:47:17.493 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.502+0000 7f34dde00700 1 --2- 172.21.15.97:0/2656657199 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f34cc038d00 0x7f34cc03b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:17.493 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.503+0000 7f34dde00700 1 --2- 172.21.15.97:0/2656657199 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f34cc038d00 0x7f34cc03b1c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f34d0006fd0 tx=0x7f34d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:17.493 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.507+0000 7f34dfe00700 1 -- 172.21.15.97:0/2656657199 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f34d8024eb0 con 0x7f34e81080e0 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.724+0000 7f34dca00700 1 -- 172.21.15.97:0/2656657199 --> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f34c0000bc0 con 0x7f34cc038d00 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.725+0000 7f34dfe00700 1 -- 172.21.15.97:0/2656657199 <== mgr.14120 v2:172.21.15.97:6800/1708150324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f34c0000bc0 con 0x7f34cc038d00 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.727+0000 7f34dca00700 1 -- 172.21.15.97:0/2656657199 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f34cc038d00 msgr2=0x7f34cc03b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.727+0000 7f34dca00700 1 --2- 172.21.15.97:0/2656657199 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f34cc038d00 0x7f34cc03b1c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f34d0006fd0 tx=0x7f34d0006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.727+0000 7f34dca00700 1 -- 172.21.15.97:0/2656657199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 msgr2=0x7f34e819d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.727+0000 7f34dca00700 1 --2- 172.21.15.97:0/2656657199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 0x7f34e819d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009fd0 tx=0x7f34d8003d70 comp rx=0 tx=0).stop 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.728+0000 7f34dca00700 1 -- 172.21.15.97:0/2656657199 shutdown_connections 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.728+0000 7f34dca00700 1 --2- 172.21.15.97:0/2656657199 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f34cc038d00 0x7f34cc03b1c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.728+0000 7f34dca00700 1 --2- 172.21.15.97:0/2656657199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34e81080e0 0x7f34e819d420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.728+0000 7f34dca00700 1 -- 172.21.15.97:0/2656657199 >> 172.21.15.97:0/2656657199 conn(0x7f34e8101b10 msgr2=0x7f34e81024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.728+0000 7f34dca00700 1 -- 172.21.15.97:0/2656657199 shutdown_connections 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:16.728+0000 7f34dca00700 1 -- 172.21.15.97:0/2656657199 wait complete. 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T19:47:17.494 INFO:teuthology.orchestra.run.smithi097.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T19:47:17.495 INFO:teuthology.orchestra.run.smithi097.stdout:Adding host smithi097... 2024-03-11T19:47:18.038 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:17 smithi097 ceph-mon[31898]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:47:19.115 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:18 smithi097 ceph-mon[31898]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi097", "addr": "172.21.15.97", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:47:21.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:20 smithi097 ceph-mon[31898]: Deploying cephadm binary to smithi097 2024-03-11T19:47:25.538 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout Added host 'smithi097' with addr '172.21.15.97' 2024-03-11T19:47:25.538 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.188+0000 7f9b93200700 1 Processor -- start 2024-03-11T19:47:25.538 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.189+0000 7f9b93200700 1 -- start start 2024-03-11T19:47:25.538 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.189+0000 7f9b93200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 0x7f9b941099e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:25.538 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.189+0000 7f9b93200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b94109fb0 con 0x7f9b941095c0 2024-03-11T19:47:25.538 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.189+0000 7f9b91e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 0x7f9b941099e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:25.538 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.190+0000 7f9b91e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 0x7f9b941099e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56242/0 (socket says 172.21.15.97:56242) 2024-03-11T19:47:25.538 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.190+0000 7f9b91e00700 1 -- 172.21.15.97:0/1257715789 learned_addr learned my addr 172.21.15.97:0/1257715789 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:25.538 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.190+0000 7f9b91e00700 1 -- 172.21.15.97:0/1257715789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9410a770 con 0x7f9b941095c0 2024-03-11T19:47:25.538 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.190+0000 7f9b91e00700 1 --2- 172.21.15.97:0/1257715789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 0x7f9b941099e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9b840098a0 tx=0x7f9b84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7701b37e45596062 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:25.538 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.191+0000 7f9b91400700 1 -- 172.21.15.97:0/1257715789 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b84004160 con 0x7f9b941095c0 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.191+0000 7f9b91400700 1 -- 172.21.15.97:0/1257715789 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b840042c0 con 0x7f9b941095c0 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.191+0000 7f9b91400700 1 -- 172.21.15.97:0/1257715789 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b84011ac0 con 0x7f9b941095c0 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.191+0000 7f9b93200700 1 -- 172.21.15.97:0/1257715789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 msgr2=0x7f9b941099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.191+0000 7f9b93200700 1 --2- 172.21.15.97:0/1257715789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 0x7f9b941099e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9b840098a0 tx=0x7f9b84009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.191+0000 7f9b93200700 1 -- 172.21.15.97:0/1257715789 shutdown_connections 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.191+0000 7f9b93200700 1 --2- 172.21.15.97:0/1257715789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 0x7f9b941099e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.191+0000 7f9b93200700 1 -- 172.21.15.97:0/1257715789 >> 172.21.15.97:0/1257715789 conn(0x7f9b9407cc20 msgr2=0x7f9b941074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.192+0000 7f9b93200700 1 -- 172.21.15.97:0/1257715789 shutdown_connections 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.192+0000 7f9b93200700 1 -- 172.21.15.97:0/1257715789 wait complete. 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.193+0000 7f9b93200700 1 Processor -- start 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.193+0000 7f9b93200700 1 -- start start 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.193+0000 7f9b93200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 0x7f9b9419d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.193+0000 7f9b93200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9419d880 con 0x7f9b941095c0 2024-03-11T19:47:25.539 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.193+0000 7f9b91e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 0x7f9b9419d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.193+0000 7f9b91e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 0x7f9b9419d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56256/0 (socket says 172.21.15.97:56256) 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.193+0000 7f9b91e00700 1 -- 172.21.15.97:0/856096333 learned_addr learned my addr 172.21.15.97:0/856096333 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.194+0000 7f9b91e00700 1 -- 172.21.15.97:0/856096333 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b84009550 con 0x7f9b941095c0 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.194+0000 7f9b91e00700 1 --2- 172.21.15.97:0/856096333 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 0x7f9b9419d340 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9b84011780 tx=0x7f9b84011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.194+0000 7f9b8aa00700 1 -- 172.21.15.97:0/856096333 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b84003af0 con 0x7f9b941095c0 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.194+0000 7f9b8aa00700 1 -- 172.21.15.97:0/856096333 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b84003c50 con 0x7f9b941095c0 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.194+0000 7f9b93200700 1 -- 172.21.15.97:0/856096333 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9419da80 con 0x7f9b941095c0 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.194+0000 7f9b8aa00700 1 -- 172.21.15.97:0/856096333 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b840197e0 con 0x7f9b941095c0 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.195+0000 7f9b93200700 1 -- 172.21.15.97:0/856096333 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9419df20 con 0x7f9b941095c0 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.196+0000 7f9b93200700 1 -- 172.21.15.97:0/856096333 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b9410d9a0 con 0x7f9b941095c0 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.196+0000 7f9b8aa00700 1 -- 172.21.15.97:0/856096333 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f9b8401aba0 con 0x7f9b941095c0 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.196+0000 7f9b8aa00700 1 --2- 172.21.15.97:0/856096333 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f9b78038d00 0x7f9b7803b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.196+0000 7f9b8aa00700 1 -- 172.21.15.97:0/856096333 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9b8404d060 con 0x7f9b941095c0 2024-03-11T19:47:25.540 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.196+0000 7f9b8be00700 1 --2- 172.21.15.97:0/856096333 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f9b78038d00 0x7f9b7803b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.197+0000 7f9b8be00700 1 --2- 172.21.15.97:0/856096333 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f9b78038d00 0x7f9b7803b1c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c006fd0 tx=0x7f9b7c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.202+0000 7f9b8aa00700 1 -- 172.21.15.97:0/856096333 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9b8401a430 con 0x7f9b941095c0 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:18.425+0000 7f9b93200700 1 -- 172.21.15.97:0/856096333 --> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi097", "addr": "172.21.15.97", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b941073f0 con 0x7f9b78038d00 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:24.667+0000 7f9b8aa00700 1 -- 172.21.15.97:0/856096333 <== mgr.14120 v2:172.21.15.97:6800/1708150324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f9b941073f0 con 0x7f9b78038d00 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:24.671+0000 7f9b93200700 1 -- 172.21.15.97:0/856096333 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f9b78038d00 msgr2=0x7f9b7803b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:24.671+0000 7f9b93200700 1 --2- 172.21.15.97:0/856096333 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f9b78038d00 0x7f9b7803b1c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c006fd0 tx=0x7f9b7c006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:24.671+0000 7f9b93200700 1 -- 172.21.15.97:0/856096333 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 msgr2=0x7f9b9419d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:24.671+0000 7f9b93200700 1 --2- 172.21.15.97:0/856096333 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 0x7f9b9419d340 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9b84011780 tx=0x7f9b84011b40 comp rx=0 tx=0).stop 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:24.671+0000 7f9b93200700 1 -- 172.21.15.97:0/856096333 shutdown_connections 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:24.671+0000 7f9b93200700 1 --2- 172.21.15.97:0/856096333 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f9b78038d00 0x7f9b7803b1c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:24.671+0000 7f9b93200700 1 --2- 172.21.15.97:0/856096333 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9b941095c0 0x7f9b9419d340 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:24.671+0000 7f9b93200700 1 -- 172.21.15.97:0/856096333 >> 172.21.15.97:0/856096333 conn(0x7f9b9407cc20 msgr2=0x7f9b94106ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:24.672+0000 7f9b93200700 1 -- 172.21.15.97:0/856096333 shutdown_connections 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:24.672+0000 7f9b93200700 1 -- 172.21.15.97:0/856096333 wait complete. 2024-03-11T19:47:25.541 INFO:teuthology.orchestra.run.smithi097.stdout:Deploying unmanaged mon service... 2024-03-11T19:47:25.847 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:25 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:25.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:25 smithi097 ceph-mon[31898]: Added host smithi097 2024-03-11T19:47:25.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:25 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:47:27.730 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:27 smithi097 ceph-mon[31898]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:47:27.730 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:27 smithi097 ceph-mon[31898]: Saving service mon spec with placement count:5 2024-03-11T19:47:27.731 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:27 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:27.736 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T19:47:27.742 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.191+0000 7f1eee200700 1 Processor -- start 2024-03-11T19:47:27.742 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.191+0000 7f1eee200700 1 -- start start 2024-03-11T19:47:27.742 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.191+0000 7f1eee200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 0x7f1ee807b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:27.742 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.191+0000 7f1eee200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee807b810 con 0x7f1ee807cdd0 2024-03-11T19:47:27.742 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.192+0000 7f1ee7400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 0x7f1ee807b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:27.742 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.192+0000 7f1ee7400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 0x7f1ee807b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57312/0 (socket says 172.21.15.97:57312) 2024-03-11T19:47:27.742 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.192+0000 7f1ee7400700 1 -- 172.21.15.97:0/479482731 learned_addr learned my addr 172.21.15.97:0/479482731 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.193+0000 7f1ee7400700 1 -- 172.21.15.97:0/479482731 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee807b950 con 0x7f1ee807cdd0 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.193+0000 7f1ee7400700 1 --2- 172.21.15.97:0/479482731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 0x7f1ee807b2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80098a0 tx=0x7f1ed8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab5c29abb21fc0b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.193+0000 7f1ee6000700 1 -- 172.21.15.97:0/479482731 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ed8004160 con 0x7f1ee807cdd0 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.193+0000 7f1ee6000700 1 -- 172.21.15.97:0/479482731 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ed80042c0 con 0x7f1ee807cdd0 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.193+0000 7f1ee6000700 1 -- 172.21.15.97:0/479482731 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ed8011ac0 con 0x7f1ee807cdd0 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.194+0000 7f1eee200700 1 -- 172.21.15.97:0/479482731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 msgr2=0x7f1ee807b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.194+0000 7f1eee200700 1 --2- 172.21.15.97:0/479482731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 0x7f1ee807b2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80098a0 tx=0x7f1ed8009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.194+0000 7f1eee200700 1 -- 172.21.15.97:0/479482731 shutdown_connections 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.194+0000 7f1eee200700 1 --2- 172.21.15.97:0/479482731 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 0x7f1ee807b2d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.194+0000 7f1eee200700 1 -- 172.21.15.97:0/479482731 >> 172.21.15.97:0/479482731 conn(0x7f1ee8102bd0 msgr2=0x7f1ee8104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.194+0000 7f1eee200700 1 -- 172.21.15.97:0/479482731 shutdown_connections 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.194+0000 7f1eee200700 1 -- 172.21.15.97:0/479482731 wait complete. 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.195+0000 7f1eee200700 1 Processor -- start 2024-03-11T19:47:27.743 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.195+0000 7f1eee200700 1 -- start start 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.196+0000 7f1eee200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 0x7f1ee819d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.196+0000 7f1eee200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee819d9e0 con 0x7f1ee807cdd0 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.196+0000 7f1ee7400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 0x7f1ee819d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.196+0000 7f1ee7400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 0x7f1ee819d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57328/0 (socket says 172.21.15.97:57328) 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.196+0000 7f1ee7400700 1 -- 172.21.15.97:0/1907300971 learned_addr learned my addr 172.21.15.97:0/1907300971 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.196+0000 7f1ee7400700 1 -- 172.21.15.97:0/1907300971 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed8009550 con 0x7f1ee807cdd0 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.196+0000 7f1ee7400700 1 --2- 172.21.15.97:0/1907300971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 0x7f1ee819d4a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8011a70 tx=0x7f1ed8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.197+0000 7f1edfe00700 1 -- 172.21.15.97:0/1907300971 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ed80039c0 con 0x7f1ee807cdd0 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.197+0000 7f1edfe00700 1 -- 172.21.15.97:0/1907300971 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ed8003b20 con 0x7f1ee807cdd0 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.197+0000 7f1eee200700 1 -- 172.21.15.97:0/1907300971 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee819dbe0 con 0x7f1ee807cdd0 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.197+0000 7f1edfe00700 1 -- 172.21.15.97:0/1907300971 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ed8024610 con 0x7f1ee807cdd0 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.197+0000 7f1eee200700 1 -- 172.21.15.97:0/1907300971 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee819e080 con 0x7f1ee807cdd0 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.198+0000 7f1edfe00700 1 -- 172.21.15.97:0/1907300971 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f1ed801a430 con 0x7f1ee807cdd0 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.198+0000 7f1eee200700 1 -- 172.21.15.97:0/1907300971 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee810d8a0 con 0x7f1ee807cdd0 2024-03-11T19:47:27.744 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.198+0000 7f1edfe00700 1 --2- 172.21.15.97:0/1907300971 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f1ecc038d60 0x7f1ecc03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:27.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.198+0000 7f1edfe00700 1 -- 172.21.15.97:0/1907300971 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1ed804cd80 con 0x7f1ee807cdd0 2024-03-11T19:47:27.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.199+0000 7f1ee6a00700 1 --2- 172.21.15.97:0/1907300971 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f1ecc038d60 0x7f1ecc03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:27.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.199+0000 7f1ee6a00700 1 --2- 172.21.15.97:0/1907300971 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f1ecc038d60 0x7f1ecc03b220 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4006fd0 tx=0x7f1ed4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:27.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.204+0000 7f1edfe00700 1 -- 172.21.15.97:0/1907300971 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1ed8052020 con 0x7f1ee807cdd0 2024-03-11T19:47:27.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.439+0000 7f1eee200700 1 -- 172.21.15.97:0/1907300971 --> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1ee8103e80 con 0x7f1ecc038d60 2024-03-11T19:47:27.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.446+0000 7f1edfe00700 1 -- 172.21.15.97:0/1907300971 <== mgr.14120 v2:172.21.15.97:6800/1708150324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1ee8103e80 con 0x7f1ecc038d60 2024-03-11T19:47:27.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.448+0000 7f1eee200700 1 -- 172.21.15.97:0/1907300971 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f1ecc038d60 msgr2=0x7f1ecc03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:27.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.448+0000 7f1eee200700 1 --2- 172.21.15.97:0/1907300971 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f1ecc038d60 0x7f1ecc03b220 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4006fd0 tx=0x7f1ed4006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:27.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.448+0000 7f1eee200700 1 -- 172.21.15.97:0/1907300971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 msgr2=0x7f1ee819d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:27.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.448+0000 7f1eee200700 1 --2- 172.21.15.97:0/1907300971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 0x7f1ee819d4a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8011a70 tx=0x7f1ed8011b50 comp rx=0 tx=0).stop 2024-03-11T19:47:27.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.448+0000 7f1eee200700 1 -- 172.21.15.97:0/1907300971 shutdown_connections 2024-03-11T19:47:27.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.448+0000 7f1eee200700 1 --2- 172.21.15.97:0/1907300971 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f1ecc038d60 0x7f1ecc03b220 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:27.745 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.448+0000 7f1eee200700 1 --2- 172.21.15.97:0/1907300971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1ee807cdd0 0x7f1ee819d4a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:27.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.448+0000 7f1eee200700 1 -- 172.21.15.97:0/1907300971 >> 172.21.15.97:0/1907300971 conn(0x7f1ee8102bd0 msgr2=0x7f1ee8103770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:27.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.448+0000 7f1eee200700 1 -- 172.21.15.97:0/1907300971 shutdown_connections 2024-03-11T19:47:27.746 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:26.449+0000 7f1eee200700 1 -- 172.21.15.97:0/1907300971 wait complete. 2024-03-11T19:47:27.746 INFO:teuthology.orchestra.run.smithi097.stdout:Deploying unmanaged mgr service... 2024-03-11T19:47:29.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:29 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:29.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:29 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:29.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:29 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:29.763 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T19:47:29.764 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.802+0000 7f52ce400700 1 Processor -- start 2024-03-11T19:47:29.764 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.802+0000 7f52ce400700 1 -- start start 2024-03-11T19:47:29.764 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.803+0000 7f52ce400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 0x7f52c8107970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:29.764 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.803+0000 7f52ce400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c8107f40 con 0x7f52c8107550 2024-03-11T19:47:29.764 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.803+0000 7f52c7400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 0x7f52c8107970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:29.764 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.803+0000 7f52c7400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 0x7f52c8107970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57336/0 (socket says 172.21.15.97:57336) 2024-03-11T19:47:29.764 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.803+0000 7f52c7400700 1 -- 172.21.15.97:0/2688931697 learned_addr learned my addr 172.21.15.97:0/2688931697 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:29.764 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.804+0000 7f52c7400700 1 -- 172.21.15.97:0/2688931697 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c81086b0 con 0x7f52c8107550 2024-03-11T19:47:29.764 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.804+0000 7f52c7400700 1 --2- 172.21.15.97:0/2688931697 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 0x7f52c8107970 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f52b80098a0 tx=0x7f52b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484c51b373dfc925 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:29.764 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.804+0000 7f52c6000700 1 -- 172.21.15.97:0/2688931697 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52b8004160 con 0x7f52c8107550 2024-03-11T19:47:29.764 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.804+0000 7f52c6000700 1 -- 172.21.15.97:0/2688931697 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52b80042c0 con 0x7f52c8107550 2024-03-11T19:47:29.764 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.804+0000 7f52c6000700 1 -- 172.21.15.97:0/2688931697 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52b8011a80 con 0x7f52c8107550 2024-03-11T19:47:29.764 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.805+0000 7f52ce400700 1 -- 172.21.15.97:0/2688931697 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 msgr2=0x7f52c8107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.805+0000 7f52ce400700 1 --2- 172.21.15.97:0/2688931697 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 0x7f52c8107970 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f52b80098a0 tx=0x7f52b8009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.805+0000 7f52ce400700 1 -- 172.21.15.97:0/2688931697 shutdown_connections 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.805+0000 7f52ce400700 1 --2- 172.21.15.97:0/2688931697 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 0x7f52c8107970 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.805+0000 7f52ce400700 1 -- 172.21.15.97:0/2688931697 >> 172.21.15.97:0/2688931697 conn(0x7f52c8102bd0 msgr2=0x7f52c8104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.805+0000 7f52ce400700 1 -- 172.21.15.97:0/2688931697 shutdown_connections 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.805+0000 7f52ce400700 1 -- 172.21.15.97:0/2688931697 wait complete. 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.806+0000 7f52ce400700 1 Processor -- start 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.807+0000 7f52ce400700 1 -- start start 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.807+0000 7f52ce400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 0x7f52c807cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.807+0000 7f52ce400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c807d0c0 con 0x7f52c8107550 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.807+0000 7f52c7400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 0x7f52c807cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.807+0000 7f52c7400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 0x7f52c807cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57350/0 (socket says 172.21.15.97:57350) 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.807+0000 7f52c7400700 1 -- 172.21.15.97:0/2158879266 learned_addr learned my addr 172.21.15.97:0/2158879266 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.808+0000 7f52c7400700 1 -- 172.21.15.97:0/2158879266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b8009550 con 0x7f52c8107550 2024-03-11T19:47:29.765 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.808+0000 7f52c7400700 1 --2- 172.21.15.97:0/2158879266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 0x7f52c807cb80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f52b8011740 tx=0x7f52b8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:29.766 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.808+0000 7f52bfe00700 1 -- 172.21.15.97:0/2158879266 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52b80039e0 con 0x7f52c8107550 2024-03-11T19:47:29.768 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.808+0000 7f52ce400700 1 -- 172.21.15.97:0/2158879266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c807b2b0 con 0x7f52c8107550 2024-03-11T19:47:29.768 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.808+0000 7f52bfe00700 1 -- 172.21.15.97:0/2158879266 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52b8003b40 con 0x7f52c8107550 2024-03-11T19:47:29.768 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.808+0000 7f52bfe00700 1 -- 172.21.15.97:0/2158879266 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52b8024740 con 0x7f52c8107550 2024-03-11T19:47:29.768 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.808+0000 7f52ce400700 1 -- 172.21.15.97:0/2158879266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c807b750 con 0x7f52c8107550 2024-03-11T19:47:29.768 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.810+0000 7f52ce400700 1 -- 172.21.15.97:0/2158879266 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c810b8e0 con 0x7f52c8107550 2024-03-11T19:47:29.768 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.810+0000 7f52bfe00700 1 -- 172.21.15.97:0/2158879266 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f52b801a430 con 0x7f52c8107550 2024-03-11T19:47:29.768 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.810+0000 7f52bfe00700 1 --2- 172.21.15.97:0/2158879266 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f52ac038cb0 0x7f52ac03b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:29.768 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.810+0000 7f52bfe00700 1 -- 172.21.15.97:0/2158879266 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f52b804cfe0 con 0x7f52c8107550 2024-03-11T19:47:29.768 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.811+0000 7f52c6a00700 1 --2- 172.21.15.97:0/2158879266 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f52ac038cb0 0x7f52ac03b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:29.768 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.811+0000 7f52c6a00700 1 --2- 172.21.15.97:0/2158879266 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f52ac038cb0 0x7f52ac03b170 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f52b4006fd0 tx=0x7f52b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:28.816+0000 7f52bfe00700 1 -- 172.21.15.97:0/2158879266 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f52b8080d20 con 0x7f52c8107550 2024-03-11T19:47:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:29.057+0000 7f52ce400700 1 -- 172.21.15.97:0/2158879266 --> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f52c807bd50 con 0x7f52ac038cb0 2024-03-11T19:47:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:29.065+0000 7f52bfe00700 1 -- 172.21.15.97:0/2158879266 <== mgr.14120 v2:172.21.15.97:6800/1708150324 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f52c807bd50 con 0x7f52ac038cb0 2024-03-11T19:47:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:29.069+0000 7f52ce400700 1 -- 172.21.15.97:0/2158879266 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f52ac038cb0 msgr2=0x7f52ac03b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:29.069+0000 7f52ce400700 1 --2- 172.21.15.97:0/2158879266 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f52ac038cb0 0x7f52ac03b170 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f52b4006fd0 tx=0x7f52b4006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:29.069+0000 7f52ce400700 1 -- 172.21.15.97:0/2158879266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 msgr2=0x7f52c807cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:29.069+0000 7f52ce400700 1 --2- 172.21.15.97:0/2158879266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 0x7f52c807cb80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f52b8011740 tx=0x7f52b8011b00 comp rx=0 tx=0).stop 2024-03-11T19:47:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:29.069+0000 7f52ce400700 1 -- 172.21.15.97:0/2158879266 shutdown_connections 2024-03-11T19:47:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:29.069+0000 7f52ce400700 1 --2- 172.21.15.97:0/2158879266 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f52ac038cb0 0x7f52ac03b170 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:29.069+0000 7f52ce400700 1 --2- 172.21.15.97:0/2158879266 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f52c8107550 0x7f52c807cb80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:29.069+0000 7f52ce400700 1 -- 172.21.15.97:0/2158879266 >> 172.21.15.97:0/2158879266 conn(0x7f52c8102bd0 msgr2=0x7f52c8103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:29.069+0000 7f52ce400700 1 -- 172.21.15.97:0/2158879266 shutdown_connections 2024-03-11T19:47:29.769 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:29.069+0000 7f52ce400700 1 -- 172.21.15.97:0/2158879266 wait complete. 2024-03-11T19:47:30.593 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:30 smithi097 ceph-mon[31898]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:47:30.593 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:30 smithi097 ceph-mon[31898]: Saving service mgr spec with placement count:2 2024-03-11T19:47:30.593 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:30 smithi097 ceph-mon[31898]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:47:31.592 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.351+0000 7fc1fba00700 1 Processor -- start 2024-03-11T19:47:31.592 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.352+0000 7fc1fba00700 1 -- start start 2024-03-11T19:47:31.592 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.352+0000 7fc1fba00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 0x7fc1f4107990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:31.592 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.352+0000 7fc1fba00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f4107f60 con 0x7fc1f4107570 2024-03-11T19:47:31.592 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.352+0000 7fc1f9000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 0x7fc1f4107990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:31.592 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.352+0000 7fc1f9000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 0x7fc1f4107990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57352/0 (socket says 172.21.15.97:57352) 2024-03-11T19:47:31.593 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.352+0000 7fc1f9000700 1 -- 172.21.15.97:0/1242520314 learned_addr learned my addr 172.21.15.97:0/1242520314 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:31.593 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.353+0000 7fc1f9000700 1 -- 172.21.15.97:0/1242520314 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f4108720 con 0x7fc1f4107570 2024-03-11T19:47:31.593 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.353+0000 7fc1f9000700 1 --2- 172.21.15.97:0/1242520314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 0x7fc1f4107990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80098a0 tx=0x7fc1e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd12b5b1ba771c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:31.593 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.353+0000 7fc1f3400700 1 -- 172.21.15.97:0/1242520314 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1e8004160 con 0x7fc1f4107570 2024-03-11T19:47:31.593 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.353+0000 7fc1f3400700 1 -- 172.21.15.97:0/1242520314 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1e80042c0 con 0x7fc1f4107570 2024-03-11T19:47:31.593 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.353+0000 7fc1f3400700 1 -- 172.21.15.97:0/1242520314 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1e8011a80 con 0x7fc1f4107570 2024-03-11T19:47:31.593 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.354+0000 7fc1fba00700 1 -- 172.21.15.97:0/1242520314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 msgr2=0x7fc1f4107990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:31.593 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.354+0000 7fc1fba00700 1 --2- 172.21.15.97:0/1242520314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 0x7fc1f4107990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80098a0 tx=0x7fc1e8009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:31.593 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.354+0000 7fc1fba00700 1 -- 172.21.15.97:0/1242520314 shutdown_connections 2024-03-11T19:47:31.593 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.354+0000 7fc1fba00700 1 --2- 172.21.15.97:0/1242520314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 0x7fc1f4107990 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:31.593 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.354+0000 7fc1fba00700 1 -- 172.21.15.97:0/1242520314 >> 172.21.15.97:0/1242520314 conn(0x7fc1f4102bd0 msgr2=0x7fc1f4104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:31.593 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.354+0000 7fc1fba00700 1 -- 172.21.15.97:0/1242520314 shutdown_connections 2024-03-11T19:47:31.593 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.355+0000 7fc1fba00700 1 -- 172.21.15.97:0/1242520314 wait complete. 2024-03-11T19:47:31.594 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.356+0000 7fc1fba00700 1 Processor -- start 2024-03-11T19:47:31.594 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.356+0000 7fc1fba00700 1 -- start start 2024-03-11T19:47:31.594 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.356+0000 7fc1fba00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 0x7fc1f419d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:31.594 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.356+0000 7fc1fba00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f419d930 con 0x7fc1f4107570 2024-03-11T19:47:31.594 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.356+0000 7fc1f9000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 0x7fc1f419d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:31.594 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.356+0000 7fc1f9000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 0x7fc1f419d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57364/0 (socket says 172.21.15.97:57364) 2024-03-11T19:47:31.594 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.356+0000 7fc1f9000700 1 -- 172.21.15.97:0/1979287630 learned_addr learned my addr 172.21.15.97:0/1979287630 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:31.594 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.357+0000 7fc1f9000700 1 -- 172.21.15.97:0/1979287630 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e8009550 con 0x7fc1f4107570 2024-03-11T19:47:31.594 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.357+0000 7fc1f9000700 1 --2- 172.21.15.97:0/1979287630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 0x7fc1f419d3f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8011740 tx=0x7fc1e8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:31.594 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.357+0000 7fc1f1600700 1 -- 172.21.15.97:0/1979287630 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1e8003b20 con 0x7fc1f4107570 2024-03-11T19:47:31.594 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.357+0000 7fc1f1600700 1 -- 172.21.15.97:0/1979287630 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1e801a910 con 0x7fc1f4107570 2024-03-11T19:47:31.594 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.357+0000 7fc1fba00700 1 -- 172.21.15.97:0/1979287630 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f419db30 con 0x7fc1f4107570 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.357+0000 7fc1f1600700 1 -- 172.21.15.97:0/1979287630 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1e8024850 con 0x7fc1f4107570 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.357+0000 7fc1fba00700 1 -- 172.21.15.97:0/1979287630 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f419dfd0 con 0x7fc1f4107570 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.358+0000 7fc1f1600700 1 -- 172.21.15.97:0/1979287630 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fc1e801a430 con 0x7fc1f4107570 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.359+0000 7fc1f1600700 1 --2- 172.21.15.97:0/1979287630 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fc1dc038d00 0x7fc1dc03b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.359+0000 7fc1fba00700 1 -- 172.21.15.97:0/1979287630 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f410b950 con 0x7fc1f4107570 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.359+0000 7fc1f1600700 1 -- 172.21.15.97:0/1979287630 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc1e804d050 con 0x7fc1f4107570 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.359+0000 7fc1f3e00700 1 --2- 172.21.15.97:0/1979287630 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fc1dc038d00 0x7fc1dc03b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.360+0000 7fc1f3e00700 1 --2- 172.21.15.97:0/1979287630 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fc1dc038d00 0x7fc1dc03b1c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0006fd0 tx=0x7fc1e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.365+0000 7fc1f1600700 1 -- 172.21.15.97:0/1979287630 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc1e8018350 con 0x7fc1f4107570 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.588+0000 7fc1fba00700 1 -- 172.21.15.97:0/1979287630 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fc1f410bb60 con 0x7fc1f4107570 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.592+0000 7fc1f1600700 1 -- 172.21.15.97:0/1979287630 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fc1e804b0d0 con 0x7fc1f4107570 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.595+0000 7fc1fba00700 1 -- 172.21.15.97:0/1979287630 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fc1dc038d00 msgr2=0x7fc1dc03b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.595+0000 7fc1fba00700 1 --2- 172.21.15.97:0/1979287630 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fc1dc038d00 0x7fc1dc03b1c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0006fd0 tx=0x7fc1e0006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.595+0000 7fc1fba00700 1 -- 172.21.15.97:0/1979287630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 msgr2=0x7fc1f419d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:31.595 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.595+0000 7fc1fba00700 1 --2- 172.21.15.97:0/1979287630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 0x7fc1f419d3f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8011740 tx=0x7fc1e8011b00 comp rx=0 tx=0).stop 2024-03-11T19:47:31.596 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.595+0000 7fc1fba00700 1 -- 172.21.15.97:0/1979287630 shutdown_connections 2024-03-11T19:47:31.596 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.595+0000 7fc1fba00700 1 --2- 172.21.15.97:0/1979287630 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7fc1dc038d00 0x7fc1dc03b1c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:31.596 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.595+0000 7fc1fba00700 1 --2- 172.21.15.97:0/1979287630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc1f4107570 0x7fc1f419d3f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:31.596 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.595+0000 7fc1fba00700 1 -- 172.21.15.97:0/1979287630 >> 172.21.15.97:0/1979287630 conn(0x7fc1f4102bd0 msgr2=0x7fc1f41035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:31.596 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.595+0000 7fc1fba00700 1 -- 172.21.15.97:0/1979287630 shutdown_connections 2024-03-11T19:47:31.596 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:30.596+0000 7fc1fba00700 1 -- 172.21.15.97:0/1979287630 wait complete. 2024-03-11T19:47:31.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:31 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1979287630' entity='client.admin' 2024-03-11T19:47:32.873 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:32 smithi097 ceph-mon[31898]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:47:32.873 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:32 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:33.625 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.617+0000 7f7464e00700 1 Processor -- start 2024-03-11T19:47:33.626 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.617+0000 7f7464e00700 1 -- start start 2024-03-11T19:47:33.626 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.617+0000 7f7464e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 0x7f746010e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:33.626 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.617+0000 7f7464e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746010ea10 con 0x7f746010e020 2024-03-11T19:47:33.626 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.618+0000 7f745de00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 0x7f746010e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:33.626 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.618+0000 7f745de00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 0x7f746010e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57372/0 (socket says 172.21.15.97:57372) 2024-03-11T19:47:33.626 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.618+0000 7f745de00700 1 -- 172.21.15.97:0/3205426155 learned_addr learned my addr 172.21.15.97:0/3205426155 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:33.627 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.618+0000 7f745de00700 1 -- 172.21.15.97:0/3205426155 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746010eb50 con 0x7f746010e020 2024-03-11T19:47:33.627 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.619+0000 7f745de00700 1 --2- 172.21.15.97:0/3205426155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 0x7f746010e440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f74500098a0 tx=0x7f7450009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65e1338fb0b1571 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:33.627 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.619+0000 7f745d400700 1 -- 172.21.15.97:0/3205426155 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7450004160 con 0x7f746010e020 2024-03-11T19:47:33.627 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.619+0000 7f745d400700 1 -- 172.21.15.97:0/3205426155 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74500042c0 con 0x7f746010e020 2024-03-11T19:47:33.627 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.619+0000 7f745d400700 1 -- 172.21.15.97:0/3205426155 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7450011a80 con 0x7f746010e020 2024-03-11T19:47:33.627 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.620+0000 7f7464e00700 1 -- 172.21.15.97:0/3205426155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 msgr2=0x7f746010e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:33.627 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.620+0000 7f7464e00700 1 --2- 172.21.15.97:0/3205426155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 0x7f746010e440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f74500098a0 tx=0x7f7450009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:33.628 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.620+0000 7f7464e00700 1 -- 172.21.15.97:0/3205426155 shutdown_connections 2024-03-11T19:47:33.628 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.620+0000 7f7464e00700 1 --2- 172.21.15.97:0/3205426155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 0x7f746010e440 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:33.628 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.620+0000 7f7464e00700 1 -- 172.21.15.97:0/3205426155 >> 172.21.15.97:0/3205426155 conn(0x7f746007ce20 msgr2=0x7f746007f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:33.628 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.620+0000 7f7464e00700 1 -- 172.21.15.97:0/3205426155 shutdown_connections 2024-03-11T19:47:33.628 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.620+0000 7f7464e00700 1 -- 172.21.15.97:0/3205426155 wait complete. 2024-03-11T19:47:33.628 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.621+0000 7f7464e00700 1 Processor -- start 2024-03-11T19:47:33.628 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.621+0000 7f7464e00700 1 -- start start 2024-03-11T19:47:33.629 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.622+0000 7f7464e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 0x7f74601a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:33.629 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.622+0000 7f7464e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74601a1ce0 con 0x7f746010e020 2024-03-11T19:47:33.629 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.622+0000 7f745de00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 0x7f74601a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:33.629 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.622+0000 7f745de00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 0x7f74601a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57374/0 (socket says 172.21.15.97:57374) 2024-03-11T19:47:33.629 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.622+0000 7f745de00700 1 -- 172.21.15.97:0/3288820447 learned_addr learned my addr 172.21.15.97:0/3288820447 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:33.629 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.622+0000 7f745de00700 1 -- 172.21.15.97:0/3288820447 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7450009550 con 0x7f746010e020 2024-03-11T19:47:33.630 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.623+0000 7f745de00700 1 --2- 172.21.15.97:0/3288820447 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 0x7f74601a17a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7450011740 tx=0x7f7450011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:33.630 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.623+0000 7f7456a00700 1 -- 172.21.15.97:0/3288820447 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7450003b20 con 0x7f746010e020 2024-03-11T19:47:33.630 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.623+0000 7f7456a00700 1 -- 172.21.15.97:0/3288820447 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f745001a910 con 0x7f746010e020 2024-03-11T19:47:33.630 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.623+0000 7f7464e00700 1 -- 172.21.15.97:0/3288820447 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74601a1ee0 con 0x7f746010e020 2024-03-11T19:47:33.630 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.623+0000 7f7456a00700 1 -- 172.21.15.97:0/3288820447 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74500248c0 con 0x7f746010e020 2024-03-11T19:47:33.630 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.623+0000 7f7464e00700 1 -- 172.21.15.97:0/3288820447 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74601a2380 con 0x7f746010e020 2024-03-11T19:47:33.630 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.624+0000 7f7464e00700 1 -- 172.21.15.97:0/3288820447 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7460111fc0 con 0x7f746010e020 2024-03-11T19:47:33.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.624+0000 7f7456a00700 1 -- 172.21.15.97:0/3288820447 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f745001a430 con 0x7f746010e020 2024-03-11T19:47:33.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.625+0000 7f7456a00700 1 --2- 172.21.15.97:0/3288820447 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f7440038cb0 0x7f744003b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:33.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.625+0000 7f7456a00700 1 -- 172.21.15.97:0/3288820447 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f745004cfe0 con 0x7f746010e020 2024-03-11T19:47:33.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.625+0000 7f7457e00700 1 --2- 172.21.15.97:0/3288820447 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f7440038cb0 0x7f744003b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:33.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.626+0000 7f7457e00700 1 --2- 172.21.15.97:0/3288820447 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f7440038cb0 0x7f744003b170 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7448006fd0 tx=0x7f7448006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:33.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.630+0000 7f7456a00700 1 -- 172.21.15.97:0/3288820447 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7450018530 con 0x7f746010e020 2024-03-11T19:47:33.631 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.869+0000 7f7464e00700 1 -- 172.21.15.97:0/3288820447 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f746010eb50 con 0x7f746010e020 2024-03-11T19:47:33.632 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.874+0000 7f7456a00700 1 -- 172.21.15.97:0/3288820447 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f746010eb50 con 0x7f746010e020 2024-03-11T19:47:33.632 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.878+0000 7f7464e00700 1 -- 172.21.15.97:0/3288820447 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f7440038cb0 msgr2=0x7f744003b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:33.632 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.878+0000 7f7464e00700 1 --2- 172.21.15.97:0/3288820447 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f7440038cb0 0x7f744003b170 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7448006fd0 tx=0x7f7448006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:33.632 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.878+0000 7f7464e00700 1 -- 172.21.15.97:0/3288820447 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 msgr2=0x7f74601a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:33.632 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.878+0000 7f7464e00700 1 --2- 172.21.15.97:0/3288820447 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 0x7f74601a17a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7450011740 tx=0x7f7450011b00 comp rx=0 tx=0).stop 2024-03-11T19:47:33.632 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.878+0000 7f7464e00700 1 -- 172.21.15.97:0/3288820447 shutdown_connections 2024-03-11T19:47:33.633 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.878+0000 7f7464e00700 1 --2- 172.21.15.97:0/3288820447 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f7440038cb0 0x7f744003b170 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:33.633 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.878+0000 7f7464e00700 1 --2- 172.21.15.97:0/3288820447 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f746010e020 0x7f74601a17a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:33.633 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.878+0000 7f7464e00700 1 -- 172.21.15.97:0/3288820447 >> 172.21.15.97:0/3288820447 conn(0x7f746007ce20 msgr2=0x7f746007e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:33.633 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.879+0000 7f7464e00700 1 -- 172.21.15.97:0/3288820447 shutdown_connections 2024-03-11T19:47:33.633 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:32.879+0000 7f7464e00700 1 -- 172.21.15.97:0/3288820447 wait complete. 2024-03-11T19:47:33.633 INFO:teuthology.orchestra.run.smithi097.stdout:Enabling the dashboard module... 2024-03-11T19:47:34.091 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:33 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3288820447' entity='client.admin' 2024-03-11T19:47:34.091 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:33 smithi097 ceph-mon[31898]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:47:35.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:35 smithi097 ceph-mon[31898]: from='mgr.14120 172.21.15.97:0/3954699349' entity='mgr.x' 2024-03-11T19:47:35.615 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:35 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1701545293' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T19:47:35.922 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:35 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: ignoring --setuser ceph since I am not root 2024-03-11T19:47:35.922 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:35 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: ignoring --setgroup ceph since I am not root 2024-03-11T19:47:36.332 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:36 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:36.327+0000 7f208e3ba200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T19:47:36.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:36 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1701545293' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T19:47:36.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:36 smithi097 ceph-mon[31898]: mgrmap e9: x(active, since 26s) 2024-03-11T19:47:36.672 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:36 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:36.447+0000 7f208e3ba200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T19:47:36.858 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.250+0000 7f418b800700 1 Processor -- start 2024-03-11T19:47:36.859 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.250+0000 7f418b800700 1 -- start start 2024-03-11T19:47:36.859 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.250+0000 7f418b800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 0x7f41841061e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:36.859 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.251+0000 7f418b800700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418407aec0 con 0x7f4184105dc0 2024-03-11T19:47:36.859 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.251+0000 7f4188e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 0x7f41841061e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:36.860 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.251+0000 7f4188e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 0x7f41841061e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47846/0 (socket says 172.21.15.97:47846) 2024-03-11T19:47:36.860 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.251+0000 7f4188e00700 1 -- 172.21.15.97:0/4259288637 learned_addr learned my addr 172.21.15.97:0/4259288637 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:36.862 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.251+0000 7f4188e00700 1 -- 172.21.15.97:0/4259288637 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4184106720 con 0x7f4184105dc0 2024-03-11T19:47:36.862 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.252+0000 7f4188e00700 1 --2- 172.21.15.97:0/4259288637 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 0x7f41841061e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f41780098a0 tx=0x7f4178009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fad95a01bbed474 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:36.862 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.252+0000 7f4183400700 1 -- 172.21.15.97:0/4259288637 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4178004160 con 0x7f4184105dc0 2024-03-11T19:47:36.862 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.252+0000 7f4183400700 1 -- 172.21.15.97:0/4259288637 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41780042c0 con 0x7f4184105dc0 2024-03-11T19:47:36.862 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.252+0000 7f4183400700 1 -- 172.21.15.97:0/4259288637 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4178011a80 con 0x7f4184105dc0 2024-03-11T19:47:36.862 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.253+0000 7f418b800700 1 -- 172.21.15.97:0/4259288637 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 msgr2=0x7f41841061e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.253+0000 7f418b800700 1 --2- 172.21.15.97:0/4259288637 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 0x7f41841061e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f41780098a0 tx=0x7f4178009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.253+0000 7f418b800700 1 -- 172.21.15.97:0/4259288637 shutdown_connections 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.253+0000 7f418b800700 1 --2- 172.21.15.97:0/4259288637 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 0x7f41841061e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.253+0000 7f418b800700 1 -- 172.21.15.97:0/4259288637 >> 172.21.15.97:0/4259288637 conn(0x7f4184101a60 msgr2=0x7f4184103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.253+0000 7f418b800700 1 -- 172.21.15.97:0/4259288637 shutdown_connections 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.253+0000 7f418b800700 1 -- 172.21.15.97:0/4259288637 wait complete. 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.254+0000 7f418b800700 1 Processor -- start 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.254+0000 7f418b800700 1 -- start start 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.255+0000 7f418b800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 0x7f418419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.255+0000 7f418b800700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418419d9c0 con 0x7f4184105dc0 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.255+0000 7f4188e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 0x7f418419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.255+0000 7f4188e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 0x7f418419d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47860/0 (socket says 172.21.15.97:47860) 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.255+0000 7f4188e00700 1 -- 172.21.15.97:0/1701545293 learned_addr learned my addr 172.21.15.97:0/1701545293 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:36.863 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.255+0000 7f4188e00700 1 -- 172.21.15.97:0/1701545293 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4178009550 con 0x7f4184105dc0 2024-03-11T19:47:36.864 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.256+0000 7f4188e00700 1 --2- 172.21.15.97:0/1701545293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 0x7f418419d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4178009130 tx=0x7f4178011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:36.866 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.256+0000 7f4181600700 1 -- 172.21.15.97:0/1701545293 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4178003aa0 con 0x7f4184105dc0 2024-03-11T19:47:36.866 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.256+0000 7f4181600700 1 -- 172.21.15.97:0/1701545293 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4178003c00 con 0x7f4184105dc0 2024-03-11T19:47:36.866 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.256+0000 7f418b800700 1 -- 172.21.15.97:0/1701545293 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418419dbc0 con 0x7f4184105dc0 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.256+0000 7f418b800700 1 -- 172.21.15.97:0/1701545293 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418419e060 con 0x7f4184105dc0 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.256+0000 7f4181600700 1 -- 172.21.15.97:0/1701545293 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41780248a0 con 0x7f4184105dc0 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.257+0000 7f4181600700 1 -- 172.21.15.97:0/1701545293 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f4178024b40 con 0x7f4184105dc0 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.257+0000 7f418b800700 1 -- 172.21.15.97:0/1701545293 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4164005360 con 0x7f4184105dc0 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.258+0000 7f4181600700 1 --2- 172.21.15.97:0/1701545293 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f416c038d60 0x7f416c03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.258+0000 7f4181600700 1 -- 172.21.15.97:0/1701545293 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f417801ac10 con 0x7f4184105dc0 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.258+0000 7f4183e00700 1 --2- 172.21.15.97:0/1701545293 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f416c038d60 0x7f416c03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.259+0000 7f4183e00700 1 --2- 172.21.15.97:0/1701545293 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f416c038d60 0x7f416c03b220 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4170006fd0 tx=0x7f4170006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.264+0000 7f4181600700 1 -- 172.21.15.97:0/1701545293 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4178024df0 con 0x7f4184105dc0 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:34.547+0000 7f418b800700 1 -- 172.21.15.97:0/1701545293 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f4164005610 con 0x7f4184105dc0 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:35.509+0000 7f4181600700 1 -- 172.21.15.97:0/1701545293 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f417802c350 con 0x7f4184105dc0 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:35.512+0000 7f4181600700 1 -- 172.21.15.97:0/1701545293 <== mon.0 v2:172.21.15.97:3300/0 8 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f417801a840 con 0x7f4184105dc0 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:35.514+0000 7f418b800700 1 -- 172.21.15.97:0/1701545293 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f416c038d60 msgr2=0x7f416c03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:35.514+0000 7f418b800700 1 --2- 172.21.15.97:0/1701545293 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f416c038d60 0x7f416c03b220 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4170006fd0 tx=0x7f4170006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:36.867 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:35.514+0000 7f418b800700 1 -- 172.21.15.97:0/1701545293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 msgr2=0x7f418419d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:36.868 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:35.514+0000 7f418b800700 1 --2- 172.21.15.97:0/1701545293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 0x7f418419d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4178009130 tx=0x7f4178011b00 comp rx=0 tx=0).stop 2024-03-11T19:47:36.868 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:35.514+0000 7f418b800700 1 -- 172.21.15.97:0/1701545293 shutdown_connections 2024-03-11T19:47:36.868 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:35.514+0000 7f418b800700 1 --2- 172.21.15.97:0/1701545293 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f416c038d60 0x7f416c03b220 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:36.868 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:35.514+0000 7f418b800700 1 --2- 172.21.15.97:0/1701545293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4184105dc0 0x7f418419d480 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:36.868 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:35.514+0000 7f418b800700 1 -- 172.21.15.97:0/1701545293 >> 172.21.15.97:0/1701545293 conn(0x7f4184101a60 msgr2=0x7f41841942d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:36.868 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:35.515+0000 7f418b800700 1 -- 172.21.15.97:0/1701545293 shutdown_connections 2024-03-11T19:47:36.868 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:35.516+0000 7f418b800700 1 -- 172.21.15.97:0/1701545293 wait complete. 2024-03-11T19:47:37.502 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:37 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:37.245+0000 7f208e3ba200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T19:47:37.502 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:37 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:37.371+0000 7f208e3ba200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T19:47:37.795 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:37 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:37.555+0000 7f208e3ba200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T19:47:37.795 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:37 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:37.681+0000 7f208e3ba200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T19:47:38.135 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:37 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3884797110' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T19:47:38.135 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:37 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:37.794+0000 7f208e3ba200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T19:47:38.135 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:38.130+0000 7f208e3ba200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T19:47:38.385 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:38.270+0000 7f208e3ba200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T19:47:38.386 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:38.381+0000 7f208e3ba200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T19:47:38.527 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.497+0000 7f2611400700 1 Processor -- start 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.497+0000 7f2611400700 1 -- start start 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.498+0000 7f2611400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 0x7f260c10a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.498+0000 7f2611400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c07ae00 con 0x7f260c108080 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.498+0000 7f260a800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 0x7f260c10a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.498+0000 7f260a800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 0x7f260c10a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47882/0 (socket says 172.21.15.97:47882) 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.498+0000 7f260a800700 1 -- 172.21.15.97:0/1221366789 learned_addr learned my addr 172.21.15.97:0/1221366789 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.499+0000 7f260a800700 1 -- 172.21.15.97:0/1221366789 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260c07af40 con 0x7f260c108080 2024-03-11T19:47:38.528 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.499+0000 7f260a800700 1 --2- 172.21.15.97:0/1221366789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 0x7f260c10a470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f25fc0098a0 tx=0x7f25fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6963336ab9fe10a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.500+0000 7f2609400700 1 -- 172.21.15.97:0/1221366789 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25fc004160 con 0x7f260c108080 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.500+0000 7f2609400700 1 -- 172.21.15.97:0/1221366789 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25fc0042c0 con 0x7f260c108080 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.500+0000 7f2609400700 1 -- 172.21.15.97:0/1221366789 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25fc011a80 con 0x7f260c108080 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.500+0000 7f2611400700 1 -- 172.21.15.97:0/1221366789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 msgr2=0x7f260c10a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.500+0000 7f2611400700 1 --2- 172.21.15.97:0/1221366789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 0x7f260c10a470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f25fc0098a0 tx=0x7f25fc009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.501+0000 7f2611400700 1 -- 172.21.15.97:0/1221366789 shutdown_connections 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.501+0000 7f2611400700 1 --2- 172.21.15.97:0/1221366789 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 0x7f260c10a470 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.501+0000 7f2611400700 1 -- 172.21.15.97:0/1221366789 >> 172.21.15.97:0/1221366789 conn(0x7f260c101a30 msgr2=0x7f260c103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.501+0000 7f2611400700 1 -- 172.21.15.97:0/1221366789 shutdown_connections 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.501+0000 7f2611400700 1 -- 172.21.15.97:0/1221366789 wait complete. 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.502+0000 7f2611400700 1 Processor -- start 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.502+0000 7f2611400700 1 -- start start 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.502+0000 7f2611400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 0x7f260c19d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.502+0000 7f2611400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c19da30 con 0x7f260c108080 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.502+0000 7f260a800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 0x7f260c19d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.502+0000 7f260a800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 0x7f260c19d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47890/0 (socket says 172.21.15.97:47890) 2024-03-11T19:47:38.529 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.502+0000 7f260a800700 1 -- 172.21.15.97:0/3884797110 learned_addr learned my addr 172.21.15.97:0/3884797110 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:38.530 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.503+0000 7f260a800700 1 -- 172.21.15.97:0/3884797110 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc009550 con 0x7f260c108080 2024-03-11T19:47:38.530 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.503+0000 7f260a800700 1 --2- 172.21.15.97:0/3884797110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 0x7f260c19d4f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009130 tx=0x7f25fc011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:38.530 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.503+0000 7f2603400700 1 -- 172.21.15.97:0/3884797110 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25fc003aa0 con 0x7f260c108080 2024-03-11T19:47:38.530 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.503+0000 7f2603400700 1 -- 172.21.15.97:0/3884797110 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25fc003c00 con 0x7f260c108080 2024-03-11T19:47:38.530 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.503+0000 7f2611400700 1 -- 172.21.15.97:0/3884797110 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f260c19dc30 con 0x7f260c108080 2024-03-11T19:47:38.530 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.503+0000 7f2603400700 1 -- 172.21.15.97:0/3884797110 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25fc0248a0 con 0x7f260c108080 2024-03-11T19:47:38.530 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.503+0000 7f2611400700 1 -- 172.21.15.97:0/3884797110 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f260c19e0d0 con 0x7f260c108080 2024-03-11T19:47:38.530 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.504+0000 7f2611400700 1 -- 172.21.15.97:0/3884797110 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f260c10b780 con 0x7f260c108080 2024-03-11T19:47:38.530 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.505+0000 7f2603400700 1 -- 172.21.15.97:0/3884797110 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f25fc01ac10 con 0x7f260c108080 2024-03-11T19:47:38.530 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.506+0000 7f2603400700 1 --2- 172.21.15.97:0/3884797110 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f25f0038d00 0x7f25f003b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.506+0000 7f2603400700 1 -- 172.21.15.97:0/3884797110 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f25fc04d430 con 0x7f260c108080 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.506+0000 7f2609e00700 1 -- 172.21.15.97:0/3884797110 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f25f0038d00 msgr2=0x7f25f003b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/1708150324 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.506+0000 7f2609e00700 1 --2- 172.21.15.97:0/3884797110 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f25f0038d00 0x7f25f003b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.510+0000 7f2603400700 1 -- 172.21.15.97:0/3884797110 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f25fc018760 con 0x7f260c108080 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.707+0000 7f2609e00700 1 -- 172.21.15.97:0/3884797110 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f25f0038d00 msgr2=0x7f25f003b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/1708150324 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.707+0000 7f2609e00700 1 --2- 172.21.15.97:0/3884797110 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f25f0038d00 0x7f25f003b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.790+0000 7f2611400700 1 -- 172.21.15.97:0/3884797110 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f260c1a0960 con 0x7f260c108080 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.791+0000 7f2603400700 1 -- 172.21.15.97:0/3884797110 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f25fc00b3e0 con 0x7f260c108080 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.793+0000 7f2611400700 1 -- 172.21.15.97:0/3884797110 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f25f0038d00 msgr2=0x7f25f003b1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.793+0000 7f2611400700 1 --2- 172.21.15.97:0/3884797110 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f25f0038d00 0x7f25f003b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.793+0000 7f2611400700 1 -- 172.21.15.97:0/3884797110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 msgr2=0x7f260c19d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.793+0000 7f2611400700 1 --2- 172.21.15.97:0/3884797110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 0x7f260c19d4f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009130 tx=0x7f25fc011b00 comp rx=0 tx=0).stop 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.793+0000 7f2611400700 1 -- 172.21.15.97:0/3884797110 shutdown_connections 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.793+0000 7f2611400700 1 --2- 172.21.15.97:0/3884797110 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f25f0038d00 0x7f25f003b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:38.531 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.793+0000 7f2611400700 1 --2- 172.21.15.97:0/3884797110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f260c108080 0x7f260c19d4f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:38.532 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.794+0000 7f2611400700 1 -- 172.21.15.97:0/3884797110 >> 172.21.15.97:0/3884797110 conn(0x7f260c101a30 msgr2=0x7f260c102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:38.532 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.794+0000 7f2611400700 1 -- 172.21.15.97:0/3884797110 shutdown_connections 2024-03-11T19:47:38.532 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:37.794+0000 7f2611400700 1 -- 172.21.15.97:0/3884797110 wait complete. 2024-03-11T19:47:38.532 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for the mgr to restart... 2024-03-11T19:47:38.532 INFO:teuthology.orchestra.run.smithi097.stdout:Waiting for mgr epoch 9... 2024-03-11T19:47:38.671 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:38.505+0000 7f208e3ba200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T19:47:38.987 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:38.735+0000 7f208e3ba200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T19:47:40.369 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:40 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:40.006+0000 7f208e3ba200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T19:47:40.369 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:40 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:40.185+0000 7f208e3ba200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T19:47:40.921 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:40 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:40.618+0000 7f208e3ba200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T19:47:40.922 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:40 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:40.745+0000 7f208e3ba200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T19:47:41.421 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:41 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:41.022+0000 7f208e3ba200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:47:41.421 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:41 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:41.267+0000 7f208e3ba200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T19:47:41.921 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:41 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:41.646+0000 7f208e3ba200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:47:42.265 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:41 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:41.960+0000 7f208e3ba200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T19:47:42.671 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:42 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:42.261+0000 7f208e3ba200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T19:47:43.671 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:43 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:43.260+0000 7f208e3ba200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T19:47:44.357 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:43 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:43.946+0000 7f208e3ba200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T19:47:44.357 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:44 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:44.074+0000 7f208e3ba200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T19:47:44.671 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:47:44 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:47:44.352+0000 7f208e3ba200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T19:47:44.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:44 smithi097 ceph-mon[31898]: Active manager daemon x restarted 2024-03-11T19:47:44.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:44 smithi097 ceph-mon[31898]: Activating manager daemon x 2024-03-11T19:47:44.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:44 smithi097 ceph-mon[31898]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T19:47:44.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:44 smithi097 ceph-mon[31898]: mgrmap e10: x(active, starting, since 0.00490627s) 2024-03-11T19:47:44.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:47:44.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T19:47:44.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T19:47:44.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T19:47:44.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T19:47:44.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:44 smithi097 ceph-mon[31898]: Manager daemon x is now available 2024-03-11T19:47:45.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:47:45.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:47:45.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T19:47:45.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T19:47:45.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:45 smithi097 ceph-mon[31898]: mgrmap e11: x(active, since 1.01221s) 2024-03-11T19:47:46.212 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout { 2024-03-11T19:47:46.213 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T19:47:46.213 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T19:47:46.213 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout } 2024-03-11T19:47:46.214 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.135+0000 7f917e400700 1 Processor -- start 2024-03-11T19:47:46.214 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.135+0000 7f917e400700 1 -- start start 2024-03-11T19:47:46.214 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.135+0000 7f917e400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 0x7f917807b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:46.214 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.135+0000 7f917e400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917807b830 con 0x7f917807cdf0 2024-03-11T19:47:46.214 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.136+0000 7f9177400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 0x7f917807b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:46.214 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.136+0000 7f9177400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 0x7f917807b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47892/0 (socket says 172.21.15.97:47892) 2024-03-11T19:47:46.214 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.136+0000 7f9177400700 1 -- 172.21.15.97:0/384889334 learned_addr learned my addr 172.21.15.97:0/384889334 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:46.215 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.136+0000 7f9177400700 1 -- 172.21.15.97:0/384889334 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917807b970 con 0x7f917807cdf0 2024-03-11T19:47:46.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.136+0000 7f9177400700 1 --2- 172.21.15.97:0/384889334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 0x7f917807b2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f91680098a0 tx=0x7f9168009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853d10a1432b85c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:46.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.137+0000 7f9176000700 1 -- 172.21.15.97:0/384889334 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9168004160 con 0x7f917807cdf0 2024-03-11T19:47:46.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.137+0000 7f9176000700 1 -- 172.21.15.97:0/384889334 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91680042c0 con 0x7f917807cdf0 2024-03-11T19:47:46.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.137+0000 7f9176000700 1 -- 172.21.15.97:0/384889334 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9168011a80 con 0x7f917807cdf0 2024-03-11T19:47:46.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.137+0000 7f917e400700 1 -- 172.21.15.97:0/384889334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 msgr2=0x7f917807b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:46.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.137+0000 7f917e400700 1 --2- 172.21.15.97:0/384889334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 0x7f917807b2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f91680098a0 tx=0x7f9168009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:46.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.137+0000 7f917e400700 1 -- 172.21.15.97:0/384889334 shutdown_connections 2024-03-11T19:47:46.222 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.138+0000 7f917e400700 1 --2- 172.21.15.97:0/384889334 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 0x7f917807b2f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.138+0000 7f917e400700 1 -- 172.21.15.97:0/384889334 >> 172.21.15.97:0/384889334 conn(0x7f9178102bd0 msgr2=0x7f9178104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.138+0000 7f917e400700 1 -- 172.21.15.97:0/384889334 shutdown_connections 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.138+0000 7f917e400700 1 -- 172.21.15.97:0/384889334 wait complete. 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.139+0000 7f917e400700 1 Processor -- start 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.139+0000 7f917e400700 1 -- start start 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.139+0000 7f917e400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 0x7f917819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.139+0000 7f917e400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917819d9a0 con 0x7f917807cdf0 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.139+0000 7f9177400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 0x7f917819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.139+0000 7f9177400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 0x7f917819d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47900/0 (socket says 172.21.15.97:47900) 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.139+0000 7f9177400700 1 -- 172.21.15.97:0/4160243849 learned_addr learned my addr 172.21.15.97:0/4160243849 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.140+0000 7f9177400700 1 -- 172.21.15.97:0/4160243849 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9168009550 con 0x7f917807cdf0 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.140+0000 7f9177400700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 0x7f917819d460 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9168009130 tx=0x7f9168011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.140+0000 7f916fe00700 1 -- 172.21.15.97:0/4160243849 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9168003aa0 con 0x7f917807cdf0 2024-03-11T19:47:46.223 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.140+0000 7f916fe00700 1 -- 172.21.15.97:0/4160243849 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9168003c00 con 0x7f917807cdf0 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.140+0000 7f916fe00700 1 -- 172.21.15.97:0/4160243849 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91680248a0 con 0x7f917807cdf0 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.140+0000 7f917e400700 1 -- 172.21.15.97:0/4160243849 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917819dba0 con 0x7f917807cdf0 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.140+0000 7f917e400700 1 -- 172.21.15.97:0/4160243849 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917819e040 con 0x7f917807cdf0 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.141+0000 7f916fe00700 1 -- 172.21.15.97:0/4160243849 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f916801a430 con 0x7f917807cdf0 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.142+0000 7f916fe00700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 0x7f915c03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.142+0000 7f916fe00700 1 -- 172.21.15.97:0/4160243849 --> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f915c03b930 con 0x7f915c038d60 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.142+0000 7f916fe00700 1 -- 172.21.15.97:0/4160243849 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f916804cf40 con 0x7f917807cdf0 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.142+0000 7f9176a00700 1 -- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 msgr2=0x7f915c03b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/1708150324 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.142+0000 7f9176a00700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 0x7f915c03b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.343+0000 7f9176a00700 1 -- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 msgr2=0x7f915c03b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/1708150324 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.343+0000 7f9176a00700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 0x7f915c03b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.744+0000 7f9176a00700 1 -- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 msgr2=0x7f915c03b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/1708150324 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:39.744+0000 7f9176a00700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 0x7f915c03b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T19:47:46.224 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:40.545+0000 7f9176a00700 1 -- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 msgr2=0x7f915c03b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/1708150324 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:40.545+0000 7f9176a00700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 0x7f915c03b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:42.147+0000 7f9176a00700 1 -- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 msgr2=0x7f915c03b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/1708150324 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:42.147+0000 7f9176a00700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 0x7f915c03b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:44.360+0000 7f916fe00700 1 -- 172.21.15.97:0/4160243849 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mgrmap(e 10) v1 ==== 45549+0+0 (secure 0 0 0) 0x7f916800a0d0 con 0x7f917807cdf0 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:44.361+0000 7f916fe00700 1 -- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 msgr2=0x7f915c03b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:44.361+0000 7f916fe00700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 0x7f915c03b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.365+0000 7f916fe00700 1 -- 172.21.15.97:0/4160243849 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mgrmap(e 11) v1 ==== 45676+0+0 (secure 0 0 0) 0x7f916804e420 con 0x7f917807cdf0 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.365+0000 7f916fe00700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f915c03c810 0x7f915c03ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.365+0000 7f916fe00700 1 -- 172.21.15.97:0/4160243849 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f915c03b930 con 0x7f915c03c810 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.366+0000 7f9176a00700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f915c03c810 0x7f915c03ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.368+0000 7f9176a00700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f915c03c810 0x7f915c03ec00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9164003a10 tx=0x7f9164009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.369+0000 7f916fe00700 1 -- 172.21.15.97:0/4160243849 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f915c03b930 con 0x7f915c03c810 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.376+0000 7f916d600700 1 -- 172.21.15.97:0/4160243849 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9150002600 con 0x7f915c03c810 2024-03-11T19:47:46.225 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.377+0000 7f916fe00700 1 -- 172.21.15.97:0/4160243849 <== mgr.14152 v2:172.21.15.97:6800/3389151509 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f9150002600 con 0x7f915c03c810 2024-03-11T19:47:46.226 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.377+0000 7f916d600700 1 -- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f915c03c810 msgr2=0x7f915c03ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:46.226 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.377+0000 7f916d600700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f915c03c810 0x7f915c03ec00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9164003a10 tx=0x7f9164009270 comp rx=0 tx=0).stop 2024-03-11T19:47:46.226 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.377+0000 7f916d600700 1 -- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 msgr2=0x7f917819d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:46.226 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.377+0000 7f916d600700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 0x7f917819d460 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9168009130 tx=0x7f9168011b00 comp rx=0 tx=0).stop 2024-03-11T19:47:46.226 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.378+0000 7f916d600700 1 -- 172.21.15.97:0/4160243849 shutdown_connections 2024-03-11T19:47:46.226 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.378+0000 7f916d600700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f915c03c810 0x7f915c03ec00 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:46.226 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.378+0000 7f916d600700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f917807cdf0 0x7f917819d460 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:46.226 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.378+0000 7f916d600700 1 --2- 172.21.15.97:0/4160243849 >> [v2:172.21.15.97:6800/1708150324,v1:172.21.15.97:6801/1708150324] conn(0x7f915c038d60 0x7f915c03b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:46.226 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.378+0000 7f916d600700 1 -- 172.21.15.97:0/4160243849 >> 172.21.15.97:0/4160243849 conn(0x7f9178102bd0 msgr2=0x7f9178103700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:46.226 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.378+0000 7f916d600700 1 -- 172.21.15.97:0/4160243849 shutdown_connections 2024-03-11T19:47:46.226 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:45.378+0000 7f916d600700 1 -- 172.21.15.97:0/4160243849 wait complete. 2024-03-11T19:47:46.227 INFO:teuthology.orchestra.run.smithi097.stdout:mgr epoch 9 is available 2024-03-11T19:47:46.227 INFO:teuthology.orchestra.run.smithi097.stdout:Generating a dashboard self-signed certificate... 2024-03-11T19:47:46.528 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:46 smithi097 ceph-mon[31898]: [11/Mar/2024:19:47:44] ENGINE Bus STARTING 2024-03-11T19:47:46.528 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:46 smithi097 ceph-mon[31898]: [11/Mar/2024:19:47:44] ENGINE Serving on https://172.21.15.97:7150 2024-03-11T19:47:46.528 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:46 smithi097 ceph-mon[31898]: [11/Mar/2024:19:47:44] ENGINE Serving on http://172.21.15.97:8765 2024-03-11T19:47:46.529 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:46 smithi097 ceph-mon[31898]: [11/Mar/2024:19:47:44] ENGINE Bus STARTED 2024-03-11T19:47:46.529 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:46 smithi097 ceph-mon[31898]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T19:47:46.529 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:46 smithi097 ceph-mon[31898]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T19:47:47.527 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:47 smithi097 ceph-mon[31898]: mgrmap e12: x(active, since 2s) 2024-03-11T19:47:48.560 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T19:47:48.560 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.418+0000 7f8600e00700 1 Processor -- start 2024-03-11T19:47:48.560 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.418+0000 7f8600e00700 1 -- start start 2024-03-11T19:47:48.560 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.418+0000 7f8600e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 0x7f85fc107970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:48.560 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.418+0000 7f8600e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc107f40 con 0x7f85fc107550 2024-03-11T19:47:48.560 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.419+0000 7f85f9e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 0x7f85fc107970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:48.560 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.419+0000 7f85f9e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 0x7f85fc107970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57718/0 (socket says 172.21.15.97:57718) 2024-03-11T19:47:48.561 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.419+0000 7f85f9e00700 1 -- 172.21.15.97:0/566588046 learned_addr learned my addr 172.21.15.97:0/566588046 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:48.561 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.419+0000 7f85f9e00700 1 -- 172.21.15.97:0/566588046 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85fc108700 con 0x7f85fc107550 2024-03-11T19:47:48.561 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.420+0000 7f85f9e00700 1 --2- 172.21.15.97:0/566588046 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 0x7f85fc107970 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0098a0 tx=0x7f85ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f0ef631e24f846f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:48.561 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.421+0000 7f85f9400700 1 -- 172.21.15.97:0/566588046 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f85ec004160 con 0x7f85fc107550 2024-03-11T19:47:48.561 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.421+0000 7f85f9400700 1 -- 172.21.15.97:0/566588046 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85ec0042c0 con 0x7f85fc107550 2024-03-11T19:47:48.561 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.421+0000 7f85f9400700 1 -- 172.21.15.97:0/566588046 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f85ec011a80 con 0x7f85fc107550 2024-03-11T19:47:48.561 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.422+0000 7f8600e00700 1 -- 172.21.15.97:0/566588046 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 msgr2=0x7f85fc107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:48.561 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.422+0000 7f8600e00700 1 --2- 172.21.15.97:0/566588046 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 0x7f85fc107970 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0098a0 tx=0x7f85ec009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:48.561 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.422+0000 7f8600e00700 1 -- 172.21.15.97:0/566588046 shutdown_connections 2024-03-11T19:47:48.561 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.422+0000 7f8600e00700 1 --2- 172.21.15.97:0/566588046 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 0x7f85fc107970 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:48.562 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.422+0000 7f8600e00700 1 -- 172.21.15.97:0/566588046 >> 172.21.15.97:0/566588046 conn(0x7f85fc102bd0 msgr2=0x7f85fc104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:48.562 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.422+0000 7f8600e00700 1 -- 172.21.15.97:0/566588046 shutdown_connections 2024-03-11T19:47:48.562 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.423+0000 7f8600e00700 1 -- 172.21.15.97:0/566588046 wait complete. 2024-03-11T19:47:48.562 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.423+0000 7f8600e00700 1 Processor -- start 2024-03-11T19:47:48.562 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.424+0000 7f8600e00700 1 -- start start 2024-03-11T19:47:48.562 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.424+0000 7f8600e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 0x7f85fc19d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:48.562 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.424+0000 7f8600e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc19d990 con 0x7f85fc107550 2024-03-11T19:47:48.562 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.424+0000 7f85f9e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 0x7f85fc19d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:48.562 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.424+0000 7f85f9e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 0x7f85fc19d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57720/0 (socket says 172.21.15.97:57720) 2024-03-11T19:47:48.563 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.424+0000 7f85f9e00700 1 -- 172.21.15.97:0/3255697687 learned_addr learned my addr 172.21.15.97:0/3255697687 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:48.563 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.425+0000 7f85f9e00700 1 -- 172.21.15.97:0/3255697687 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec009550 con 0x7f85fc107550 2024-03-11T19:47:48.563 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.425+0000 7f85f9e00700 1 --2- 172.21.15.97:0/3255697687 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 0x7f85fc19d450 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f85ec011740 tx=0x7f85ec011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:48.563 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.425+0000 7f85f2a00700 1 -- 172.21.15.97:0/3255697687 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f85ec003b20 con 0x7f85fc107550 2024-03-11T19:47:48.563 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.425+0000 7f85f2a00700 1 -- 172.21.15.97:0/3255697687 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85ec01a910 con 0x7f85fc107550 2024-03-11T19:47:48.564 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.425+0000 7f8600e00700 1 -- 172.21.15.97:0/3255697687 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85fc19db90 con 0x7f85fc107550 2024-03-11T19:47:48.564 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.425+0000 7f85f2a00700 1 -- 172.21.15.97:0/3255697687 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f85ec024850 con 0x7f85fc107550 2024-03-11T19:47:48.564 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.425+0000 7f8600e00700 1 -- 172.21.15.97:0/3255697687 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85fc19e030 con 0x7f85fc107550 2024-03-11T19:47:48.564 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.426+0000 7f85f2a00700 1 -- 172.21.15.97:0/3255697687 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f85ec01aa80 con 0x7f85fc107550 2024-03-11T19:47:48.564 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.427+0000 7f8600e00700 1 -- 172.21.15.97:0/3255697687 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85fc10b930 con 0x7f85fc107550 2024-03-11T19:47:48.564 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.427+0000 7f85f2a00700 1 --2- 172.21.15.97:0/3255697687 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f85e0038d60 0x7f85e003b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:48.564 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.427+0000 7f85f2a00700 1 -- 172.21.15.97:0/3255697687 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f85ec04d0c0 con 0x7f85fc107550 2024-03-11T19:47:48.565 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.427+0000 7f85f3e00700 1 --2- 172.21.15.97:0/3255697687 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f85e0038d60 0x7f85e003b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:48.565 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.428+0000 7f85f3e00700 1 --2- 172.21.15.97:0/3255697687 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f85e0038d60 0x7f85e003b220 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f85e4006fd0 tx=0x7f85e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:48.565 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.433+0000 7f85f2a00700 1 -- 172.21.15.97:0/3255697687 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f85ec01ae10 con 0x7f85fc107550 2024-03-11T19:47:48.565 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.660+0000 7f8600e00700 1 -- 172.21.15.97:0/3255697687 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f85fc103d00 con 0x7f85e0038d60 2024-03-11T19:47:48.565 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.813+0000 7f85f2a00700 1 -- 172.21.15.97:0/3255697687 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f85fc103d00 con 0x7f85e0038d60 2024-03-11T19:47:48.566 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.816+0000 7f8600e00700 1 -- 172.21.15.97:0/3255697687 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f85e0038d60 msgr2=0x7f85e003b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:48.566 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.816+0000 7f8600e00700 1 --2- 172.21.15.97:0/3255697687 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f85e0038d60 0x7f85e003b220 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f85e4006fd0 tx=0x7f85e4006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:48.566 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.816+0000 7f8600e00700 1 -- 172.21.15.97:0/3255697687 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 msgr2=0x7f85fc19d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:48.566 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.816+0000 7f8600e00700 1 --2- 172.21.15.97:0/3255697687 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 0x7f85fc19d450 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f85ec011740 tx=0x7f85ec011b00 comp rx=0 tx=0).stop 2024-03-11T19:47:48.566 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.817+0000 7f8600e00700 1 -- 172.21.15.97:0/3255697687 shutdown_connections 2024-03-11T19:47:48.566 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.817+0000 7f8600e00700 1 --2- 172.21.15.97:0/3255697687 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f85e0038d60 0x7f85e003b220 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:48.566 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.817+0000 7f8600e00700 1 --2- 172.21.15.97:0/3255697687 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f85fc107550 0x7f85fc19d450 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:48.566 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.817+0000 7f8600e00700 1 -- 172.21.15.97:0/3255697687 >> 172.21.15.97:0/3255697687 conn(0x7f85fc102bd0 msgr2=0x7f85fc1035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:48.566 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.817+0000 7f8600e00700 1 -- 172.21.15.97:0/3255697687 shutdown_connections 2024-03-11T19:47:48.567 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:47.817+0000 7f8600e00700 1 -- 172.21.15.97:0/3255697687 wait complete. 2024-03-11T19:47:48.567 INFO:teuthology.orchestra.run.smithi097.stdout:Creating initial admin user... 2024-03-11T19:47:48.861 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:47:48.861 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:47:48.861 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:47:49.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:49 smithi097 ceph-mon[31898]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:47:49.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:49 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:47:50.379 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$xgwrsKS3gzSK8Y7e16QQJuel42d4PL3iFldYi5loJfo1NrnmrU92G", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710186469, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T19:47:50.379 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.155+0000 7f7a52200700 1 Processor -- start 2024-03-11T19:47:50.379 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.155+0000 7f7a52200700 1 -- start start 2024-03-11T19:47:50.384 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.155+0000 7f7a52200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 0x7f7a4c10a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:50.384 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.155+0000 7f7a52200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c07aec0 con 0x7f7a4c107fc0 2024-03-11T19:47:50.384 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.156+0000 7f7a4b400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 0x7f7a4c10a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:50.384 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.156+0000 7f7a4b400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 0x7f7a4c10a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57722/0 (socket says 172.21.15.97:57722) 2024-03-11T19:47:50.384 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.156+0000 7f7a4b400700 1 -- 172.21.15.97:0/1300451094 learned_addr learned my addr 172.21.15.97:0/1300451094 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:50.384 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.156+0000 7f7a4b400700 1 -- 172.21.15.97:0/1300451094 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c10a8f0 con 0x7f7a4c107fc0 2024-03-11T19:47:50.384 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.157+0000 7f7a4b400700 1 --2- 172.21.15.97:0/1300451094 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 0x7f7a4c10a3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c0098a0 tx=0x7f7a3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eb2db3145e00eba server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:50.384 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.157+0000 7f7a4a000700 1 -- 172.21.15.97:0/1300451094 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a3c004160 con 0x7f7a4c107fc0 2024-03-11T19:47:50.384 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.157+0000 7f7a4a000700 1 -- 172.21.15.97:0/1300451094 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a3c0042c0 con 0x7f7a4c107fc0 2024-03-11T19:47:50.384 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.157+0000 7f7a4a000700 1 -- 172.21.15.97:0/1300451094 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a3c011a80 con 0x7f7a4c107fc0 2024-03-11T19:47:50.384 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.158+0000 7f7a52200700 1 -- 172.21.15.97:0/1300451094 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 msgr2=0x7f7a4c10a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:50.384 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.158+0000 7f7a52200700 1 --2- 172.21.15.97:0/1300451094 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 0x7f7a4c10a3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c0098a0 tx=0x7f7a3c009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:50.384 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.158+0000 7f7a52200700 1 -- 172.21.15.97:0/1300451094 shutdown_connections 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.158+0000 7f7a52200700 1 --2- 172.21.15.97:0/1300451094 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 0x7f7a4c10a3b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.158+0000 7f7a52200700 1 -- 172.21.15.97:0/1300451094 >> 172.21.15.97:0/1300451094 conn(0x7f7a4c101a30 msgr2=0x7f7a4c103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.158+0000 7f7a52200700 1 -- 172.21.15.97:0/1300451094 shutdown_connections 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.158+0000 7f7a52200700 1 -- 172.21.15.97:0/1300451094 wait complete. 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.160+0000 7f7a52200700 1 Processor -- start 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.160+0000 7f7a52200700 1 -- start start 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.160+0000 7f7a52200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 0x7f7a4c19b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.160+0000 7f7a52200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c19b790 con 0x7f7a4c107fc0 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.161+0000 7f7a4b400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 0x7f7a4c19b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.161+0000 7f7a4b400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 0x7f7a4c19b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57726/0 (socket says 172.21.15.97:57726) 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.161+0000 7f7a4b400700 1 -- 172.21.15.97:0/2368087846 learned_addr learned my addr 172.21.15.97:0/2368087846 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.161+0000 7f7a4b400700 1 -- 172.21.15.97:0/2368087846 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c009550 con 0x7f7a4c107fc0 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.161+0000 7f7a4b400700 1 --2- 172.21.15.97:0/2368087846 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 0x7f7a4c19b250 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009130 tx=0x7f7a3c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.162+0000 7f7a43e00700 1 -- 172.21.15.97:0/2368087846 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a3c003aa0 con 0x7f7a4c107fc0 2024-03-11T19:47:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.162+0000 7f7a43e00700 1 -- 172.21.15.97:0/2368087846 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a3c003c00 con 0x7f7a4c107fc0 2024-03-11T19:47:50.386 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.162+0000 7f7a52200700 1 -- 172.21.15.97:0/2368087846 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4c19b990 con 0x7f7a4c107fc0 2024-03-11T19:47:50.386 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.162+0000 7f7a43e00700 1 -- 172.21.15.97:0/2368087846 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a3c0248a0 con 0x7f7a4c107fc0 2024-03-11T19:47:50.386 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.162+0000 7f7a52200700 1 -- 172.21.15.97:0/2368087846 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4c19be30 con 0x7f7a4c107fc0 2024-03-11T19:47:50.386 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.163+0000 7f7a43e00700 1 -- 172.21.15.97:0/2368087846 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f7a3c024ae0 con 0x7f7a4c107fc0 2024-03-11T19:47:50.386 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.163+0000 7f7a52200700 1 -- 172.21.15.97:0/2368087846 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4c10b790 con 0x7f7a4c107fc0 2024-03-11T19:47:50.386 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.164+0000 7f7a43e00700 1 --2- 172.21.15.97:0/2368087846 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7a30038d00 0x7f7a3003b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:50.386 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.164+0000 7f7a43e00700 1 -- 172.21.15.97:0/2368087846 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7a3c04d050 con 0x7f7a4c107fc0 2024-03-11T19:47:50.386 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.164+0000 7f7a4aa00700 1 --2- 172.21.15.97:0/2368087846 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7a30038d00 0x7f7a3003b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:50.386 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.165+0000 7f7a4aa00700 1 --2- 172.21.15.97:0/2368087846 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7a30038d00 0x7f7a3003b1c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a38006fd0 tx=0x7f7a38006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:50.386 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.170+0000 7f7a43e00700 1 -- 172.21.15.97:0/2368087846 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7a3c080d20 con 0x7f7a4c107fc0 2024-03-11T19:47:50.386 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.409+0000 7f7a52200700 1 -- 172.21.15.97:0/2368087846 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7a4c0630c0 con 0x7f7a30038d00 2024-03-11T19:47:50.386 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.672+0000 7f7a43e00700 1 -- 172.21.15.97:0/2368087846 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f7a4c0630c0 con 0x7f7a30038d00 2024-03-11T19:47:50.386 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.675+0000 7f7a52200700 1 -- 172.21.15.97:0/2368087846 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7a30038d00 msgr2=0x7f7a3003b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:50.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.675+0000 7f7a52200700 1 --2- 172.21.15.97:0/2368087846 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7a30038d00 0x7f7a3003b1c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a38006fd0 tx=0x7f7a38006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:50.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.675+0000 7f7a52200700 1 -- 172.21.15.97:0/2368087846 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 msgr2=0x7f7a4c19b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:50.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.675+0000 7f7a52200700 1 --2- 172.21.15.97:0/2368087846 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 0x7f7a4c19b250 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009130 tx=0x7f7a3c011b00 comp rx=0 tx=0).stop 2024-03-11T19:47:50.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.675+0000 7f7a52200700 1 -- 172.21.15.97:0/2368087846 shutdown_connections 2024-03-11T19:47:50.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.675+0000 7f7a52200700 1 --2- 172.21.15.97:0/2368087846 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7a30038d00 0x7f7a3003b1c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:50.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.675+0000 7f7a52200700 1 --2- 172.21.15.97:0/2368087846 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7a4c107fc0 0x7f7a4c19b250 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:50.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.675+0000 7f7a52200700 1 -- 172.21.15.97:0/2368087846 >> 172.21.15.97:0/2368087846 conn(0x7f7a4c101a30 msgr2=0x7f7a4c1024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:50.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.675+0000 7f7a52200700 1 -- 172.21.15.97:0/2368087846 shutdown_connections 2024-03-11T19:47:50.387 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:49.675+0000 7f7a52200700 1 -- 172.21.15.97:0/2368087846 wait complete. 2024-03-11T19:47:50.387 INFO:teuthology.orchestra.run.smithi097.stdout:Fetching dashboard port number... 2024-03-11T19:47:50.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:50 smithi097 ceph-mon[31898]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:47:50.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:50 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:47:52.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:51 smithi097 ceph-mon[31898]: mgrmap e13: x(active, since 6s) 2024-03-11T19:47:52.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:51 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1748940395' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T19:47:52.100 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T19:47:52.100 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.105+0000 7f797ac00700 1 Processor -- start 2024-03-11T19:47:52.100 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.106+0000 7f797ac00700 1 -- start start 2024-03-11T19:47:52.106 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.106+0000 7f797ac00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 0x7f797410e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:52.106 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.106+0000 7f797ac00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797410ea10 con 0x7f797410e020 2024-03-11T19:47:52.106 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.106+0000 7f7973e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 0x7f797410e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:52.106 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.106+0000 7f7973e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 0x7f797410e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57736/0 (socket says 172.21.15.97:57736) 2024-03-11T19:47:52.106 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.106+0000 7f7973e00700 1 -- 172.21.15.97:0/879767250 learned_addr learned my addr 172.21.15.97:0/879767250 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.107+0000 7f7973e00700 1 -- 172.21.15.97:0/879767250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797410eb50 con 0x7f797410e020 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.107+0000 7f7973e00700 1 --2- 172.21.15.97:0/879767250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 0x7f797410e440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79640098a0 tx=0x7f7964009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202cd1713c2201f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.108+0000 7f7972a00700 1 -- 172.21.15.97:0/879767250 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7964004160 con 0x7f797410e020 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.108+0000 7f7972a00700 1 -- 172.21.15.97:0/879767250 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79640042c0 con 0x7f797410e020 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.108+0000 7f7972a00700 1 -- 172.21.15.97:0/879767250 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7964011a80 con 0x7f797410e020 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.108+0000 7f797ac00700 1 -- 172.21.15.97:0/879767250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 msgr2=0x7f797410e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.108+0000 7f797ac00700 1 --2- 172.21.15.97:0/879767250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 0x7f797410e440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79640098a0 tx=0x7f7964009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.108+0000 7f797ac00700 1 -- 172.21.15.97:0/879767250 shutdown_connections 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.108+0000 7f797ac00700 1 --2- 172.21.15.97:0/879767250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 0x7f797410e440 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.108+0000 7f797ac00700 1 -- 172.21.15.97:0/879767250 >> 172.21.15.97:0/879767250 conn(0x7f797407ce20 msgr2=0x7f797407f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.108+0000 7f797ac00700 1 -- 172.21.15.97:0/879767250 shutdown_connections 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.109+0000 7f797ac00700 1 -- 172.21.15.97:0/879767250 wait complete. 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.109+0000 7f797ac00700 1 Processor -- start 2024-03-11T19:47:52.107 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.110+0000 7f797ac00700 1 -- start start 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.110+0000 7f797ac00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 0x7f79741a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.110+0000 7f797ac00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79741a1ce0 con 0x7f797410e020 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.110+0000 7f7973e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 0x7f79741a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.110+0000 7f7973e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 0x7f79741a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57746/0 (socket says 172.21.15.97:57746) 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.110+0000 7f7973e00700 1 -- 172.21.15.97:0/1748940395 learned_addr learned my addr 172.21.15.97:0/1748940395 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.111+0000 7f7973e00700 1 -- 172.21.15.97:0/1748940395 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7964009550 con 0x7f797410e020 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.111+0000 7f7973e00700 1 --2- 172.21.15.97:0/1748940395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 0x7f79741a17a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7964011740 tx=0x7f7964011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.111+0000 7f7970c00700 1 -- 172.21.15.97:0/1748940395 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7964003b20 con 0x7f797410e020 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.111+0000 7f7970c00700 1 -- 172.21.15.97:0/1748940395 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f796401a910 con 0x7f797410e020 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.111+0000 7f7970c00700 1 -- 172.21.15.97:0/1748940395 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79640248c0 con 0x7f797410e020 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.111+0000 7f797ac00700 1 -- 172.21.15.97:0/1748940395 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79741a1ee0 con 0x7f797410e020 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.111+0000 7f797ac00700 1 -- 172.21.15.97:0/1748940395 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79741a2380 con 0x7f797410e020 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.112+0000 7f7970c00700 1 -- 172.21.15.97:0/1748940395 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f796401a430 con 0x7f797410e020 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.112+0000 7f797ac00700 1 -- 172.21.15.97:0/1748940395 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7974111fc0 con 0x7f797410e020 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.113+0000 7f7970c00700 1 --2- 172.21.15.97:0/1748940395 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7958038e00 0x7f795803b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.113+0000 7f7970c00700 1 -- 172.21.15.97:0/1748940395 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f796404d010 con 0x7f797410e020 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.113+0000 7f7973400700 1 --2- 172.21.15.97:0/1748940395 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7958038e00 0x7f795803b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.114+0000 7f7973400700 1 --2- 172.21.15.97:0/1748940395 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7958038e00 0x7f795803b2c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7960006fd0 tx=0x7f7960006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.118+0000 7f7970c00700 1 -- 172.21.15.97:0/1748940395 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f796401aa80 con 0x7f797410e020 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.331+0000 7f797ac00700 1 -- 172.21.15.97:0/1748940395 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f797410eb50 con 0x7f797410e020 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.332+0000 7f7970c00700 1 -- 172.21.15.97:0/1748940395 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f797410eb50 con 0x7f797410e020 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.335+0000 7f797ac00700 1 -- 172.21.15.97:0/1748940395 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7958038e00 msgr2=0x7f795803b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.335+0000 7f797ac00700 1 --2- 172.21.15.97:0/1748940395 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7958038e00 0x7f795803b2c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7960006fd0 tx=0x7f7960006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.335+0000 7f797ac00700 1 -- 172.21.15.97:0/1748940395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 msgr2=0x7f79741a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.335+0000 7f797ac00700 1 --2- 172.21.15.97:0/1748940395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 0x7f79741a17a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7964011740 tx=0x7f7964011b00 comp rx=0 tx=0).stop 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.335+0000 7f797ac00700 1 -- 172.21.15.97:0/1748940395 shutdown_connections 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.335+0000 7f797ac00700 1 --2- 172.21.15.97:0/1748940395 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7958038e00 0x7f795803b2c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.335+0000 7f797ac00700 1 --2- 172.21.15.97:0/1748940395 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f797410e020 0x7f79741a17a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.335+0000 7f797ac00700 1 -- 172.21.15.97:0/1748940395 >> 172.21.15.97:0/1748940395 conn(0x7f797407ce20 msgr2=0x7f797407e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.335+0000 7f797ac00700 1 -- 172.21.15.97:0/1748940395 shutdown_connections 2024-03-11T19:47:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:51.335+0000 7f797ac00700 1 -- 172.21.15.97:0/1748940395 wait complete. 2024-03-11T19:47:52.131 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T19:47:52.132 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout disabled 2024-03-11T19:47:52.153 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T19:47:52.153 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stdout inactive 2024-03-11T19:47:52.153 INFO:teuthology.orchestra.run.smithi097.stdout:firewalld.service is not enabled 2024-03-11T19:47:52.153 INFO:teuthology.orchestra.run.smithi097.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T19:47:52.157 INFO:teuthology.orchestra.run.smithi097.stdout:Ceph Dashboard is now available at: 2024-03-11T19:47:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:47:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: URL: https://smithi097.front.sepia.ceph.com:8443/ 2024-03-11T19:47:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: User: admin 2024-03-11T19:47:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: Password: twc4srx7py 2024-03-11T19:47:52.157 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:47:52.158 INFO:teuthology.orchestra.run.smithi097.stdout:Saving cluster configuration to /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config directory 2024-03-11T19:47:52.159 INFO:teuthology.orchestra.run.smithi097.stdout:Enabling autotune for osd_memory_target 2024-03-11T19:47:54.246 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.262+0000 7fac61600700 1 Processor -- start 2024-03-11T19:47:54.246 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.263+0000 7fac61600700 1 -- start start 2024-03-11T19:47:54.246 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.263+0000 7fac61600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 0x7fac5c109a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:54.246 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.263+0000 7fac61600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c109ff0 con 0x7fac5c109600 2024-03-11T19:47:54.246 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.263+0000 7fac5be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 0x7fac5c109a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:54.246 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.264+0000 7fac5be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 0x7fac5c109a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57762/0 (socket says 172.21.15.97:57762) 2024-03-11T19:47:54.246 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.264+0000 7fac5be00700 1 -- 172.21.15.97:0/3390600155 learned_addr learned my addr 172.21.15.97:0/3390600155 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:54.246 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.264+0000 7fac5be00700 1 -- 172.21.15.97:0/3390600155 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c10a760 con 0x7fac5c109600 2024-03-11T19:47:54.246 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.264+0000 7fac5be00700 1 --2- 172.21.15.97:0/3390600155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 0x7fac5c109a20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0098a0 tx=0x7fac4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7057c727644a8366 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.264+0000 7fac5aa00700 1 -- 172.21.15.97:0/3390600155 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac4c004160 con 0x7fac5c109600 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.265+0000 7fac5aa00700 1 -- 172.21.15.97:0/3390600155 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac4c0042c0 con 0x7fac5c109600 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.265+0000 7fac5aa00700 1 -- 172.21.15.97:0/3390600155 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac4c011a80 con 0x7fac5c109600 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.265+0000 7fac61600700 1 -- 172.21.15.97:0/3390600155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 msgr2=0x7fac5c109a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.265+0000 7fac61600700 1 --2- 172.21.15.97:0/3390600155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 0x7fac5c109a20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0098a0 tx=0x7fac4c009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.265+0000 7fac61600700 1 -- 172.21.15.97:0/3390600155 shutdown_connections 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.265+0000 7fac61600700 1 --2- 172.21.15.97:0/3390600155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 0x7fac5c109a20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.265+0000 7fac61600700 1 -- 172.21.15.97:0/3390600155 >> 172.21.15.97:0/3390600155 conn(0x7fac5c07cc20 msgr2=0x7fac5c1074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.265+0000 7fac61600700 1 -- 172.21.15.97:0/3390600155 shutdown_connections 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.266+0000 7fac61600700 1 -- 172.21.15.97:0/3390600155 wait complete. 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.266+0000 7fac61600700 1 Processor -- start 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.267+0000 7fac61600700 1 -- start start 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.267+0000 7fac61600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 0x7fac5c19d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.267+0000 7fac61600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c19d760 con 0x7fac5c109600 2024-03-11T19:47:54.247 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.267+0000 7fac5be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 0x7fac5c19d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.267+0000 7fac5be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 0x7fac5c19d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57766/0 (socket says 172.21.15.97:57766) 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.267+0000 7fac5be00700 1 -- 172.21.15.97:0/2762713317 learned_addr learned my addr 172.21.15.97:0/2762713317 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.268+0000 7fac5be00700 1 -- 172.21.15.97:0/2762713317 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4c009550 con 0x7fac5c109600 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.268+0000 7fac5be00700 1 --2- 172.21.15.97:0/2762713317 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 0x7fac5c19d220 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fac4c011740 tx=0x7fac4c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.268+0000 7fac58c00700 1 -- 172.21.15.97:0/2762713317 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac4c003af0 con 0x7fac5c109600 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.268+0000 7fac58c00700 1 -- 172.21.15.97:0/2762713317 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac4c01a910 con 0x7fac5c109600 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.268+0000 7fac61600700 1 -- 172.21.15.97:0/2762713317 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5c19d960 con 0x7fac5c109600 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.268+0000 7fac58c00700 1 -- 172.21.15.97:0/2762713317 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac4c024870 con 0x7fac5c109600 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.268+0000 7fac61600700 1 -- 172.21.15.97:0/2762713317 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5c19de00 con 0x7fac5c109600 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.270+0000 7fac61600700 1 -- 172.21.15.97:0/2762713317 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5c10d990 con 0x7fac5c109600 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.270+0000 7fac58c00700 1 -- 172.21.15.97:0/2762713317 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fac4c01a430 con 0x7fac5c109600 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.270+0000 7fac58c00700 1 --2- 172.21.15.97:0/2762713317 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fac40038d50 0x7fac4003b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.270+0000 7fac58c00700 1 -- 172.21.15.97:0/2762713317 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fac4c04d260 con 0x7fac5c109600 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.271+0000 7fac5b400700 1 --2- 172.21.15.97:0/2762713317 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fac40038d50 0x7fac4003b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:54.248 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.271+0000 7fac5b400700 1 --2- 172.21.15.97:0/2762713317 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fac40038d50 0x7fac4003b210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fac44006fd0 tx=0x7fac44006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.276+0000 7fac58c00700 1 -- 172.21.15.97:0/2762713317 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fac4c018530 con 0x7fac5c109600 2024-03-11T19:47:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.497+0000 7fac61600700 1 -- 172.21.15.97:0/2762713317 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fac5c1a06e0 con 0x7fac5c109600 2024-03-11T19:47:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.498+0000 7fac58c00700 1 -- 172.21.15.97:0/2762713317 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fac4c018350 con 0x7fac5c109600 2024-03-11T19:47:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.501+0000 7fac61600700 1 -- 172.21.15.97:0/2762713317 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fac40038d50 msgr2=0x7fac4003b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.501+0000 7fac61600700 1 --2- 172.21.15.97:0/2762713317 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fac40038d50 0x7fac4003b210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fac44006fd0 tx=0x7fac44006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.501+0000 7fac61600700 1 -- 172.21.15.97:0/2762713317 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 msgr2=0x7fac5c19d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.501+0000 7fac61600700 1 --2- 172.21.15.97:0/2762713317 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 0x7fac5c19d220 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fac4c011740 tx=0x7fac4c011a50 comp rx=0 tx=0).stop 2024-03-11T19:47:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.501+0000 7fac61600700 1 -- 172.21.15.97:0/2762713317 shutdown_connections 2024-03-11T19:47:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.501+0000 7fac61600700 1 --2- 172.21.15.97:0/2762713317 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fac40038d50 0x7fac4003b210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.501+0000 7fac61600700 1 --2- 172.21.15.97:0/2762713317 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac5c109600 0x7fac5c19d220 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.501+0000 7fac61600700 1 -- 172.21.15.97:0/2762713317 >> 172.21.15.97:0/2762713317 conn(0x7fac5c07cc20 msgr2=0x7fac5c106b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.501+0000 7fac61600700 1 -- 172.21.15.97:0/2762713317 shutdown_connections 2024-03-11T19:47:54.249 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:53.501+0000 7fac61600700 1 -- 172.21.15.97:0/2762713317 wait complete. 2024-03-11T19:47:55.916 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.869+0000 7f6976a00700 1 Processor -- start 2024-03-11T19:47:55.916 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.869+0000 7f6976a00700 1 -- start start 2024-03-11T19:47:55.916 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.870+0000 7f6976a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 0x7f697007b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:55.916 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.870+0000 7f6976a00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697007b920 con 0x7f697007cea0 2024-03-11T19:47:55.916 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.870+0000 7f696fe00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 0x7f697007b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:55.916 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.870+0000 7f696fe00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 0x7f697007b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58932/0 (socket says 172.21.15.97:58932) 2024-03-11T19:47:55.916 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.870+0000 7f696fe00700 1 -- 172.21.15.97:0/3633866180 learned_addr learned my addr 172.21.15.97:0/3633866180 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:55.916 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.871+0000 7f696fe00700 1 -- 172.21.15.97:0/3633866180 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697007ba60 con 0x7f697007cea0 2024-03-11T19:47:55.916 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.871+0000 7f696fe00700 1 --2- 172.21.15.97:0/3633866180 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 0x7f697007b350 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f69600098a0 tx=0x7f6960009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9c98e38784611a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:55.916 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.871+0000 7f696ea00700 1 -- 172.21.15.97:0/3633866180 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6960004160 con 0x7f697007cea0 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.871+0000 7f696ea00700 1 -- 172.21.15.97:0/3633866180 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69600042c0 con 0x7f697007cea0 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.871+0000 7f696ea00700 1 -- 172.21.15.97:0/3633866180 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6960011a80 con 0x7f697007cea0 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.872+0000 7f6976a00700 1 -- 172.21.15.97:0/3633866180 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 msgr2=0x7f697007b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.872+0000 7f6976a00700 1 --2- 172.21.15.97:0/3633866180 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 0x7f697007b350 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f69600098a0 tx=0x7f6960009bb0 comp rx=0 tx=0).stop 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.872+0000 7f6976a00700 1 -- 172.21.15.97:0/3633866180 shutdown_connections 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.872+0000 7f6976a00700 1 --2- 172.21.15.97:0/3633866180 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 0x7f697007b350 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.872+0000 7f6976a00700 1 -- 172.21.15.97:0/3633866180 >> 172.21.15.97:0/3633866180 conn(0x7f6970102bd0 msgr2=0x7f6970104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.872+0000 7f6976a00700 1 -- 172.21.15.97:0/3633866180 shutdown_connections 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.872+0000 7f6976a00700 1 -- 172.21.15.97:0/3633866180 wait complete. 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.873+0000 7f6976a00700 1 Processor -- start 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.874+0000 7f6976a00700 1 -- start start 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.874+0000 7f6976a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 0x7f69701a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.874+0000 7f6976a00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69701a1cf0 con 0x7f697007cea0 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.874+0000 7f696fe00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 0x7f69701a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.874+0000 7f696fe00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 0x7f69701a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58946/0 (socket says 172.21.15.97:58946) 2024-03-11T19:47:55.917 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.874+0000 7f696fe00700 1 -- 172.21.15.97:0/3619091367 learned_addr learned my addr 172.21.15.97:0/3619091367 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.875+0000 7f696fe00700 1 -- 172.21.15.97:0/3619091367 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6960009550 con 0x7f697007cea0 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.875+0000 7f696fe00700 1 --2- 172.21.15.97:0/3619091367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 0x7f69701a17b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6960011740 tx=0x7f6960011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.875+0000 7f696cc00700 1 -- 172.21.15.97:0/3619091367 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6960003b20 con 0x7f697007cea0 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.875+0000 7f696cc00700 1 -- 172.21.15.97:0/3619091367 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f696001a910 con 0x7f697007cea0 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.875+0000 7f6976a00700 1 -- 172.21.15.97:0/3619091367 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69701a1ef0 con 0x7f697007cea0 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.875+0000 7f696cc00700 1 -- 172.21.15.97:0/3619091367 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69600248c0 con 0x7f697007cea0 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.875+0000 7f6976a00700 1 -- 172.21.15.97:0/3619091367 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69701a2390 con 0x7f697007cea0 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.876+0000 7f6976a00700 1 -- 172.21.15.97:0/3619091367 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697019b720 con 0x7f697007cea0 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.876+0000 7f696cc00700 1 -- 172.21.15.97:0/3619091367 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f696001aa80 con 0x7f697007cea0 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.877+0000 7f696cc00700 1 --2- 172.21.15.97:0/3619091367 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6954038da0 0x7f695403b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.877+0000 7f696cc00700 1 -- 172.21.15.97:0/3619091367 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f696004d3e0 con 0x7f697007cea0 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.877+0000 7f696f400700 1 --2- 172.21.15.97:0/3619091367 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6954038da0 0x7f695403b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.878+0000 7f696f400700 1 --2- 172.21.15.97:0/3619091367 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6954038da0 0x7f695403b260 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f695c006fd0 tx=0x7f695c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:55.918 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:54.882+0000 7f696cc00700 1 -- 172.21.15.97:0/3619091367 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f696001adf0 con 0x7f697007cea0 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:55.212+0000 7f6976a00700 1 -- 172.21.15.97:0/3619091367 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f69701095a0 con 0x7f697007cea0 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:55.216+0000 7f696cc00700 1 -- 172.21.15.97:0/3619091367 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f6960018be0 con 0x7f697007cea0 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:55.221+0000 7f6976a00700 1 -- 172.21.15.97:0/3619091367 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6954038da0 msgr2=0x7f695403b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:55.221+0000 7f6976a00700 1 --2- 172.21.15.97:0/3619091367 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6954038da0 0x7f695403b260 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f695c006fd0 tx=0x7f695c006e40 comp rx=0 tx=0).stop 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:55.221+0000 7f6976a00700 1 -- 172.21.15.97:0/3619091367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 msgr2=0x7f69701a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:55.221+0000 7f6976a00700 1 --2- 172.21.15.97:0/3619091367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 0x7f69701a17b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6960011740 tx=0x7f6960011b00 comp rx=0 tx=0).stop 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:55.221+0000 7f6976a00700 1 -- 172.21.15.97:0/3619091367 shutdown_connections 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:55.221+0000 7f6976a00700 1 --2- 172.21.15.97:0/3619091367 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6954038da0 0x7f695403b260 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:55.221+0000 7f6976a00700 1 --2- 172.21.15.97:0/3619091367 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f697007cea0 0x7f69701a17b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:55.221+0000 7f6976a00700 1 -- 172.21.15.97:0/3619091367 >> 172.21.15.97:0/3619091367 conn(0x7f6970102bd0 msgr2=0x7f6970198860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:55.221+0000 7f6976a00700 1 -- 172.21.15.97:0/3619091367 shutdown_connections 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr 2024-03-11T19:47:55.221+0000 7f6976a00700 1 -- 172.21.15.97:0/3619091367 wait complete. 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:47:55.919 INFO:teuthology.orchestra.run.smithi097.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout: ceph telemetry on 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout:For more information see: 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:47:55.920 INFO:teuthology.orchestra.run.smithi097.stdout:Bootstrap complete. 2024-03-11T19:47:56.021 INFO:tasks.cephadm:Fetching config... 2024-03-11T19:47:56.022 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:47:56.022 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T19:47:56.068 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T19:47:56.068 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:47:56.068 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T19:47:56.150 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T19:47:56.150 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:47:56.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T19:47:56.275 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T19:47:56.275 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:47:56.275 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T19:47:56.370 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T19:47:56.370 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDunaBTS23ZpNBCk9u6s5TwTfwvmYdlHlINjgLXa/dM4yKbv1/qlv/AZ3waIglSfww/e6nN/aoPEM6FZrN0Qk+YGnIs4qdo73JYkknG//UZaO1lnJdIGaFSRyVdPkbj3pm8K8h1btdUWqTCJI5utQpRrjDprRq6oaS6/aVCquvvXOC8O4/aqc0Lkzj+hmA0MRMX5aJuOKjPnUBPjiFk7hZHQKp+WIg+SLW67SB7h1oiiJrL4IYS0EPM7iITiHM9BBDnq4IP9K0bIDX50Zmh9MVceR92BfMK/waZTEA9ks39LwktWhY0e3mMujW9lypb7rkrZOT69NoL/oTwT6Dk0LOUBJZ7yVWX/C+IgcwWAGv7uslecqJPC899K1U0xnbPvYZsK1LdS+W365Uhv/bK3qWuVGgacpIUZ+pB6p9cBkIQfuZWEe5Wi+D7oL5nQi85ogxgmjLYUaQUr7USyoWEvAXgaX62aGOle9NCSzd/+ftkP/8PIpMg7+8OeTBEpXfdGxs= ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T19:47:56.446 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:47:56 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3619091367' entity='client.admin' 2024-03-11T19:47:56.482 INFO:teuthology.orchestra.run.smithi097.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDunaBTS23ZpNBCk9u6s5TwTfwvmYdlHlINjgLXa/dM4yKbv1/qlv/AZ3waIglSfww/e6nN/aoPEM6FZrN0Qk+YGnIs4qdo73JYkknG//UZaO1lnJdIGaFSRyVdPkbj3pm8K8h1btdUWqTCJI5utQpRrjDprRq6oaS6/aVCquvvXOC8O4/aqc0Lkzj+hmA0MRMX5aJuOKjPnUBPjiFk7hZHQKp+WIg+SLW67SB7h1oiiJrL4IYS0EPM7iITiHM9BBDnq4IP9K0bIDX50Zmh9MVceR92BfMK/waZTEA9ks39LwktWhY0e3mMujW9lypb7rkrZOT69NoL/oTwT6Dk0LOUBJZ7yVWX/C+IgcwWAGv7uslecqJPC899K1U0xnbPvYZsK1LdS+W365Uhv/bK3qWuVGgacpIUZ+pB6p9cBkIQfuZWEe5Wi+D7oL5nQi85ogxgmjLYUaQUr7USyoWEvAXgaX62aGOle9NCSzd/+ftkP/8PIpMg7+8OeTBEpXfdGxs= ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:47:56.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDunaBTS23ZpNBCk9u6s5TwTfwvmYdlHlINjgLXa/dM4yKbv1/qlv/AZ3waIglSfww/e6nN/aoPEM6FZrN0Qk+YGnIs4qdo73JYkknG//UZaO1lnJdIGaFSRyVdPkbj3pm8K8h1btdUWqTCJI5utQpRrjDprRq6oaS6/aVCquvvXOC8O4/aqc0Lkzj+hmA0MRMX5aJuOKjPnUBPjiFk7hZHQKp+WIg+SLW67SB7h1oiiJrL4IYS0EPM7iITiHM9BBDnq4IP9K0bIDX50Zmh9MVceR92BfMK/waZTEA9ks39LwktWhY0e3mMujW9lypb7rkrZOT69NoL/oTwT6Dk0LOUBJZ7yVWX/C+IgcwWAGv7uslecqJPC899K1U0xnbPvYZsK1LdS+W365Uhv/bK3qWuVGgacpIUZ+pB6p9cBkIQfuZWEe5Wi+D7oL5nQi85ogxgmjLYUaQUr7USyoWEvAXgaX62aGOle9NCSzd/+ftkP/8PIpMg7+8OeTBEpXfdGxs= ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T19:47:56.591 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDunaBTS23ZpNBCk9u6s5TwTfwvmYdlHlINjgLXa/dM4yKbv1/qlv/AZ3waIglSfww/e6nN/aoPEM6FZrN0Qk+YGnIs4qdo73JYkknG//UZaO1lnJdIGaFSRyVdPkbj3pm8K8h1btdUWqTCJI5utQpRrjDprRq6oaS6/aVCquvvXOC8O4/aqc0Lkzj+hmA0MRMX5aJuOKjPnUBPjiFk7hZHQKp+WIg+SLW67SB7h1oiiJrL4IYS0EPM7iITiHM9BBDnq4IP9K0bIDX50Zmh9MVceR92BfMK/waZTEA9ks39LwktWhY0e3mMujW9lypb7rkrZOT69NoL/oTwT6Dk0LOUBJZ7yVWX/C+IgcwWAGv7uslecqJPC899K1U0xnbPvYZsK1LdS+W365Uhv/bK3qWuVGgacpIUZ+pB6p9cBkIQfuZWEe5Wi+D7oL5nQi85ogxgmjLYUaQUr7USyoWEvAXgaX62aGOle9NCSzd/+ftkP/8PIpMg7+8OeTBEpXfdGxs= ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:47:56.606 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDunaBTS23ZpNBCk9u6s5TwTfwvmYdlHlINjgLXa/dM4yKbv1/qlv/AZ3waIglSfww/e6nN/aoPEM6FZrN0Qk+YGnIs4qdo73JYkknG//UZaO1lnJdIGaFSRyVdPkbj3pm8K8h1btdUWqTCJI5utQpRrjDprRq6oaS6/aVCquvvXOC8O4/aqc0Lkzj+hmA0MRMX5aJuOKjPnUBPjiFk7hZHQKp+WIg+SLW67SB7h1oiiJrL4IYS0EPM7iITiHM9BBDnq4IP9K0bIDX50Zmh9MVceR92BfMK/waZTEA9ks39LwktWhY0e3mMujW9lypb7rkrZOT69NoL/oTwT6Dk0LOUBJZ7yVWX/C+IgcwWAGv7uslecqJPC899K1U0xnbPvYZsK1LdS+W365Uhv/bK3qWuVGgacpIUZ+pB6p9cBkIQfuZWEe5Wi+D7oL5nQi85ogxgmjLYUaQUr7USyoWEvAXgaX62aGOle9NCSzd/+ftkP/8PIpMg7+8OeTBEpXfdGxs= ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T19:47:56.652 INFO:teuthology.orchestra.run.smithi161.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDunaBTS23ZpNBCk9u6s5TwTfwvmYdlHlINjgLXa/dM4yKbv1/qlv/AZ3waIglSfww/e6nN/aoPEM6FZrN0Qk+YGnIs4qdo73JYkknG//UZaO1lnJdIGaFSRyVdPkbj3pm8K8h1btdUWqTCJI5utQpRrjDprRq6oaS6/aVCquvvXOC8O4/aqc0Lkzj+hmA0MRMX5aJuOKjPnUBPjiFk7hZHQKp+WIg+SLW67SB7h1oiiJrL4IYS0EPM7iITiHM9BBDnq4IP9K0bIDX50Zmh9MVceR92BfMK/waZTEA9ks39LwktWhY0e3mMujW9lypb7rkrZOT69NoL/oTwT6Dk0LOUBJZ7yVWX/C+IgcwWAGv7uslecqJPC899K1U0xnbPvYZsK1LdS+W365Uhv/bK3qWuVGgacpIUZ+pB6p9cBkIQfuZWEe5Wi+D7oL5nQi85ogxgmjLYUaQUr7USyoWEvAXgaX62aGOle9NCSzd/+ftkP/8PIpMg7+8OeTBEpXfdGxs= ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:47:56.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T19:47:57.077 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:47:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.890+0000 7f6382e00700 1 -- 172.21.15.97:0/1689629620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f637c1005a0 msgr2=0x7f637c1009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:47:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.890+0000 7f6382e00700 1 --2- 172.21.15.97:0/1689629620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f637c1005a0 0x7f637c1009c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f636c009910 tx=0x7f636c009c20 comp rx=0 tx=0).stop 2024-03-11T19:47:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.890+0000 7f6382e00700 1 -- 172.21.15.97:0/1689629620 shutdown_connections 2024-03-11T19:47:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.890+0000 7f6382e00700 1 --2- 172.21.15.97:0/1689629620 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f637c1005a0 0x7f637c1009c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:47:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.890+0000 7f6382e00700 1 -- 172.21.15.97:0/1689629620 >> 172.21.15.97:0/1689629620 conn(0x7f637c0fbb40 msgr2=0x7f637c0fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.891+0000 7f6382e00700 1 -- 172.21.15.97:0/1689629620 shutdown_connections 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.891+0000 7f6382e00700 1 -- 172.21.15.97:0/1689629620 wait complete. 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.892+0000 7f6382e00700 1 Processor -- start 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.892+0000 7f6382e00700 1 -- start start 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.892+0000 7f6382e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f637c1005a0 0x7f637c198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.892+0000 7f6382e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c199310 con 0x7f637c1005a0 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.893+0000 7f637be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f637c1005a0 0x7f637c198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.893+0000 7f637be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f637c1005a0 0x7f637c198dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58958/0 (socket says 172.21.15.97:58958) 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.893+0000 7f637be00700 1 -- 172.21.15.97:0/211984584 learned_addr learned my addr 172.21.15.97:0/211984584 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.893+0000 7f637be00700 1 -- 172.21.15.97:0/211984584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c0095f0 con 0x7f637c1005a0 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.893+0000 7f637be00700 1 --2- 172.21.15.97:0/211984584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f637c1005a0 0x7f637c198dd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f636c004f40 tx=0x7f636c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.894+0000 7f6378c00700 1 -- 172.21.15.97:0/211984584 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f636c01d070 con 0x7f637c1005a0 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.894+0000 7f6378c00700 1 -- 172.21.15.97:0/211984584 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f636c0044a0 con 0x7f637c1005a0 2024-03-11T19:47:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.894+0000 7f6378c00700 1 -- 172.21.15.97:0/211984584 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f636c003a40 con 0x7f637c1005a0 2024-03-11T19:47:59.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.894+0000 7f6382e00700 1 -- 172.21.15.97:0/211984584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637c199510 con 0x7f637c1005a0 2024-03-11T19:47:59.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.894+0000 7f6382e00700 1 -- 172.21.15.97:0/211984584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637c1999b0 con 0x7f637c1005a0 2024-03-11T19:47:59.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.895+0000 7f6378c00700 1 -- 172.21.15.97:0/211984584 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f636c003cb0 con 0x7f637c1005a0 2024-03-11T19:47:59.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.895+0000 7f6382e00700 1 -- 172.21.15.97:0/211984584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f637c0632a0 con 0x7f637c1005a0 2024-03-11T19:47:59.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.896+0000 7f6378c00700 1 --2- 172.21.15.97:0/211984584 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6360038db0 0x7f636003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:47:59.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.896+0000 7f6378c00700 1 -- 172.21.15.97:0/211984584 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f636c04cb20 con 0x7f637c1005a0 2024-03-11T19:47:59.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.896+0000 7f637b400700 1 --2- 172.21.15.97:0/211984584 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6360038db0 0x7f636003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:47:59.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.897+0000 7f637b400700 1 --2- 172.21.15.97:0/211984584 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6360038db0 0x7f636003b270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6368006fd0 tx=0x7f6368006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:47:59.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:47:59.901+0000 7f6378c00700 1 -- 172.21.15.97:0/211984584 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f636c029950 con 0x7f637c1005a0 2024-03-11T19:48:00.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:00.123+0000 7f6382e00700 1 -- 172.21.15.97:0/211984584 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f637c106430 con 0x7f637c1005a0 2024-03-11T19:48:00.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:00.128+0000 7f6378c00700 1 -- 172.21.15.97:0/211984584 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f636c026070 con 0x7f637c1005a0 2024-03-11T19:48:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:00.131+0000 7f6382e00700 1 -- 172.21.15.97:0/211984584 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6360038db0 msgr2=0x7f636003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:00.131+0000 7f6382e00700 1 --2- 172.21.15.97:0/211984584 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6360038db0 0x7f636003b270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6368006fd0 tx=0x7f6368006e40 comp rx=0 tx=0).stop 2024-03-11T19:48:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:00.131+0000 7f6382e00700 1 -- 172.21.15.97:0/211984584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f637c1005a0 msgr2=0x7f637c198dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:00.131+0000 7f6382e00700 1 --2- 172.21.15.97:0/211984584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f637c1005a0 0x7f637c198dd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f636c004f40 tx=0x7f636c005020 comp rx=0 tx=0).stop 2024-03-11T19:48:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:00.131+0000 7f6382e00700 1 -- 172.21.15.97:0/211984584 shutdown_connections 2024-03-11T19:48:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:00.131+0000 7f6382e00700 1 --2- 172.21.15.97:0/211984584 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6360038db0 0x7f636003b270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:00.131+0000 7f6382e00700 1 --2- 172.21.15.97:0/211984584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f637c1005a0 0x7f637c198dd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:00.131+0000 7f6382e00700 1 -- 172.21.15.97:0/211984584 >> 172.21.15.97:0/211984584 conn(0x7f637c0fbb40 msgr2=0x7f637c069060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:00.132+0000 7f6382e00700 1 -- 172.21.15.97:0/211984584 shutdown_connections 2024-03-11T19:48:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:00.132+0000 7f6382e00700 1 -- 172.21.15.97:0/211984584 wait complete. 2024-03-11T19:48:00.883 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T19:48:00.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T19:48:01.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:01 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/211984584' entity='client.admin' 2024-03-11T19:48:01.971 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:48:03.297 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:03.297 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:03.297 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:48:03.297 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:03.297 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:48:03.297 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:04.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.123+0000 7ff5b6000700 1 -- 172.21.15.97:0/1900055172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5b01016a0 msgr2=0x7ff5b0101ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:04.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.123+0000 7ff5b6000700 1 --2- 172.21.15.97:0/1900055172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5b01016a0 0x7ff5b0101ac0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0009910 tx=0x7ff5a0009c20 comp rx=0 tx=0).stop 2024-03-11T19:48:04.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.123+0000 7ff5b6000700 1 -- 172.21.15.97:0/1900055172 shutdown_connections 2024-03-11T19:48:04.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.123+0000 7ff5b6000700 1 --2- 172.21.15.97:0/1900055172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5b01016a0 0x7ff5b0101ac0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:04.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.123+0000 7ff5b6000700 1 -- 172.21.15.97:0/1900055172 >> 172.21.15.97:0/1900055172 conn(0x7ff5b00fcc40 msgr2=0x7ff5b00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:04.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.123+0000 7ff5b6000700 1 -- 172.21.15.97:0/1900055172 shutdown_connections 2024-03-11T19:48:04.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.123+0000 7ff5b6000700 1 -- 172.21.15.97:0/1900055172 wait complete. 2024-03-11T19:48:04.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.124+0000 7ff5b6000700 1 Processor -- start 2024-03-11T19:48:04.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.124+0000 7ff5b6000700 1 -- start start 2024-03-11T19:48:04.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.125+0000 7ff5b6000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5b01016a0 0x7ff5b0198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:04.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.125+0000 7ff5b6000700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b0199340 con 0x7ff5b01016a0 2024-03-11T19:48:04.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.125+0000 7ff5af400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5b01016a0 0x7ff5b0198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:04.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.125+0000 7ff5af400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5b01016a0 0x7ff5b0198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58178/0 (socket says 172.21.15.97:58178) 2024-03-11T19:48:04.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.125+0000 7ff5af400700 1 -- 172.21.15.97:0/1967500119 learned_addr learned my addr 172.21.15.97:0/1967500119 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:48:04.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.125+0000 7ff5af400700 1 -- 172.21.15.97:0/1967500119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a00095f0 con 0x7ff5b01016a0 2024-03-11T19:48:04.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.126+0000 7ff5af400700 1 --2- 172.21.15.97:0/1967500119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5b01016a0 0x7ff5b0198e00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0004f40 tx=0x7ff5a0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:04.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.126+0000 7ff5a7e00700 1 -- 172.21.15.97:0/1967500119 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5a001d070 con 0x7ff5b01016a0 2024-03-11T19:48:04.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.126+0000 7ff5a7e00700 1 -- 172.21.15.97:0/1967500119 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5a00044a0 con 0x7ff5b01016a0 2024-03-11T19:48:04.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.126+0000 7ff5b6000700 1 -- 172.21.15.97:0/1967500119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b0199540 con 0x7ff5b01016a0 2024-03-11T19:48:04.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.126+0000 7ff5a7e00700 1 -- 172.21.15.97:0/1967500119 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5a0003a40 con 0x7ff5b01016a0 2024-03-11T19:48:04.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.126+0000 7ff5b6000700 1 -- 172.21.15.97:0/1967500119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b01999e0 con 0x7ff5b01016a0 2024-03-11T19:48:04.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.127+0000 7ff5a7e00700 1 -- 172.21.15.97:0/1967500119 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7ff5a0003cb0 con 0x7ff5b01016a0 2024-03-11T19:48:04.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.127+0000 7ff5b6000700 1 -- 172.21.15.97:0/1967500119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b0192900 con 0x7ff5b01016a0 2024-03-11T19:48:04.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.128+0000 7ff5a7e00700 1 --2- 172.21.15.97:0/1967500119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff594038db0 0x7ff59403b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:04.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.128+0000 7ff5a7e00700 1 -- 172.21.15.97:0/1967500119 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff5a004cc30 con 0x7ff5b01016a0 2024-03-11T19:48:04.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.128+0000 7ff5aea00700 1 --2- 172.21.15.97:0/1967500119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff594038db0 0x7ff59403b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:04.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.129+0000 7ff5aea00700 1 --2- 172.21.15.97:0/1967500119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff594038db0 0x7ff59403b270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff598006fd0 tx=0x7ff598006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:04.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.133+0000 7ff5a7e00700 1 -- 172.21.15.97:0/1967500119 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff5a0004610 con 0x7ff5b01016a0 2024-03-11T19:48:04.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.358+0000 7ff5b6000700 1 -- 172.21.15.97:0/1967500119 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5b00620b0 con 0x7ff594038db0 2024-03-11T19:48:04.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.365+0000 7ff5a7e00700 1 -- 172.21.15.97:0/1967500119 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff5b00620b0 con 0x7ff594038db0 2024-03-11T19:48:04.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.368+0000 7ff5b6000700 1 -- 172.21.15.97:0/1967500119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff594038db0 msgr2=0x7ff59403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:04.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.368+0000 7ff5b6000700 1 --2- 172.21.15.97:0/1967500119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff594038db0 0x7ff59403b270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff598006fd0 tx=0x7ff598006e40 comp rx=0 tx=0).stop 2024-03-11T19:48:04.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.368+0000 7ff5b6000700 1 -- 172.21.15.97:0/1967500119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5b01016a0 msgr2=0x7ff5b0198e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:04.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.368+0000 7ff5b6000700 1 --2- 172.21.15.97:0/1967500119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5b01016a0 0x7ff5b0198e00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0004f40 tx=0x7ff5a0005020 comp rx=0 tx=0).stop 2024-03-11T19:48:04.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.368+0000 7ff5b6000700 1 -- 172.21.15.97:0/1967500119 shutdown_connections 2024-03-11T19:48:04.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.368+0000 7ff5b6000700 1 --2- 172.21.15.97:0/1967500119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff594038db0 0x7ff59403b270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:04.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.368+0000 7ff5b6000700 1 --2- 172.21.15.97:0/1967500119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff5b01016a0 0x7ff5b0198e00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:04.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.368+0000 7ff5b6000700 1 -- 172.21.15.97:0/1967500119 >> 172.21.15.97:0/1967500119 conn(0x7ff5b00fcc40 msgr2=0x7ff5b00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:04.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.368+0000 7ff5b6000700 1 -- 172.21.15.97:0/1967500119 shutdown_connections 2024-03-11T19:48:04.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:04.368+0000 7ff5b6000700 1 -- 172.21.15.97:0/1967500119 wait complete. 2024-03-11T19:48:05.145 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi114 2024-03-11T19:48:05.146 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-11T19:48:05.146 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.conf 2024-03-11T19:48:05.164 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-11T19:48:05.165 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:48:05.221 INFO:tasks.cephadm:Adding host smithi114 to orchestrator... 2024-03-11T19:48:05.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch host add smithi114 2024-03-11T19:48:05.656 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:48:05.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:05 smithi097 ceph-mon[31898]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:48:05.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:05.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:05 smithi097 ceph-mon[31898]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:05.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:48:05.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:05.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:48:05.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:05 smithi097 ceph-mon[31898]: Updating smithi097:/etc/ceph/ceph.conf 2024-03-11T19:48:06.649 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:06 smithi097 ceph-mon[31898]: Updating smithi097:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:07.520 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:07 smithi097 ceph-mon[31898]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:48:07.520 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:07 smithi097 ceph-mon[31898]: Updating smithi097:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:48:07.520 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:07 smithi097 ceph-mon[31898]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:07.520 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:07 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:07.520 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:07 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:07.520 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:07 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:07.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.684+0000 7f9a2ca00700 1 -- 172.21.15.97:0/3887947907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a281064e0 msgr2=0x7f9a281068c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:07.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.684+0000 7f9a2ca00700 1 --2- 172.21.15.97:0/3887947907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a281064e0 0x7f9a281068c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9a18009910 tx=0x7f9a18009c20 comp rx=0 tx=0).stop 2024-03-11T19:48:07.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.684+0000 7f9a2ca00700 1 -- 172.21.15.97:0/3887947907 shutdown_connections 2024-03-11T19:48:07.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.684+0000 7f9a2ca00700 1 --2- 172.21.15.97:0/3887947907 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a281064e0 0x7f9a281068c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:07.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.684+0000 7f9a2ca00700 1 -- 172.21.15.97:0/3887947907 >> 172.21.15.97:0/3887947907 conn(0x7f9a280fb1c0 msgr2=0x7f9a280fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:07.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.684+0000 7f9a2ca00700 1 -- 172.21.15.97:0/3887947907 shutdown_connections 2024-03-11T19:48:07.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.684+0000 7f9a2ca00700 1 -- 172.21.15.97:0/3887947907 wait complete. 2024-03-11T19:48:07.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.685+0000 7f9a2ca00700 1 Processor -- start 2024-03-11T19:48:07.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.686+0000 7f9a2ca00700 1 -- start start 2024-03-11T19:48:07.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.686+0000 7f9a2ca00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a281064e0 0x7f9a28198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:07.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.686+0000 7f9a2ca00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a28199440 con 0x7f9a281064e0 2024-03-11T19:48:07.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.686+0000 7f9a26800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a281064e0 0x7f9a28198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:07.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.686+0000 7f9a26800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a281064e0 0x7f9a28198f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58192/0 (socket says 172.21.15.97:58192) 2024-03-11T19:48:07.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.686+0000 7f9a26800700 1 -- 172.21.15.97:0/1410902839 learned_addr learned my addr 172.21.15.97:0/1410902839 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:48:07.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.687+0000 7f9a26800700 1 -- 172.21.15.97:0/1410902839 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a180095f0 con 0x7f9a281064e0 2024-03-11T19:48:07.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.687+0000 7f9a26800700 1 --2- 172.21.15.97:0/1410902839 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a281064e0 0x7f9a28198f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9a18006010 tx=0x7f9a18004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:07.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.687+0000 7f9a1f400700 1 -- 172.21.15.97:0/1410902839 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9a1801d070 con 0x7f9a281064e0 2024-03-11T19:48:07.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.687+0000 7f9a1f400700 1 -- 172.21.15.97:0/1410902839 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a1800bc30 con 0x7f9a281064e0 2024-03-11T19:48:07.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.687+0000 7f9a2ca00700 1 -- 172.21.15.97:0/1410902839 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a281996a0 con 0x7f9a281064e0 2024-03-11T19:48:07.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.687+0000 7f9a1f400700 1 -- 172.21.15.97:0/1410902839 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9a18003930 con 0x7f9a281064e0 2024-03-11T19:48:07.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.687+0000 7f9a2ca00700 1 -- 172.21.15.97:0/1410902839 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a28199ba0 con 0x7f9a281064e0 2024-03-11T19:48:07.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.688+0000 7f9a2ca00700 1 -- 172.21.15.97:0/1410902839 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a280632a0 con 0x7f9a281064e0 2024-03-11T19:48:07.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.688+0000 7f9a1f400700 1 -- 172.21.15.97:0/1410902839 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f9a18003ae0 con 0x7f9a281064e0 2024-03-11T19:48:07.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.689+0000 7f9a1f400700 1 --2- 172.21.15.97:0/1410902839 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9a0c038db0 0x7f9a0c03b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:07.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.689+0000 7f9a1f400700 1 -- 172.21.15.97:0/1410902839 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9a1804c780 con 0x7f9a281064e0 2024-03-11T19:48:07.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.689+0000 7f9a25e00700 1 --2- 172.21.15.97:0/1410902839 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9a0c038db0 0x7f9a0c03b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:07.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.690+0000 7f9a25e00700 1 --2- 172.21.15.97:0/1410902839 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9a0c038db0 0x7f9a0c03b270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9a14006fd0 tx=0x7f9a14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:07.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.694+0000 7f9a1f400700 1 -- 172.21.15.97:0/1410902839 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9a1802ac00 con 0x7f9a281064e0 2024-03-11T19:48:07.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:07.920+0000 7f9a2ca00700 1 -- 172.21.15.97:0/1410902839 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi114", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a28106fd0 con 0x7f9a0c038db0 2024-03-11T19:48:09.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:09 smithi097 ceph-mon[31898]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:48:09.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:09 smithi097 ceph-mon[31898]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:10.823 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:10 smithi097 ceph-mon[31898]: Deploying cephadm binary to smithi114 2024-03-11T19:48:11.856 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:11 smithi097 ceph-mon[31898]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:12.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:12.634+0000 7f9a1f400700 1 -- 172.21.15.97:0/1410902839 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9a28106fd0 con 0x7f9a0c038db0 2024-03-11T19:48:12.640 INFO:teuthology.orchestra.run.smithi097.stdout:Added host 'smithi114' with addr '172.21.15.114' 2024-03-11T19:48:12.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:12.638+0000 7f9a2ca00700 1 -- 172.21.15.97:0/1410902839 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9a0c038db0 msgr2=0x7f9a0c03b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:12.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:12.638+0000 7f9a2ca00700 1 --2- 172.21.15.97:0/1410902839 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9a0c038db0 0x7f9a0c03b270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9a14006fd0 tx=0x7f9a14006e40 comp rx=0 tx=0).stop 2024-03-11T19:48:12.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:12.638+0000 7f9a2ca00700 1 -- 172.21.15.97:0/1410902839 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a281064e0 msgr2=0x7f9a28198f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:12.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:12.638+0000 7f9a2ca00700 1 --2- 172.21.15.97:0/1410902839 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a281064e0 0x7f9a28198f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9a18006010 tx=0x7f9a18004ff0 comp rx=0 tx=0).stop 2024-03-11T19:48:12.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:12.638+0000 7f9a2ca00700 1 -- 172.21.15.97:0/1410902839 shutdown_connections 2024-03-11T19:48:12.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:12.638+0000 7f9a2ca00700 1 --2- 172.21.15.97:0/1410902839 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9a0c038db0 0x7f9a0c03b270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:12.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:12.638+0000 7f9a2ca00700 1 --2- 172.21.15.97:0/1410902839 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9a281064e0 0x7f9a28198f00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:12.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:12.638+0000 7f9a2ca00700 1 -- 172.21.15.97:0/1410902839 >> 172.21.15.97:0/1410902839 conn(0x7f9a280fb1c0 msgr2=0x7f9a28108fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:12.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:12.639+0000 7f9a2ca00700 1 -- 172.21.15.97:0/1410902839 shutdown_connections 2024-03-11T19:48:12.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:12.639+0000 7f9a2ca00700 1 -- 172.21.15.97:0/1410902839 wait complete. 2024-03-11T19:48:13.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T19:48:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:48:13.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:13 smithi097 ceph-mon[31898]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:13.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:13 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:13.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:13 smithi097 ceph-mon[31898]: Added host smithi114 2024-03-11T19:48:13.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:13 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:48:13.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:13 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:14.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:14 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:14.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:14 smithi097 ceph-mon[31898]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:15.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.698+0000 7f2f2bc00700 1 -- 172.21.15.97:0/105965700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f24107640 msgr2=0x7f2f24107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:15.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.698+0000 7f2f2bc00700 1 --2- 172.21.15.97:0/105965700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f24107640 0x7f2f24107a20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2f18009910 tx=0x7f2f18009c20 comp rx=0 tx=0).stop 2024-03-11T19:48:15.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.698+0000 7f2f2bc00700 1 -- 172.21.15.97:0/105965700 shutdown_connections 2024-03-11T19:48:15.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.698+0000 7f2f2bc00700 1 --2- 172.21.15.97:0/105965700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f24107640 0x7f2f24107a20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:15.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.698+0000 7f2f2bc00700 1 -- 172.21.15.97:0/105965700 >> 172.21.15.97:0/105965700 conn(0x7f2f240fd150 msgr2=0x7f2f240ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:15.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.698+0000 7f2f2bc00700 1 -- 172.21.15.97:0/105965700 shutdown_connections 2024-03-11T19:48:15.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.698+0000 7f2f2bc00700 1 -- 172.21.15.97:0/105965700 wait complete. 2024-03-11T19:48:15.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.699+0000 7f2f2bc00700 1 Processor -- start 2024-03-11T19:48:15.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.699+0000 7f2f2bc00700 1 -- start start 2024-03-11T19:48:15.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.700+0000 7f2f2bc00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f24107640 0x7f2f24199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:15.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.700+0000 7f2f2bc00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f241996c0 con 0x7f2f24107640 2024-03-11T19:48:15.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.700+0000 7f2f29200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f24107640 0x7f2f24199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:15.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.700+0000 7f2f29200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f24107640 0x7f2f24199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46782/0 (socket says 172.21.15.97:46782) 2024-03-11T19:48:15.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.700+0000 7f2f29200700 1 -- 172.21.15.97:0/669258392 learned_addr learned my addr 172.21.15.97:0/669258392 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:48:15.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.700+0000 7f2f29200700 1 -- 172.21.15.97:0/669258392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f180095f0 con 0x7f2f24107640 2024-03-11T19:48:15.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.700+0000 7f2f29200700 1 --2- 172.21.15.97:0/669258392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f24107640 0x7f2f24199180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2f18006010 tx=0x7f2f18004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:15.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.701+0000 7f2f21600700 1 -- 172.21.15.97:0/669258392 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f1801d070 con 0x7f2f24107640 2024-03-11T19:48:15.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.701+0000 7f2f21600700 1 -- 172.21.15.97:0/669258392 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f1800bbd0 con 0x7f2f24107640 2024-03-11T19:48:15.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.701+0000 7f2f2bc00700 1 -- 172.21.15.97:0/669258392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f24199920 con 0x7f2f24107640 2024-03-11T19:48:15.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.701+0000 7f2f21600700 1 -- 172.21.15.97:0/669258392 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f18003880 con 0x7f2f24107640 2024-03-11T19:48:15.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.701+0000 7f2f2bc00700 1 -- 172.21.15.97:0/669258392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f24199da0 con 0x7f2f24107640 2024-03-11T19:48:15.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.702+0000 7f2f21600700 1 -- 172.21.15.97:0/669258392 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f2f1800bd40 con 0x7f2f24107640 2024-03-11T19:48:15.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.702+0000 7f2f21600700 1 --2- 172.21.15.97:0/669258392 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2f0c038db0 0x7f2f0c03b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:15.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.702+0000 7f2f2bc00700 1 -- 172.21.15.97:0/669258392 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f240632a0 con 0x7f2f24107640 2024-03-11T19:48:15.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.702+0000 7f2f21600700 1 -- 172.21.15.97:0/669258392 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2f18051a10 con 0x7f2f24107640 2024-03-11T19:48:15.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.703+0000 7f2f23e00700 1 --2- 172.21.15.97:0/669258392 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2f0c038db0 0x7f2f0c03b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:15.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.703+0000 7f2f23e00700 1 --2- 172.21.15.97:0/669258392 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2f0c038db0 0x7f2f0c03b270 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2f10006fd0 tx=0x7f2f10006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:15.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.708+0000 7f2f21600700 1 -- 172.21.15.97:0/669258392 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2f18026020 con 0x7f2f24107640 2024-03-11T19:48:15.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.931+0000 7f2f2bc00700 1 -- 172.21.15.97:0/669258392 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f2410bc30 con 0x7f2f0c038db0 2024-03-11T19:48:15.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.933+0000 7f2f21600700 1 -- 172.21.15.97:0/669258392 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f2f2410bc30 con 0x7f2f0c038db0 2024-03-11T19:48:15.938 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:48:15.938 INFO:teuthology.orchestra.run.smithi097.stdout:[{"addr": "172.21.15.97", "hostname": "smithi097", "labels": [], "status": ""}, {"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}] 2024-03-11T19:48:15.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.936+0000 7f2f2bc00700 1 -- 172.21.15.97:0/669258392 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2f0c038db0 msgr2=0x7f2f0c03b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:15.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.936+0000 7f2f2bc00700 1 --2- 172.21.15.97:0/669258392 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2f0c038db0 0x7f2f0c03b270 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2f10006fd0 tx=0x7f2f10006e40 comp rx=0 tx=0).stop 2024-03-11T19:48:15.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.936+0000 7f2f2bc00700 1 -- 172.21.15.97:0/669258392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f24107640 msgr2=0x7f2f24199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:15.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.936+0000 7f2f2bc00700 1 --2- 172.21.15.97:0/669258392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f24107640 0x7f2f24199180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2f18006010 tx=0x7f2f18004ff0 comp rx=0 tx=0).stop 2024-03-11T19:48:15.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.937+0000 7f2f2bc00700 1 -- 172.21.15.97:0/669258392 shutdown_connections 2024-03-11T19:48:15.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.937+0000 7f2f2bc00700 1 --2- 172.21.15.97:0/669258392 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2f0c038db0 0x7f2f0c03b270 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:15.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.937+0000 7f2f2bc00700 1 --2- 172.21.15.97:0/669258392 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f24107640 0x7f2f24199180 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:15.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.937+0000 7f2f2bc00700 1 -- 172.21.15.97:0/669258392 >> 172.21.15.97:0/669258392 conn(0x7f2f240fd150 msgr2=0x7f2f24075980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:15.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.937+0000 7f2f2bc00700 1 -- 172.21.15.97:0/669258392 shutdown_connections 2024-03-11T19:48:15.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:15.937+0000 7f2f2bc00700 1 -- 172.21.15.97:0/669258392 wait complete. 2024-03-11T19:48:16.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:16 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:16.699 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi161 2024-03-11T19:48:16.699 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-11T19:48:16.699 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/etc/ceph/ceph.conf 2024-03-11T19:48:16.718 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-11T19:48:16.718 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:48:16.775 INFO:tasks.cephadm:Adding host smithi161 to orchestrator... 2024-03-11T19:48:16.776 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch host add smithi161 2024-03-11T19:48:17.209 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:48:17.659 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:17 smithi097 ceph-mon[31898]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:48:17.659 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:17 smithi097 ceph-mon[31898]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:19.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.360+0000 7f0b23c00700 1 -- 172.21.15.97:0/2603869989 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b1c107640 msgr2=0x7f0b1c107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:19.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.361+0000 7f0b23c00700 1 --2- 172.21.15.97:0/2603869989 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b1c107640 0x7f0b1c107a20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0b10009910 tx=0x7f0b10009c20 comp rx=0 tx=0).stop 2024-03-11T19:48:19.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.361+0000 7f0b23c00700 1 -- 172.21.15.97:0/2603869989 shutdown_connections 2024-03-11T19:48:19.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.361+0000 7f0b23c00700 1 --2- 172.21.15.97:0/2603869989 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b1c107640 0x7f0b1c107a20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:19.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.361+0000 7f0b23c00700 1 -- 172.21.15.97:0/2603869989 >> 172.21.15.97:0/2603869989 conn(0x7f0b1c0fd150 msgr2=0x7f0b1c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:19.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.361+0000 7f0b23c00700 1 -- 172.21.15.97:0/2603869989 shutdown_connections 2024-03-11T19:48:19.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.361+0000 7f0b23c00700 1 -- 172.21.15.97:0/2603869989 wait complete. 2024-03-11T19:48:19.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.362+0000 7f0b23c00700 1 Processor -- start 2024-03-11T19:48:19.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.362+0000 7f0b23c00700 1 -- start start 2024-03-11T19:48:19.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.362+0000 7f0b23c00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b1c107640 0x7f0b1c073f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:19.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.362+0000 7f0b23c00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c075e60 con 0x7f0b1c107640 2024-03-11T19:48:19.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.363+0000 7f0b21200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b1c107640 0x7f0b1c073f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:19.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.363+0000 7f0b21200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b1c107640 0x7f0b1c073f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46812/0 (socket says 172.21.15.97:46812) 2024-03-11T19:48:19.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.363+0000 7f0b21200700 1 -- 172.21.15.97:0/2915955440 learned_addr learned my addr 172.21.15.97:0/2915955440 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:48:19.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.363+0000 7f0b21200700 1 -- 172.21.15.97:0/2915955440 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b100095f0 con 0x7f0b1c107640 2024-03-11T19:48:19.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.363+0000 7f0b21200700 1 --2- 172.21.15.97:0/2915955440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b1c107640 0x7f0b1c073f70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0b10006010 tx=0x7f0b10004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:19.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.364+0000 7f0b19600700 1 -- 172.21.15.97:0/2915955440 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b1001d070 con 0x7f0b1c107640 2024-03-11T19:48:19.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.364+0000 7f0b19600700 1 -- 172.21.15.97:0/2915955440 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b1000bbd0 con 0x7f0b1c107640 2024-03-11T19:48:19.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.364+0000 7f0b19600700 1 -- 172.21.15.97:0/2915955440 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b10003880 con 0x7f0b1c107640 2024-03-11T19:48:19.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.364+0000 7f0b23c00700 1 -- 172.21.15.97:0/2915955440 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1c0744b0 con 0x7f0b1c107640 2024-03-11T19:48:19.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.364+0000 7f0b23c00700 1 -- 172.21.15.97:0/2915955440 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1c074930 con 0x7f0b1c107640 2024-03-11T19:48:19.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.365+0000 7f0b19600700 1 -- 172.21.15.97:0/2915955440 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f0b1000bd40 con 0x7f0b1c107640 2024-03-11T19:48:19.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.365+0000 7f0b23c00700 1 -- 172.21.15.97:0/2915955440 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1c102d80 con 0x7f0b1c107640 2024-03-11T19:48:19.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.365+0000 7f0b19600700 1 --2- 172.21.15.97:0/2915955440 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0b04038d50 0x7f0b0403b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:19.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.365+0000 7f0b19600700 1 -- 172.21.15.97:0/2915955440 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0b1004ca60 con 0x7f0b1c107640 2024-03-11T19:48:19.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.366+0000 7f0b1be00700 1 --2- 172.21.15.97:0/2915955440 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0b04038d50 0x7f0b0403b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:19.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.366+0000 7f0b1be00700 1 --2- 172.21.15.97:0/2915955440 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0b04038d50 0x7f0b0403b210 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0b08006fd0 tx=0x7f0b08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:19.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.373+0000 7f0b19600700 1 -- 172.21.15.97:0/2915955440 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0b1002ae20 con 0x7f0b1c107640 2024-03-11T19:48:19.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:19.600+0000 7f0b23c00700 1 -- 172.21.15.97:0/2915955440 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi161", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b1c074ca0 con 0x7f0b04038d50 2024-03-11T19:48:19.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:19 smithi097 ceph-mon[31898]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:19.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:19.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:19.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:19.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:19.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:48:19.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:19.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:48:20.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:20 smithi097 ceph-mon[31898]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-11T19:48:20.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:20 smithi097 ceph-mon[31898]: Updating smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:20.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:20 smithi097 ceph-mon[31898]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:48:20.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:20 smithi097 ceph-mon[31898]: Updating smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:48:20.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:20 smithi097 ceph-mon[31898]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi161", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:48:20.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:20 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:20.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:20 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:20.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:20 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:21.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:21 smithi097 ceph-mon[31898]: Deploying cephadm binary to smithi161 2024-03-11T19:48:21.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:21 smithi097 ceph-mon[31898]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:23 smithi097 ceph-mon[31898]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:24.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:24.306+0000 7f0b19600700 1 -- 172.21.15.97:0/2915955440 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f0b1c074ca0 con 0x7f0b04038d50 2024-03-11T19:48:24.311 INFO:teuthology.orchestra.run.smithi097.stdout:Added host 'smithi161' with addr '172.21.15.161' 2024-03-11T19:48:24.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:24.310+0000 7f0b23c00700 1 -- 172.21.15.97:0/2915955440 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0b04038d50 msgr2=0x7f0b0403b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:24.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:24.310+0000 7f0b23c00700 1 --2- 172.21.15.97:0/2915955440 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0b04038d50 0x7f0b0403b210 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0b08006fd0 tx=0x7f0b08006e40 comp rx=0 tx=0).stop 2024-03-11T19:48:24.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:24.310+0000 7f0b23c00700 1 -- 172.21.15.97:0/2915955440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b1c107640 msgr2=0x7f0b1c073f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:24.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:24.310+0000 7f0b23c00700 1 --2- 172.21.15.97:0/2915955440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b1c107640 0x7f0b1c073f70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0b10006010 tx=0x7f0b10004ff0 comp rx=0 tx=0).stop 2024-03-11T19:48:24.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:24.310+0000 7f0b23c00700 1 -- 172.21.15.97:0/2915955440 shutdown_connections 2024-03-11T19:48:24.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:24.310+0000 7f0b23c00700 1 --2- 172.21.15.97:0/2915955440 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0b04038d50 0x7f0b0403b210 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:24.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:24.310+0000 7f0b23c00700 1 --2- 172.21.15.97:0/2915955440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0b1c107640 0x7f0b1c073f70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:24.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:24.310+0000 7f0b23c00700 1 -- 172.21.15.97:0/2915955440 >> 172.21.15.97:0/2915955440 conn(0x7f0b1c0fd150 msgr2=0x7f0b1c0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:24.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:24.311+0000 7f0b23c00700 1 -- 172.21.15.97:0/2915955440 shutdown_connections 2024-03-11T19:48:24.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:24.311+0000 7f0b23c00700 1 -- 172.21.15.97:0/2915955440 wait complete. 2024-03-11T19:48:25.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T19:48:25.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:25 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:25.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:25 smithi097 ceph-mon[31898]: Added host smithi161 2024-03-11T19:48:25.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:25 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:48:25.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:25 smithi097 ceph-mon[31898]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:25.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:25 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:25.612 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:48:26.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:26 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:27.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:27 smithi097 ceph-mon[31898]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:27.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:27 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:27.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.617+0000 7f671c400700 1 -- 172.21.15.97:0/2685841065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6714107640 msgr2=0x7f6714107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:27.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.617+0000 7f671c400700 1 --2- 172.21.15.97:0/2685841065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6714107640 0x7f6714107a20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6708009910 tx=0x7f6708009c20 comp rx=0 tx=0).stop 2024-03-11T19:48:27.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.617+0000 7f671c400700 1 -- 172.21.15.97:0/2685841065 shutdown_connections 2024-03-11T19:48:27.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.617+0000 7f671c400700 1 --2- 172.21.15.97:0/2685841065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6714107640 0x7f6714107a20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:27.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.617+0000 7f671c400700 1 -- 172.21.15.97:0/2685841065 >> 172.21.15.97:0/2685841065 conn(0x7f67140fd150 msgr2=0x7f67140ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:27.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.617+0000 7f671c400700 1 -- 172.21.15.97:0/2685841065 shutdown_connections 2024-03-11T19:48:27.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.617+0000 7f671c400700 1 -- 172.21.15.97:0/2685841065 wait complete. 2024-03-11T19:48:27.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.618+0000 7f671c400700 1 Processor -- start 2024-03-11T19:48:27.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.619+0000 7f671c400700 1 -- start start 2024-03-11T19:48:27.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.619+0000 7f671c400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6714107640 0x7f6714196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:27.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.619+0000 7f671c400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67141974e0 con 0x7f6714107640 2024-03-11T19:48:27.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.619+0000 7f6719a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6714107640 0x7f6714196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:27.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.619+0000 7f6719a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6714107640 0x7f6714196fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39846/0 (socket says 172.21.15.97:39846) 2024-03-11T19:48:27.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.619+0000 7f6719a00700 1 -- 172.21.15.97:0/4187370523 learned_addr learned my addr 172.21.15.97:0/4187370523 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:48:27.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.620+0000 7f6719a00700 1 -- 172.21.15.97:0/4187370523 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67080095f0 con 0x7f6714107640 2024-03-11T19:48:27.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.620+0000 7f6719a00700 1 --2- 172.21.15.97:0/4187370523 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6714107640 0x7f6714196fa0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6708006010 tx=0x7f6708004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:27.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.620+0000 7f6712000700 1 -- 172.21.15.97:0/4187370523 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f670801d070 con 0x7f6714107640 2024-03-11T19:48:27.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.620+0000 7f6712000700 1 -- 172.21.15.97:0/4187370523 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f670800bbd0 con 0x7f6714107640 2024-03-11T19:48:27.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.620+0000 7f6712000700 1 -- 172.21.15.97:0/4187370523 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6708003880 con 0x7f6714107640 2024-03-11T19:48:27.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.620+0000 7f671c400700 1 -- 172.21.15.97:0/4187370523 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6714197740 con 0x7f6714107640 2024-03-11T19:48:27.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.620+0000 7f671c400700 1 -- 172.21.15.97:0/4187370523 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6714197bc0 con 0x7f6714107640 2024-03-11T19:48:27.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.621+0000 7f6712000700 1 -- 172.21.15.97:0/4187370523 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f6708003ac0 con 0x7f6714107640 2024-03-11T19:48:27.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.622+0000 7f671c400700 1 -- 172.21.15.97:0/4187370523 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67140632a0 con 0x7f6714107640 2024-03-11T19:48:27.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.622+0000 7f6712000700 1 --2- 172.21.15.97:0/4187370523 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66fc038d50 0x7f66fc03b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:27.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.622+0000 7f6712000700 1 -- 172.21.15.97:0/4187370523 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f670804dad0 con 0x7f6714107640 2024-03-11T19:48:27.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.622+0000 7f6719000700 1 --2- 172.21.15.97:0/4187370523 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66fc038d50 0x7f66fc03b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:27.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.623+0000 7f6719000700 1 --2- 172.21.15.97:0/4187370523 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66fc038d50 0x7f66fc03b210 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6700006fd0 tx=0x7f6700006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:27.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.628+0000 7f6712000700 1 -- 172.21.15.97:0/4187370523 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6708029700 con 0x7f6714107640 2024-03-11T19:48:27.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.853+0000 7f671c400700 1 -- 172.21.15.97:0/4187370523 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f671410bc30 con 0x7f66fc038d50 2024-03-11T19:48:27.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.855+0000 7f6712000700 1 -- 172.21.15.97:0/4187370523 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f671410bc30 con 0x7f66fc038d50 2024-03-11T19:48:27.860 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:48:27.860 INFO:teuthology.orchestra.run.smithi097.stdout:[{"addr": "172.21.15.97", "hostname": "smithi097", "labels": [], "status": ""}, {"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.161", "hostname": "smithi161", "labels": [], "status": ""}] 2024-03-11T19:48:27.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.859+0000 7f671c400700 1 -- 172.21.15.97:0/4187370523 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66fc038d50 msgr2=0x7f66fc03b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:27.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.859+0000 7f671c400700 1 --2- 172.21.15.97:0/4187370523 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66fc038d50 0x7f66fc03b210 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6700006fd0 tx=0x7f6700006e40 comp rx=0 tx=0).stop 2024-03-11T19:48:27.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.859+0000 7f671c400700 1 -- 172.21.15.97:0/4187370523 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6714107640 msgr2=0x7f6714196fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:27.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.859+0000 7f671c400700 1 --2- 172.21.15.97:0/4187370523 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6714107640 0x7f6714196fa0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6708006010 tx=0x7f6708004ff0 comp rx=0 tx=0).stop 2024-03-11T19:48:27.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.859+0000 7f671c400700 1 -- 172.21.15.97:0/4187370523 shutdown_connections 2024-03-11T19:48:27.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.859+0000 7f671c400700 1 --2- 172.21.15.97:0/4187370523 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66fc038d50 0x7f66fc03b210 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:27.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.859+0000 7f671c400700 1 --2- 172.21.15.97:0/4187370523 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6714107640 0x7f6714196fa0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:27.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.859+0000 7f671c400700 1 -- 172.21.15.97:0/4187370523 >> 172.21.15.97:0/4187370523 conn(0x7f67140fd150 msgr2=0x7f67140fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:27.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.859+0000 7f671c400700 1 -- 172.21.15.97:0/4187370523 shutdown_connections 2024-03-11T19:48:27.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:27.859+0000 7f671c400700 1 -- 172.21.15.97:0/4187370523 wait complete. 2024-03-11T19:48:28.634 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T19:48:28.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T19:48:29.073 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:48:29.533 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:29 smithi097 ceph-mon[31898]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:48:29.533 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:29 smithi097 ceph-mon[31898]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:31.206 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.201+0000 7f2f74e00700 1 -- 172.21.15.97:0/2836793725 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f70102810 msgr2=0x7f2f70102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:31.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.201+0000 7f2f74e00700 1 --2- 172.21.15.97:0/2836793725 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f70102810 0x7f2f70102bf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009910 tx=0x7f2f60009c20 comp rx=0 tx=0).stop 2024-03-11T19:48:31.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.202+0000 7f2f74e00700 1 -- 172.21.15.97:0/2836793725 shutdown_connections 2024-03-11T19:48:31.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.202+0000 7f2f74e00700 1 --2- 172.21.15.97:0/2836793725 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f70102810 0x7f2f70102bf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:31.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.202+0000 7f2f74e00700 1 -- 172.21.15.97:0/2836793725 >> 172.21.15.97:0/2836793725 conn(0x7f2f700fc040 msgr2=0x7f2f700fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:31.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.202+0000 7f2f74e00700 1 -- 172.21.15.97:0/2836793725 shutdown_connections 2024-03-11T19:48:31.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.203+0000 7f2f74e00700 1 -- 172.21.15.97:0/2836793725 wait complete. 2024-03-11T19:48:31.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.204+0000 7f2f74e00700 1 Processor -- start 2024-03-11T19:48:31.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.204+0000 7f2f74e00700 1 -- start start 2024-03-11T19:48:31.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.204+0000 7f2f74e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f70102810 0x7f2f70196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:31.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.204+0000 7f2f74e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f701974e0 con 0x7f2f70102810 2024-03-11T19:48:31.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.204+0000 7f2f6de00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f70102810 0x7f2f70196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:31.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.204+0000 7f2f6de00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f70102810 0x7f2f70196fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39860/0 (socket says 172.21.15.97:39860) 2024-03-11T19:48:31.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.204+0000 7f2f6de00700 1 -- 172.21.15.97:0/2197778282 learned_addr learned my addr 172.21.15.97:0/2197778282 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:48:31.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.205+0000 7f2f6de00700 1 -- 172.21.15.97:0/2197778282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f600095f0 con 0x7f2f70102810 2024-03-11T19:48:31.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.205+0000 7f2f6de00700 1 --2- 172.21.15.97:0/2197778282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f70102810 0x7f2f70196fa0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f60006010 tx=0x7f2f60004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:31.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.205+0000 7f2f66a00700 1 -- 172.21.15.97:0/2197778282 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f6001d070 con 0x7f2f70102810 2024-03-11T19:48:31.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.205+0000 7f2f74e00700 1 -- 172.21.15.97:0/2197778282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f701976e0 con 0x7f2f70102810 2024-03-11T19:48:31.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.205+0000 7f2f66a00700 1 -- 172.21.15.97:0/2197778282 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f6000bc30 con 0x7f2f70102810 2024-03-11T19:48:31.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.205+0000 7f2f66a00700 1 -- 172.21.15.97:0/2197778282 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f600039a0 con 0x7f2f70102810 2024-03-11T19:48:31.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.205+0000 7f2f74e00700 1 -- 172.21.15.97:0/2197778282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f70197b00 con 0x7f2f70102810 2024-03-11T19:48:31.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.206+0000 7f2f66a00700 1 -- 172.21.15.97:0/2197778282 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f2f60003bf0 con 0x7f2f70102810 2024-03-11T19:48:31.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.207+0000 7f2f74e00700 1 -- 172.21.15.97:0/2197778282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f700632a0 con 0x7f2f70102810 2024-03-11T19:48:31.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.207+0000 7f2f66a00700 1 --2- 172.21.15.97:0/2197778282 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2f54038db0 0x7f2f5403b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:31.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.207+0000 7f2f66a00700 1 -- 172.21.15.97:0/2197778282 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2f6004cb70 con 0x7f2f70102810 2024-03-11T19:48:31.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.207+0000 7f2f6d400700 1 --2- 172.21.15.97:0/2197778282 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2f54038db0 0x7f2f5403b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:31.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.208+0000 7f2f6d400700 1 --2- 172.21.15.97:0/2197778282 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2f54038db0 0x7f2f5403b270 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c006fd0 tx=0x7f2f5c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:31.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.213+0000 7f2f66a00700 1 -- 172.21.15.97:0/2197778282 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2f60029950 con 0x7f2f70102810 2024-03-11T19:48:31.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:31 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:31.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:31 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:31.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:31 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:31.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:31 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:31.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:31 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:48:31.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:31 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:31.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:31 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:48:31.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:31 smithi097 ceph-mon[31898]: Updating smithi161:/etc/ceph/ceph.conf 2024-03-11T19:48:31.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:31 smithi097 ceph-mon[31898]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:31.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:31 smithi097 ceph-mon[31898]: Updating smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:31.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:31.443+0000 7f2f74e00700 1 -- 172.21.15.97:0/2197778282 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2f7010bb20 con 0x7f2f70102810 2024-03-11T19:48:32.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:32.380+0000 7f2f66a00700 1 -- 172.21.15.97:0/2197778282 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2f60026020 con 0x7f2f70102810 2024-03-11T19:48:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:32.384+0000 7f2f74e00700 1 -- 172.21.15.97:0/2197778282 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2f54038db0 msgr2=0x7f2f5403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:32.384+0000 7f2f74e00700 1 --2- 172.21.15.97:0/2197778282 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2f54038db0 0x7f2f5403b270 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c006fd0 tx=0x7f2f5c006e40 comp rx=0 tx=0).stop 2024-03-11T19:48:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:32.384+0000 7f2f74e00700 1 -- 172.21.15.97:0/2197778282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f70102810 msgr2=0x7f2f70196fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:32.384+0000 7f2f74e00700 1 --2- 172.21.15.97:0/2197778282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f70102810 0x7f2f70196fa0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f60006010 tx=0x7f2f60004ff0 comp rx=0 tx=0).stop 2024-03-11T19:48:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:32.385+0000 7f2f74e00700 1 -- 172.21.15.97:0/2197778282 shutdown_connections 2024-03-11T19:48:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:32.385+0000 7f2f74e00700 1 --2- 172.21.15.97:0/2197778282 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2f54038db0 0x7f2f5403b270 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:32.385+0000 7f2f74e00700 1 --2- 172.21.15.97:0/2197778282 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2f70102810 0x7f2f70196fa0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:32.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:32.385+0000 7f2f74e00700 1 -- 172.21.15.97:0/2197778282 >> 172.21.15.97:0/2197778282 conn(0x7f2f700fc040 msgr2=0x7f2f701035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:32.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:32.385+0000 7f2f74e00700 1 -- 172.21.15.97:0/2197778282 shutdown_connections 2024-03-11T19:48:32.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:32.385+0000 7f2f74e00700 1 -- 172.21.15.97:0/2197778282 wait complete. 2024-03-11T19:48:32.391 INFO:teuthology.orchestra.run.smithi097.stderr:adjusted tunables profile to default 2024-03-11T19:48:32.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:32 smithi097 ceph-mon[31898]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:48:32.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:32 smithi097 ceph-mon[31898]: Updating smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:48:32.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:32 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:32.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:32 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:32.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:32 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:32.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:32 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2197778282' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T19:48:33.132 INFO:tasks.cephadm:Adding mon.a on smithi097 2024-03-11T19:48:33.133 INFO:tasks.cephadm:Adding mon.b on smithi114 2024-03-11T19:48:33.133 INFO:tasks.cephadm:Adding mon.c on smithi161 2024-03-11T19:48:33.133 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi097:172.21.15.97=a;smithi114:172.21.15.114=b;smithi161:172.21.15.161=c' 2024-03-11T19:48:33.373 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:33.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:33 smithi097 ceph-mon[31898]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:33.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:33 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2197778282' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T19:48:33.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:33 smithi097 ceph-mon[31898]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:48:34.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.746+0000 7f14534a0700 1 -- 172.21.15.161:0/1125757494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f144c107690 msgr2=0x7f144c107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:34.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.746+0000 7f14534a0700 1 --2- 172.21.15.161:0/1125757494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f144c107690 0x7f144c107a70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1448009910 tx=0x7f1448009c20 comp rx=0 tx=0).stop 2024-03-11T19:48:34.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.746+0000 7f14534a0700 1 -- 172.21.15.161:0/1125757494 shutdown_connections 2024-03-11T19:48:34.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.746+0000 7f14534a0700 1 --2- 172.21.15.161:0/1125757494 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f144c107690 0x7f144c107a70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:34.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.746+0000 7f14534a0700 1 -- 172.21.15.161:0/1125757494 >> 172.21.15.161:0/1125757494 conn(0x7f144c0fd1a0 msgr2=0x7f144c0ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:34.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.746+0000 7f14534a0700 1 -- 172.21.15.161:0/1125757494 shutdown_connections 2024-03-11T19:48:34.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.746+0000 7f14534a0700 1 -- 172.21.15.161:0/1125757494 wait complete. 2024-03-11T19:48:34.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.747+0000 7f14534a0700 1 Processor -- start 2024-03-11T19:48:34.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.747+0000 7f14534a0700 1 -- start start 2024-03-11T19:48:34.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.747+0000 7f14534a0700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f144c107690 0x7f144c199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:34.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.747+0000 7f14534a0700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c199760 con 0x7f144c107690 2024-03-11T19:48:34.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.748+0000 7f145123c700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f144c107690 0x7f144c199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:34.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.748+0000 7f145123c700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f144c107690 0x7f144c199220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.161:49624/0 (socket says 172.21.15.161:49624) 2024-03-11T19:48:34.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.748+0000 7f145123c700 1 -- 172.21.15.161:0/3651854396 learned_addr learned my addr 172.21.15.161:0/3651854396 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-11T19:48:34.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.748+0000 7f145123c700 1 -- 172.21.15.161:0/3651854396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14480095f0 con 0x7f144c107690 2024-03-11T19:48:34.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.749+0000 7f145123c700 1 --2- 172.21.15.161:0/3651854396 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f144c107690 0x7f144c199220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1448006010 tx=0x7f1448004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:34.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.749+0000 7f14427fc700 1 -- 172.21.15.161:0/3651854396 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f144801d070 con 0x7f144c107690 2024-03-11T19:48:34.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.749+0000 7f14534a0700 1 -- 172.21.15.161:0/3651854396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144c1999c0 con 0x7f144c107690 2024-03-11T19:48:34.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.749+0000 7f14534a0700 1 -- 172.21.15.161:0/3651854396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144c199e40 con 0x7f144c107690 2024-03-11T19:48:34.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.749+0000 7f14427fc700 1 -- 172.21.15.161:0/3651854396 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f144800bbd0 con 0x7f144c107690 2024-03-11T19:48:34.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.749+0000 7f14427fc700 1 -- 172.21.15.161:0/3651854396 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1448003810 con 0x7f144c107690 2024-03-11T19:48:34.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.750+0000 7f14534a0700 1 -- 172.21.15.161:0/3651854396 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1430005330 con 0x7f144c107690 2024-03-11T19:48:34.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.750+0000 7f14427fc700 1 -- 172.21.15.161:0/3651854396 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f144800bd40 con 0x7f144c107690 2024-03-11T19:48:34.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.751+0000 7f14427fc700 1 --2- 172.21.15.161:0/3651854396 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1438038d50 0x7f143803b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:34.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.751+0000 7f14427fc700 1 -- 172.21.15.161:0/3651854396 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f144804cb30 con 0x7f144c107690 2024-03-11T19:48:34.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.751+0000 7f1450a3b700 1 --2- 172.21.15.161:0/3651854396 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1438038d50 0x7f143803b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:34.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.752+0000 7f1450a3b700 1 --2- 172.21.15.161:0/3651854396 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1438038d50 0x7f143803b210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f143c006fd0 tx=0x7f143c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:34.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.767+0000 7f14427fc700 1 -- 172.21.15.161:0/3651854396 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1448003970 con 0x7f144c107690 2024-03-11T19:48:34.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.970+0000 7f14534a0700 1 -- 172.21.15.161:0/3651854396 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi097:172.21.15.97=a;smithi114:172.21.15.114=b;smithi161:172.21.15.161=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f1430000c80 con 0x7f1438038d50 2024-03-11T19:48:34.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.986+0000 7f14427fc700 1 -- 172.21.15.161:0/3651854396 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1430000c80 con 0x7f1438038d50 2024-03-11T19:48:34.989 INFO:teuthology.orchestra.run.smithi161.stdout:Scheduled mon update... 2024-03-11T19:48:34.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.987+0000 7f14534a0700 1 -- 172.21.15.161:0/3651854396 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1438038d50 msgr2=0x7f143803b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:34.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.987+0000 7f14534a0700 1 --2- 172.21.15.161:0/3651854396 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1438038d50 0x7f143803b210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f143c006fd0 tx=0x7f143c006e40 comp rx=0 tx=0).stop 2024-03-11T19:48:34.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.987+0000 7f14534a0700 1 -- 172.21.15.161:0/3651854396 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f144c107690 msgr2=0x7f144c199220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:34.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.987+0000 7f14534a0700 1 --2- 172.21.15.161:0/3651854396 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f144c107690 0x7f144c199220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1448006010 tx=0x7f1448004ff0 comp rx=0 tx=0).stop 2024-03-11T19:48:34.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.987+0000 7f14534a0700 1 -- 172.21.15.161:0/3651854396 shutdown_connections 2024-03-11T19:48:34.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.987+0000 7f14534a0700 1 --2- 172.21.15.161:0/3651854396 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1438038d50 0x7f143803b210 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:34.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.987+0000 7f14534a0700 1 --2- 172.21.15.161:0/3651854396 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f144c107690 0x7f144c199220 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:34.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.987+0000 7f14534a0700 1 -- 172.21.15.161:0/3651854396 >> 172.21.15.161:0/3651854396 conn(0x7f144c0fd1a0 msgr2=0x7f144c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:34.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.987+0000 7f14534a0700 1 -- 172.21.15.161:0/3651854396 shutdown_connections 2024-03-11T19:48:34.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:34.987+0000 7f14534a0700 1 -- 172.21.15.161:0/3651854396 wait complete. 2024-03-11T19:48:35.636 DEBUG:teuthology.orchestra.run.smithi114:mon.b> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T19:48:35.639 DEBUG:teuthology.orchestra.run.smithi161:mon.c> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T19:48:35.641 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T19:48:35.641 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T19:48:35.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:35 smithi097 ceph-mon[31898]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:35.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:35.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:48:35.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:35.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:48:35.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:35.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:48:35.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:36.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:36 smithi097 ceph-mon[31898]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi097:172.21.15.97=a;smithi114:172.21.15.114=b;smithi161:172.21.15.161=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:48:36.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:36 smithi097 ceph-mon[31898]: Saving service mon spec with placement smithi097:172.21.15.97=a;smithi114:172.21.15.114=b;smithi161:172.21.15.161=c;count:3 2024-03-11T19:48:36.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:36 smithi097 ceph-mon[31898]: Deploying daemon mon.c on smithi161 2024-03-11T19:48:37.485 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:48:37.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:37 smithi097 ceph-mon[31898]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:39.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.213+0000 7f6c387cd700 1 -- 172.21.15.161:0/709141194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c30107740 msgr2=0x7f6c30107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:39.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.213+0000 7f6c387cd700 1 --2- 172.21.15.161:0/709141194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c30107740 0x7f6c30107b20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009910 tx=0x7f6c20009c20 comp rx=0 tx=0).stop 2024-03-11T19:48:39.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.214+0000 7f6c387cd700 1 -- 172.21.15.161:0/709141194 shutdown_connections 2024-03-11T19:48:39.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.214+0000 7f6c387cd700 1 --2- 172.21.15.161:0/709141194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c30107740 0x7f6c30107b20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:39.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.214+0000 7f6c387cd700 1 -- 172.21.15.161:0/709141194 >> 172.21.15.161:0/709141194 conn(0x7f6c300fd270 msgr2=0x7f6c300ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:39.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.214+0000 7f6c387cd700 1 -- 172.21.15.161:0/709141194 shutdown_connections 2024-03-11T19:48:39.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.214+0000 7f6c387cd700 1 -- 172.21.15.161:0/709141194 wait complete. 2024-03-11T19:48:39.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.214+0000 7f6c387cd700 1 Processor -- start 2024-03-11T19:48:39.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.214+0000 7f6c387cd700 1 -- start start 2024-03-11T19:48:39.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.214+0000 7f6c387cd700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c30107740 0x7f6c30073f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:39.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.214+0000 7f6c387cd700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c30075e60 con 0x7f6c30107740 2024-03-11T19:48:39.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.215+0000 7f6c36569700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c30107740 0x7f6c30073f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:39.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.215+0000 7f6c36569700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c30107740 0x7f6c30073f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.161:49636/0 (socket says 172.21.15.161:49636) 2024-03-11T19:48:39.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.215+0000 7f6c36569700 1 -- 172.21.15.161:0/513474717 learned_addr learned my addr 172.21.15.161:0/513474717 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-11T19:48:39.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.216+0000 7f6c36569700 1 -- 172.21.15.161:0/513474717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c200095f0 con 0x7f6c30107740 2024-03-11T19:48:39.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.216+0000 7f6c36569700 1 --2- 172.21.15.161:0/513474717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c30107740 0x7f6c30073f70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c20006010 tx=0x7f6c20004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:39.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.216+0000 7f6c277fe700 1 -- 172.21.15.161:0/513474717 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c2001d070 con 0x7f6c30107740 2024-03-11T19:48:39.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.216+0000 7f6c387cd700 1 -- 172.21.15.161:0/513474717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c300744b0 con 0x7f6c30107740 2024-03-11T19:48:39.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.216+0000 7f6c277fe700 1 -- 172.21.15.161:0/513474717 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c2000bbd0 con 0x7f6c30107740 2024-03-11T19:48:39.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.216+0000 7f6c277fe700 1 -- 172.21.15.161:0/513474717 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c20003880 con 0x7f6c30107740 2024-03-11T19:48:39.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.217+0000 7f6c387cd700 1 -- 172.21.15.161:0/513474717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c30074930 con 0x7f6c30107740 2024-03-11T19:48:39.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.218+0000 7f6c387cd700 1 -- 172.21.15.161:0/513474717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c30102e80 con 0x7f6c30107740 2024-03-11T19:48:39.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.218+0000 7f6c277fe700 1 -- 172.21.15.161:0/513474717 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f6c2000bd40 con 0x7f6c30107740 2024-03-11T19:48:39.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.218+0000 7f6c277fe700 1 --2- 172.21.15.161:0/513474717 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6c1c038d50 0x7f6c1c03b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:39.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.218+0000 7f6c277fe700 1 -- 172.21.15.161:0/513474717 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6c2004cb20 con 0x7f6c30107740 2024-03-11T19:48:39.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.219+0000 7f6c35d68700 1 --2- 172.21.15.161:0/513474717 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6c1c038d50 0x7f6c1c03b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:39.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.220+0000 7f6c35d68700 1 --2- 172.21.15.161:0/513474717 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6c1c038d50 0x7f6c1c03b210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c006fd0 tx=0x7f6c2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:39.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.235+0000 7f6c277fe700 1 -- 172.21.15.161:0/513474717 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6c200215f0 con 0x7f6c30107740 2024-03-11T19:48:39.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.524+0000 7f6c387cd700 1 -- 172.21.15.161:0/513474717 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c30074b20 con 0x7f6c30107740 2024-03-11T19:48:39.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.525+0000 7f6c277fe700 1 -- 172.21.15.161:0/513474717 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7f6c20026030 con 0x7f6c30107740 2024-03-11T19:48:39.530 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:48:39.530 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":1,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","modified":"2024-03-11T19:46:31.030802Z","created":"2024-03-11T19:46:31.030802Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T19:48:39.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.527+0000 7f6c387cd700 1 -- 172.21.15.161:0/513474717 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6c1c038d50 msgr2=0x7f6c1c03b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:39.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.527+0000 7f6c387cd700 1 --2- 172.21.15.161:0/513474717 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6c1c038d50 0x7f6c1c03b210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c006fd0 tx=0x7f6c2c006e40 comp rx=0 tx=0).stop 2024-03-11T19:48:39.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.527+0000 7f6c387cd700 1 -- 172.21.15.161:0/513474717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c30107740 msgr2=0x7f6c30073f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:39.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.527+0000 7f6c387cd700 1 --2- 172.21.15.161:0/513474717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c30107740 0x7f6c30073f70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c20006010 tx=0x7f6c20004ff0 comp rx=0 tx=0).stop 2024-03-11T19:48:39.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.527+0000 7f6c387cd700 1 -- 172.21.15.161:0/513474717 shutdown_connections 2024-03-11T19:48:39.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.527+0000 7f6c387cd700 1 --2- 172.21.15.161:0/513474717 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6c1c038d50 0x7f6c1c03b210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:39.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.527+0000 7f6c387cd700 1 --2- 172.21.15.161:0/513474717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6c30107740 0x7f6c30073f70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:39.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.527+0000 7f6c387cd700 1 -- 172.21.15.161:0/513474717 >> 172.21.15.161:0/513474717 conn(0x7f6c300fd270 msgr2=0x7f6c30109290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:39.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.527+0000 7f6c387cd700 1 -- 172.21.15.161:0/513474717 shutdown_connections 2024-03-11T19:48:39.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:39.527+0000 7f6c387cd700 1 -- 172.21.15.161:0/513474717 wait complete. 2024-03-11T19:48:39.533 INFO:teuthology.orchestra.run.smithi161.stderr:dumped monmap epoch 1 2024-03-11T19:48:39.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:39 smithi097 ceph-mon[31898]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:41.165 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T19:48:41.165 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T19:48:41.408 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:48:45.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:44 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:48:45.079 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:48:45.079 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: mon.a calling monitor election 2024-03-11T19:48:45.079 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:45.079 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: mon.c calling monitor election 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: monmap e2: 2 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],c=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: fsmap 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: mgrmap e13: x(active, since 60s) 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: overall HEALTH_OK 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:45.080 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:45.081 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:45.081 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:48:46.008 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:46.008 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:45 smithi161 ceph-mon[36332]: Deploying daemon mon.b on smithi114 2024-03-11T19:48:46.008 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:46.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.003+0000 7efc88eb4700 1 -- 172.21.15.161:0/3670196013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc6c0035c0 msgr2=0x7efc6c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:46.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.003+0000 7efc88eb4700 1 --2- 172.21.15.161:0/3670196013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc6c0035c0 0x7efc6c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efc84073f40 tx=0x7efc7400feb0 comp rx=0 tx=0).stop 2024-03-11T19:48:46.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.003+0000 7efc88eb4700 1 -- 172.21.15.161:0/3670196013 shutdown_connections 2024-03-11T19:48:46.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.003+0000 7efc88eb4700 1 --2- 172.21.15.161:0/3670196013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc6c0035c0 0x7efc6c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.003+0000 7efc88eb4700 1 -- 172.21.15.161:0/3670196013 >> 172.21.15.161:0/3670196013 conn(0x7efc840fd210 msgr2=0x7efc840ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.003+0000 7efc88eb4700 1 -- 172.21.15.161:0/3670196013 shutdown_connections 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.003+0000 7efc88eb4700 1 -- 172.21.15.161:0/3670196013 wait complete. 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.004+0000 7efc88eb4700 1 Processor -- start 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.004+0000 7efc88eb4700 1 -- start start 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.004+0000 7efc88eb4700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc6c0035c0 0x7efc8419d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.004+0000 7efc88eb4700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc8419dc20 0x7efc841a1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.004+0000 7efc88eb4700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8419e1b0 con 0x7efc6c0035c0 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.004+0000 7efc88eb4700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8419e320 con 0x7efc8419dc20 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.005+0000 7efc81d9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc8419dc20 0x7efc841a1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.005+0000 7efc81d9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc8419dc20 0x7efc841a1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38298/0 (socket says 172.21.15.161:38298) 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.005+0000 7efc81d9b700 1 -- 172.21.15.161:0/3934418119 learned_addr learned my addr 172.21.15.161:0/3934418119 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.005+0000 7efc81d9b700 1 -- 172.21.15.161:0/3934418119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc8419dc20 msgr2=0x7efc841a1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-11T19:48:46.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.005+0000 7efc81d9b700 1 -- 172.21.15.161:0/3934418119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc8419dc20 msgr2=0x7efc841a1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-11T19:48:46.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.005+0000 7efc81d9b700 1 --2- 172.21.15.161:0/3934418119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc8419dc20 0x7efc841a1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-11T19:48:46.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.005+0000 7efc81d9b700 1 --2- 172.21.15.161:0/3934418119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc8419dc20 0x7efc841a1fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:48:46.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.005+0000 7efc8259c700 1 --2- 172.21.15.161:0/3934418119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc6c0035c0 0x7efc8419d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:46.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.006+0000 7efc8259c700 1 -- 172.21.15.161:0/3934418119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc8419dc20 msgr2=0x7efc841a1fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:48:46.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.006+0000 7efc8259c700 1 --2- 172.21.15.161:0/3934418119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc8419dc20 0x7efc841a1fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:46.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.006+0000 7efc8259c700 1 -- 172.21.15.161:0/3934418119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7400f900 con 0x7efc6c0035c0 2024-03-11T19:48:46.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.006+0000 7efc8259c700 1 --2- 172.21.15.161:0/3934418119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc6c0035c0 0x7efc8419d6e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efc84073f40 tx=0x7efc740056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:46.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.006+0000 7efc737fe700 1 -- 172.21.15.161:0/3934418119 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7efc74004b60 con 0x7efc6c0035c0 2024-03-11T19:48:46.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.006+0000 7efc88eb4700 1 -- 172.21.15.161:0/3934418119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc841a2550 con 0x7efc6c0035c0 2024-03-11T19:48:46.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.006+0000 7efc737fe700 1 -- 172.21.15.161:0/3934418119 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc74013e40 con 0x7efc6c0035c0 2024-03-11T19:48:46.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.006+0000 7efc737fe700 1 -- 172.21.15.161:0/3934418119 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7efc74011ad0 con 0x7efc6c0035c0 2024-03-11T19:48:46.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.007+0000 7efc88eb4700 1 -- 172.21.15.161:0/3934418119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc841a2aa0 con 0x7efc6c0035c0 2024-03-11T19:48:46.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.008+0000 7efc88eb4700 1 -- 172.21.15.161:0/3934418119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc840632a0 con 0x7efc6c0035c0 2024-03-11T19:48:46.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.008+0000 7efc737fe700 1 -- 172.21.15.161:0/3934418119 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7efc74011d40 con 0x7efc6c0035c0 2024-03-11T19:48:46.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.008+0000 7efc737fe700 1 --2- 172.21.15.161:0/3934418119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efc64038af0 0x7efc6403afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:46.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.008+0000 7efc737fe700 1 -- 172.21.15.161:0/3934418119 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efc7404e380 con 0x7efc6c0035c0 2024-03-11T19:48:46.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.009+0000 7efc81d9b700 1 --2- 172.21.15.161:0/3934418119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efc64038af0 0x7efc6403afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:46.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.009+0000 7efc81d9b700 1 --2- 172.21.15.161:0/3934418119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efc64038af0 0x7efc6403afb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc78009180 tx=0x7efc78008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:46.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.023+0000 7efc737fe700 1 -- 172.21.15.161:0/3934418119 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efc74034530 con 0x7efc6c0035c0 2024-03-11T19:48:46.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:46.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:45 smithi097 ceph-mon[31898]: Deploying daemon mon.b on smithi114 2024-03-11T19:48:46.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:46.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.294+0000 7efc88eb4700 1 -- 172.21.15.161:0/3934418119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc8410fbd0 con 0x7efc6c0035c0 2024-03-11T19:48:46.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.295+0000 7efc737fe700 1 -- 172.21.15.161:0/3934418119 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7efc74034350 con 0x7efc6c0035c0 2024-03-11T19:48:46.299 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:48:46.300 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":2,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","modified":"2024-03-11T19:48:39.641668Z","created":"2024-03-11T19:46:31.030802Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T19:48:46.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.297+0000 7efc88eb4700 1 -- 172.21.15.161:0/3934418119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efc64038af0 msgr2=0x7efc6403afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:46.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.297+0000 7efc88eb4700 1 --2- 172.21.15.161:0/3934418119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efc64038af0 0x7efc6403afb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc78009180 tx=0x7efc78008040 comp rx=0 tx=0).stop 2024-03-11T19:48:46.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.297+0000 7efc88eb4700 1 -- 172.21.15.161:0/3934418119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc6c0035c0 msgr2=0x7efc8419d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:46.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.297+0000 7efc88eb4700 1 --2- 172.21.15.161:0/3934418119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc6c0035c0 0x7efc8419d6e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efc84073f40 tx=0x7efc740056f0 comp rx=0 tx=0).stop 2024-03-11T19:48:46.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.297+0000 7efc88eb4700 1 -- 172.21.15.161:0/3934418119 shutdown_connections 2024-03-11T19:48:46.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.297+0000 7efc88eb4700 1 --2- 172.21.15.161:0/3934418119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efc64038af0 0x7efc6403afb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:46.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.297+0000 7efc88eb4700 1 --2- 172.21.15.161:0/3934418119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc6c0035c0 0x7efc8419d6e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:46.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.297+0000 7efc88eb4700 1 --2- 172.21.15.161:0/3934418119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc8419dc20 0x7efc841a1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:46.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.297+0000 7efc88eb4700 1 -- 172.21.15.161:0/3934418119 >> 172.21.15.161:0/3934418119 conn(0x7efc840fd210 msgr2=0x7efc841961c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:46.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.298+0000 7efc88eb4700 1 -- 172.21.15.161:0/3934418119 shutdown_connections 2024-03-11T19:48:46.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:46.298+0000 7efc88eb4700 1 -- 172.21.15.161:0/3934418119 wait complete. 2024-03-11T19:48:46.303 INFO:teuthology.orchestra.run.smithi161.stderr:dumped monmap epoch 2 2024-03-11T19:48:46.858 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:46 smithi161 ceph-mon[36332]: from='client.? 172.21.15.161:0/3934418119' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T19:48:46.858 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:46 smithi161 ceph-mon[36332]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:46.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:46 smithi097 ceph-mon[31898]: from='client.? 172.21.15.161:0/3934418119' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T19:48:46.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:46 smithi097 ceph-mon[31898]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:46.922 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:48:46 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:48:46.641+0000 7f2043600700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-11T19:48:47.860 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T19:48:47.860 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T19:48:48.103 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:48:48.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 podman[36357]: 2024-03-11 19:48:48.356430469 +0000 UTC m=+0.017722742 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:48:48.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 podman[36357]: 2024-03-11 19:48:48.643037598 +0000 UTC m=+0.304329880 container init 02bc52469f3a26ff444c17563b5024af6d00eb9a8d0c3fa261bfa19d5d49458d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-b, RELEASE=squid-94a5397, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T19:48:48.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 podman[36357]: 2024-03-11 19:48:48.646116636 +0000 UTC m=+0.307408900 container start 02bc52469f3a26ff444c17563b5024af6d00eb9a8d0c3fa261bfa19d5d49458d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-b, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:48:48.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T19:48:48.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-11T19:48:48.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pidfile_write: ignore empty --pid-file 2024-03-11T19:48:48.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: load: jerasure load: lrc 2024-03-11T19:48:48.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: RocksDB version: 7.9.2 2024-03-11T19:48:48.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Git sha 0 2024-03-11T19:48:48.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T19:48:48.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: DB SUMMARY 2024-03-11T19:48:48.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: DB Session ID: UYBDT4G4XZ7I6JWN9FQY 2024-03-11T19:48:48.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: CURRENT file: CURRENT 2024-03-11T19:48:48.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T19:48:48.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T19:48:48.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-11T19:48:48.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-11T19:48:48.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.error_if_exists: 0 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.create_if_missing: 0 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.paranoid_checks: 1 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.env: 0x565324e04ae0 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.info_log: 0x5653269898c0 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.statistics: (nil) 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.use_fsync: 0 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_log_file_size: 0 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T19:48:48.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.allow_fallocate: 1 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.use_direct_reads: 0 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.db_log_dir: 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.wal_dir: 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T19:48:48.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.write_buffer_manager: 0x56532695d400 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.unordered_write: 0 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T19:48:48.998 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.row_cache: None 2024-03-11T19:48:48.999 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.wal_filter: None 2024-03-11T19:48:49.000 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T19:48:49.000 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T19:48:49.000 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.two_write_queues: 0 2024-03-11T19:48:49.000 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T19:48:49.000 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.wal_compression: 0 2024-03-11T19:48:49.000 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.atomic_flush: 0 2024-03-11T19:48:49.001 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T19:48:49.001 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T19:48:49.001 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T19:48:49.001 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.log_readahead_size: 0 2024-03-11T19:48:49.001 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T19:48:49.001 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T19:48:49.001 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T19:48:49.001 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T19:48:49.001 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T19:48:49.001 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T19:48:49.001 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T19:48:49.001 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_background_jobs: 2 2024-03-11T19:48:49.001 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_background_compactions: -1 2024-03-11T19:48:49.002 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_subcompactions: 1 2024-03-11T19:48:49.002 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T19:48:49.002 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T19:48:49.002 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T19:48:49.002 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T19:48:49.002 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T19:48:49.002 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T19:48:49.002 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T19:48:49.002 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T19:48:49.002 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_open_files: -1 2024-03-11T19:48:49.002 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T19:48:49.002 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T19:48:49.002 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_background_flushes: -1 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Compression algorithms supported: 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: kZSTD supported: 0 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: kXpressCompression supported: 0 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: kZlibCompression supported: 1 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: kSnappyCompression supported: 1 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: kLZ4Compression supported: 1 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: kBZip2Compression supported: 0 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T19:48:49.003 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.merge_operator: 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compaction_filter: None 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5653269899c0) 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_type: 0 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout: data_block_index_type: 0 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_shortening: 1 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout: checksum: 4 2024-03-11T19:48:49.004 INFO:journalctl@ceph.mon.b.smithi114.stdout: no_block_cache: 0 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache: 0x56532698d090 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_options: 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: capacity : 536870912 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: num_shard_bits : 4 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: strict_capacity_limit : 0 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_compressed: (nil) 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: persistent_cache: (nil) 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_size: 4096 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_size_deviation: 10 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_restart_interval: 16 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_block_restart_interval: 1 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: metadata_block_size: 4096 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: partition_filters: 0 2024-03-11T19:48:49.005 INFO:journalctl@ceph.mon.b.smithi114.stdout: use_delta_encoding: 1 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout: filter_policy: bloomfilter 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout: whole_key_filtering: 1 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout: verify_compression: 0 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout: format_version: 5 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout: enable_index_compression: 1 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_align: 0 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout: max_auto_readahead_size: 262144 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout: prepopulate_block_cache: 0 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout: initial_auto_readahead_size: 8192 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compression: NoCompression 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T19:48:49.006 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.num_levels: 7 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T19:48:49.007 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T19:48:49.008 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T19:48:49.009 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.table_properties_collectors: 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.inplace_update_support: 0 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.bloom_locality: 0 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.max_successive_merges: 0 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T19:48:49.010 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.ttl: 2592000 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.enable_blob_files: false 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.min_blob_size: 0 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T19:48:49.011 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 98747c36-3c5f-4e21-9954-e4cb9c9e09f0 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710186528675878, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710186528676288, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710186528, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "98747c36-3c5f-4e21-9954-e4cb9c9e09f0", "db_session_id": "UYBDT4G4XZ7I6JWN9FQY", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710186528676367, "job": 1, "event": "recovery_finished"} 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x565326a7e000 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: DB pointer 0x565326a68000 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** DB Stats ** 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T19:48:49.012 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2024-03-11T19:48:49.013 INFO:journalctl@ceph.mon.b.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T19:48:49.014 INFO:journalctl@ceph.mon.b.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T19:48:49.014 INFO:journalctl@ceph.mon.b.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T19:48:49.014 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-03-11T19:48:49.014 INFO:journalctl@ceph.mon.b.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T19:48:49.014 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-03-11T19:48:49.014 INFO:journalctl@ceph.mon.b.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T19:48:49.014 INFO:journalctl@ceph.mon.b.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T19:48:49.014 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T19:48:49.014 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T19:48:49.014 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T19:48:49.014 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T19:48:49.014 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout: Block cache BinnedLRUCache@0x56532698d090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: using public_addr v2:172.21.15.114:0/0 -> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: starting mon.b rank -1 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(???) e0 preinit fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(synchronizing).mds e1 new map 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout: e1 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T19:48:49.015 INFO:journalctl@ceph.mon.b.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout: legacy client fscid: -1 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout: No filesystems configured 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.016 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Deploying cephadm binary to smithi114 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Added host smithi114 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.017 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Updating smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Updating smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi161", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Deploying cephadm binary to smithi161 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.018 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Added host smithi161 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.019 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Updating smithi161:/etc/ceph/ceph.conf 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Updating smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Updating smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2197778282' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2197778282' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.020 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi097:172.21.15.97=a;smithi114:172.21.15.114=b;smithi161:172.21.15.161=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Saving service mon spec with placement smithi097:172.21.15.97=a;smithi114:172.21.15.114=b;smithi161:172.21.15.161=c;count:3 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Deploying daemon mon.c on smithi161 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.a calling monitor election 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:49.021 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.022 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:49.022 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:49.022 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.c calling monitor election 2024-03-11T19:48:49.022 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.022 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:49.022 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:49.022 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.022 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:49.022 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T19:48:49.022 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: monmap e2: 2 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],c=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:48:49.022 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: fsmap 2024-03-11T19:48:49.022 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:48:49.022 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mgrmap e13: x(active, since 60s) 2024-03-11T19:48:49.023 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: overall HEALTH_OK 2024-03-11T19:48:49.023 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.023 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.023 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:49.023 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:48:49.023 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:49.023 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: Deploying daemon mon.b on smithi114 2024-03-11T19:48:49.023 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:49.023 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: from='client.? 172.21.15.161:0/3934418119' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T19:48:49.023 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:49.023 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 ceph-mon[36376]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T19:48:49.023 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 bash[36357]: 02bc52469f3a26ff444c17563b5024af6d00eb9a8d0c3fa261bfa19d5d49458d 2024-03-11T19:48:49.024 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:48 smithi114 systemd[1]: Started Ceph mon.b for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:48:53.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.952+0000 7f2bea777700 1 -- 172.21.15.161:0/888047403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2bcc0035c0 msgr2=0x7f2bcc005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:53.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.952+0000 7f2bea777700 1 --2- 172.21.15.161:0/888047403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2bcc0035c0 0x7f2bcc005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2be4069640 tx=0x7f2bd800fe60 comp rx=0 tx=0).stop 2024-03-11T19:48:53.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.952+0000 7f2bea777700 1 -- 172.21.15.161:0/888047403 shutdown_connections 2024-03-11T19:48:53.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.952+0000 7f2bea777700 1 --2- 172.21.15.161:0/888047403 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2bcc0035c0 0x7f2bcc005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:53.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.952+0000 7f2bea777700 1 -- 172.21.15.161:0/888047403 >> 172.21.15.161:0/888047403 conn(0x7f2be40fc120 msgr2=0x7f2be40fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:53.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.952+0000 7f2bea777700 1 -- 172.21.15.161:0/888047403 shutdown_connections 2024-03-11T19:48:53.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.952+0000 7f2bea777700 1 -- 172.21.15.161:0/888047403 wait complete. 2024-03-11T19:48:53.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.953+0000 7f2bea777700 1 Processor -- start 2024-03-11T19:48:53.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.953+0000 7f2bea777700 1 -- start start 2024-03-11T19:48:53.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2bea777700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2bcc0035c0 0x7f2be41972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:53.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2bea777700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be41977f0 0x7f2be419bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:53.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2bea777700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2be419c0c0 0x7f2be419c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:53.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2bea777700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be4197ee0 con 0x7f2bcc0035c0 2024-03-11T19:48:53.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2bea777700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be41a3e30 con 0x7f2be419c0c0 2024-03-11T19:48:53.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2bea777700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be41a3f70 con 0x7f2be41977f0 2024-03-11T19:48:53.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2be37fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be41977f0 0x7f2be419bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:53.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2be37fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be41977f0 0x7f2be419bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38776/0 (socket says 172.21.15.161:38776) 2024-03-11T19:48:53.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2be37fe700 1 -- 172.21.15.161:0/2674494022 learned_addr learned my addr 172.21.15.161:0/2674494022 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-11T19:48:53.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2be8d14700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2be419c0c0 0x7f2be419c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:53.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2be37fe700 1 -- 172.21.15.161:0/2674494022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be41977f0 msgr2=0x7f2be419bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2be37fe700 1 -- 172.21.15.161:0/2674494022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be41977f0 msgr2=0x7f2be419bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2be37fe700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be41977f0 0x7f2be419bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2be37fe700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be41977f0 0x7f2be419bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2be8d14700 1 -- 172.21.15.161:0/2674494022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2be419c0c0 msgr2=0x7f2be419c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2be8d14700 1 -- 172.21.15.161:0/2674494022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2be419c0c0 msgr2=0x7f2be419c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2be8d14700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2be419c0c0 0x7f2be419c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2be3fff700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2bcc0035c0 0x7f2be41972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.954+0000 7f2be8d14700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2be419c0c0 0x7f2be419c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.955+0000 7f2be3fff700 1 -- 172.21.15.161:0/2674494022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be41977f0 msgr2=0x7f2be419bb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.955+0000 7f2be3fff700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be41977f0 0x7f2be419bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.955+0000 7f2be3fff700 1 -- 172.21.15.161:0/2674494022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2be419c0c0 msgr2=0x7f2be419c570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.955+0000 7f2be3fff700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2be419c0c0 0x7f2be419c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.955+0000 7f2be3fff700 1 -- 172.21.15.161:0/2674494022 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd800f8b0 con 0x7f2bcc0035c0 2024-03-11T19:48:53.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.955+0000 7f2be3fff700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2bcc0035c0 0x7f2be41972b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8015040 tx=0x7f2bd8012ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:53.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.956+0000 7f2be17fa700 1 -- 172.21.15.161:0/2674494022 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8004060 con 0x7f2bcc0035c0 2024-03-11T19:48:53.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.956+0000 7f2be17fa700 1 -- 172.21.15.161:0/2674494022 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2bd80041c0 con 0x7f2bcc0035c0 2024-03-11T19:48:53.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.956+0000 7f2bea777700 1 -- 172.21.15.161:0/2674494022 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be41a41f0 con 0x7f2bcc0035c0 2024-03-11T19:48:53.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.956+0000 7f2be17fa700 1 -- 172.21.15.161:0/2674494022 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8027870 con 0x7f2bcc0035c0 2024-03-11T19:48:53.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.956+0000 7f2bea777700 1 -- 172.21.15.161:0/2674494022 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be41a46f0 con 0x7f2bcc0035c0 2024-03-11T19:48:53.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.957+0000 7f2bea777700 1 -- 172.21.15.161:0/2674494022 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be4190f80 con 0x7f2bcc0035c0 2024-03-11T19:48:53.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.957+0000 7f2be17fa700 1 -- 172.21.15.161:0/2674494022 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f2bd8024050 con 0x7f2bcc0035c0 2024-03-11T19:48:53.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.958+0000 7f2be17fa700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2bd00391e0 0x7f2bd003b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:53.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.958+0000 7f2be17fa700 1 -- 172.21.15.161:0/2674494022 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2bd8000e40 con 0x7f2bcc0035c0 2024-03-11T19:48:53.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.958+0000 7f2be37fe700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2bd00391e0 0x7f2bd003b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:53.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.959+0000 7f2be37fe700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2bd00391e0 0x7f2bd003b6a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009180 tx=0x7f2bd4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:53.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:53.973+0000 7f2be17fa700 1 -- 172.21.15.161:0/2674494022 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2bd8010c60 con 0x7f2bcc0035c0 2024-03-11T19:48:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: mon.c calling monitor election 2024-03-11T19:48:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:48:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: mon.a calling monitor election 2024-03-11T19:48:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:54.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:54.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:54.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T19:48:54.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: monmap e3: 3 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:48:54.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: fsmap 2024-03-11T19:48:54.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:48:54.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: mgrmap e13: x(active, since 69s) 2024-03-11T19:48:54.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T19:48:54.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T19:48:54.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T19:48:54.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: mon.b (rank 2) addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] is down (out of quorum) 2024-03-11T19:48:54.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:54.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:54.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: mon.c calling monitor election 2024-03-11T19:48:54.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:48:54.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: mon.a calling monitor election 2024-03-11T19:48:54.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:54.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:54.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:54.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:54.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:54.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:54.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:54.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:54.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:54.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T19:48:54.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: monmap e3: 3 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:48:54.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: fsmap 2024-03-11T19:48:54.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:48:54.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: mgrmap e13: x(active, since 69s) 2024-03-11T19:48:54.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T19:48:54.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T19:48:54.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T19:48:54.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: mon.b (rank 2) addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] is down (out of quorum) 2024-03-11T19:48:54.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:54.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:54.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.271+0000 7f2bea777700 1 -- 172.21.15.161:0/2674494022 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2be4198670 con 0x7f2bcc0035c0 2024-03-11T19:48:54.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.272+0000 7f2be17fa700 1 -- 172.21.15.161:0/2674494022 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1283 (secure 0 0 0) 0x7f2bd8022090 con 0x7f2bcc0035c0 2024-03-11T19:48:54.276 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-11T19:48:54.276 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":3,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","modified":"2024-03-11T19:48:48.698865Z","created":"2024-03-11T19:46:31.030802Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T19:48:54.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.274+0000 7f2bea777700 1 -- 172.21.15.161:0/2674494022 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2bd00391e0 msgr2=0x7f2bd003b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:54.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.274+0000 7f2bea777700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2bd00391e0 0x7f2bd003b6a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009180 tx=0x7f2bd4008040 comp rx=0 tx=0).stop 2024-03-11T19:48:54.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.274+0000 7f2bea777700 1 -- 172.21.15.161:0/2674494022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2bcc0035c0 msgr2=0x7f2be41972b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:54.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.274+0000 7f2bea777700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2bcc0035c0 0x7f2be41972b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8015040 tx=0x7f2bd8012ef0 comp rx=0 tx=0).stop 2024-03-11T19:48:54.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.274+0000 7f2bea777700 1 -- 172.21.15.161:0/2674494022 shutdown_connections 2024-03-11T19:48:54.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.274+0000 7f2bea777700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2bd00391e0 0x7f2bd003b6a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:54.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.274+0000 7f2bea777700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2bcc0035c0 0x7f2be41972b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:54.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.274+0000 7f2bea777700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2be419c0c0 0x7f2be419c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:54.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.274+0000 7f2bea777700 1 --2- 172.21.15.161:0/2674494022 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be41977f0 0x7f2be419bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:54.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.274+0000 7f2bea777700 1 -- 172.21.15.161:0/2674494022 >> 172.21.15.161:0/2674494022 conn(0x7f2be40fc120 msgr2=0x7f2be4106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:54.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.274+0000 7f2bea777700 1 -- 172.21.15.161:0/2674494022 shutdown_connections 2024-03-11T19:48:54.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:48:54.274+0000 7f2bea777700 1 -- 172.21.15.161:0/2674494022 wait complete. 2024-03-11T19:48:54.280 INFO:teuthology.orchestra.run.smithi161.stderr:dumped monmap epoch 3 2024-03-11T19:48:54.821 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T19:48:54.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T19:48:55.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:54 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:55.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:54 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:55.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:54 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:48:55.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:54 smithi161 ceph-mon[36332]: from='client.? 172.21.15.161:0/2674494022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T19:48:55.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:54 smithi161 ceph-mon[36332]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:55.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:54 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:55.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:54 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:55.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:54 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:55.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:54 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:55.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:54 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:48:55.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:54 smithi097 ceph-mon[31898]: from='client.? 172.21.15.161:0/2674494022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T19:48:55.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:54 smithi097 ceph-mon[31898]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:55.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:54 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:55.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:54 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:55.243 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:48:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: mon.c calling monitor election 2024-03-11T19:48:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:48:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: mon.a calling monitor election 2024-03-11T19:48:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:48:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: monmap e3: 3 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: fsmap 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: mgrmap e13: x(active, since 69s) 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: mon.b (rank 2) addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] is down (out of quorum) 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:55.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:55.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:48:55.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='client.? 172.21.15.161:0/2674494022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T19:48:55.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:55.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:55.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:56.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: mon.b calling monitor election 2024-03-11T19:48:56.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: mon.b calling monitor election 2024-03-11T19:48:56.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: mon.c calling monitor election 2024-03-11T19:48:56.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: mon.a calling monitor election 2024-03-11T19:48:56.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T19:48:56.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: monmap e3: 3 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:48:56.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: fsmap 2024-03-11T19:48:56.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:48:56.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: mgrmap e13: x(active, since 71s) 2024-03-11T19:48:56.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T19:48:56.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: Cluster is now healthy 2024-03-11T19:48:56.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: overall HEALTH_OK 2024-03-11T19:48:56.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:56.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:56.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:56.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:56 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: mon.b calling monitor election 2024-03-11T19:48:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: mon.b calling monitor election 2024-03-11T19:48:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: mon.c calling monitor election 2024-03-11T19:48:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: mon.a calling monitor election 2024-03-11T19:48:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T19:48:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: monmap e3: 3 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:48:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: fsmap 2024-03-11T19:48:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:48:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: mgrmap e13: x(active, since 71s) 2024-03-11T19:48:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T19:48:56.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: Cluster is now healthy 2024-03-11T19:48:56.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: overall HEALTH_OK 2024-03-11T19:48:56.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:56.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:56.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:56.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:56 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: mon.b calling monitor election 2024-03-11T19:48:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: mon.b calling monitor election 2024-03-11T19:48:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: mon.c calling monitor election 2024-03-11T19:48:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: mon.a calling monitor election 2024-03-11T19:48:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T19:48:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: monmap e3: 3 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T19:48:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: fsmap 2024-03-11T19:48:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T19:48:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: mgrmap e13: x(active, since 71s) 2024-03-11T19:48:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T19:48:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: Cluster is now healthy 2024-03-11T19:48:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: overall HEALTH_OK 2024-03-11T19:48:57.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:56 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.405+0000 7f57faa00700 1 -- 172.21.15.97:0/3673189498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57f4107640 msgr2=0x7f57f4107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:57.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.405+0000 7f57faa00700 1 --2- 172.21.15.97:0/3673189498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57f4107640 0x7f57f4107a20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57e4009910 tx=0x7f57e4009c20 comp rx=0 tx=0).stop 2024-03-11T19:48:57.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.405+0000 7f57faa00700 1 -- 172.21.15.97:0/3673189498 shutdown_connections 2024-03-11T19:48:57.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.405+0000 7f57faa00700 1 --2- 172.21.15.97:0/3673189498 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57f4107640 0x7f57f4107a20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:57.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.405+0000 7f57faa00700 1 -- 172.21.15.97:0/3673189498 >> 172.21.15.97:0/3673189498 conn(0x7f57f40fd150 msgr2=0x7f57f40ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:57.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.405+0000 7f57faa00700 1 -- 172.21.15.97:0/3673189498 shutdown_connections 2024-03-11T19:48:57.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.405+0000 7f57faa00700 1 -- 172.21.15.97:0/3673189498 wait complete. 2024-03-11T19:48:57.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.406+0000 7f57faa00700 1 Processor -- start 2024-03-11T19:48:57.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.406+0000 7f57faa00700 1 -- start start 2024-03-11T19:48:57.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.407+0000 7f57faa00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57f4107640 0x7f57f4199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:57.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.407+0000 7f57faa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57f4199950 0x7f57f419dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:57.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.407+0000 7f57faa00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57f419e220 0x7f57f419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:57.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.407+0000 7f57faa00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f419a040 con 0x7f57f419e220 2024-03-11T19:48:57.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.407+0000 7f57faa00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f41a5f90 con 0x7f57f4199950 2024-03-11T19:48:57.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.407+0000 7f57faa00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f41a60d0 con 0x7f57f4107640 2024-03-11T19:48:57.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.407+0000 7f57f8a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57f419e220 0x7f57f419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:57.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.407+0000 7f57f8a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57f419e220 0x7f57f419e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59806/0 (socket says 172.21.15.97:59806) 2024-03-11T19:48:57.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.407+0000 7f57f8a00700 1 -- 172.21.15.97:0/1783410229 learned_addr learned my addr 172.21.15.97:0/1783410229 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:48:57.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.407+0000 7f57f3e00700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57f4107640 0x7f57f4199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:57.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.408+0000 7f57f3400700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57f4199950 0x7f57f419dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:57.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.408+0000 7f57f8a00700 1 -- 172.21.15.97:0/1783410229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57f4107640 msgr2=0x7f57f4199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:57.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.408+0000 7f57f8a00700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57f4107640 0x7f57f4199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:57.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.408+0000 7f57f8a00700 1 -- 172.21.15.97:0/1783410229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57f4199950 msgr2=0x7f57f419dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:57.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.408+0000 7f57f8a00700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57f4199950 0x7f57f419dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:57.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.408+0000 7f57f8a00700 1 -- 172.21.15.97:0/1783410229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e40095f0 con 0x7f57f419e220 2024-03-11T19:48:57.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.408+0000 7f57f8a00700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57f419e220 0x7f57f419e6d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00ba50 tx=0x7f57ec00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:57.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.410+0000 7f57f0c00700 1 -- 172.21.15.97:0/1783410229 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec00cab0 con 0x7f57f419e220 2024-03-11T19:48:57.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.410+0000 7f57f0c00700 1 -- 172.21.15.97:0/1783410229 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57ec00cc10 con 0x7f57f419e220 2024-03-11T19:48:57.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.410+0000 7f57faa00700 1 -- 172.21.15.97:0/1783410229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f41a63b0 con 0x7f57f419e220 2024-03-11T19:48:57.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.410+0000 7f57faa00700 1 -- 172.21.15.97:0/1783410229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f41a68b0 con 0x7f57f419e220 2024-03-11T19:48:57.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.410+0000 7f57f0c00700 1 -- 172.21.15.97:0/1783410229 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec0076c0 con 0x7f57f419e220 2024-03-11T19:48:57.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.411+0000 7f57f0c00700 1 -- 172.21.15.97:0/1783410229 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f57ec007820 con 0x7f57f419e220 2024-03-11T19:48:57.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.412+0000 7f57faa00700 1 -- 172.21.15.97:0/1783410229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d4005330 con 0x7f57f419e220 2024-03-11T19:48:57.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.412+0000 7f57f0c00700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f57d8039230 0x7f57d803b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:48:57.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.412+0000 7f57f0c00700 1 -- 172.21.15.97:0/1783410229 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f57ec04b110 con 0x7f57f419e220 2024-03-11T19:48:57.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.413+0000 7f57f3e00700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f57d8039230 0x7f57d803b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:48:57.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.413+0000 7f57f3e00700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f57d8039230 0x7f57d803b6f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f57e4006010 tx=0x7f57e400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:48:57.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.418+0000 7f57f0c00700 1 -- 172.21.15.97:0/1783410229 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f57ec01b8b0 con 0x7f57f419e220 2024-03-11T19:48:57.602 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: Updating smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:57.602 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: Updating smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:57.602 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: Updating smithi097:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:57.602 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:57.603 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:57.603 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.603 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.603 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.603 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.603 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.603 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.603 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.603 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:48:57.603 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:48:57.603 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:57.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.639+0000 7f57faa00700 1 -- 172.21.15.97:0/1783410229 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f57d40055e0 con 0x7f57f419e220 2024-03-11T19:48:57.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.639+0000 7f57f0c00700 1 -- 172.21.15.97:0/1783410229 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f57ec01b2c0 con 0x7f57f419e220 2024-03-11T19:48:57.645 INFO:teuthology.orchestra.run.smithi097.stdout:# minimal ceph.conf for e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:48:57.645 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2024-03-11T19:48:57.645 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:48:57.645 INFO:teuthology.orchestra.run.smithi097.stdout: mon_host = [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-03-11T19:48:57.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.642+0000 7f57faa00700 1 -- 172.21.15.97:0/1783410229 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f57d8039230 msgr2=0x7f57d803b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.642+0000 7f57faa00700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f57d8039230 0x7f57d803b6f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f57e4006010 tx=0x7f57e400b540 comp rx=0 tx=0).stop 2024-03-11T19:48:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.642+0000 7f57faa00700 1 -- 172.21.15.97:0/1783410229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57f419e220 msgr2=0x7f57f419e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:48:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.642+0000 7f57faa00700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57f419e220 0x7f57f419e6d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00ba50 tx=0x7f57ec00be10 comp rx=0 tx=0).stop 2024-03-11T19:48:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.642+0000 7f57faa00700 1 -- 172.21.15.97:0/1783410229 shutdown_connections 2024-03-11T19:48:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.642+0000 7f57faa00700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f57d8039230 0x7f57d803b6f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.642+0000 7f57faa00700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f57f419e220 0x7f57f419e6d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.642+0000 7f57faa00700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57f4107640 0x7f57f4199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.642+0000 7f57faa00700 1 --2- 172.21.15.97:0/1783410229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57f4199950 0x7f57f419dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:48:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.642+0000 7f57faa00700 1 -- 172.21.15.97:0/1783410229 >> 172.21.15.97:0/1783410229 conn(0x7f57f40fd150 msgr2=0x7f57f4109170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:48:57.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.643+0000 7f57faa00700 1 -- 172.21.15.97:0/1783410229 shutdown_connections 2024-03-11T19:48:57.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:48:57.643+0000 7f57faa00700 1 -- 172.21.15.97:0/1783410229 wait complete. 2024-03-11T19:48:57.921 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:48:57 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:48:57.698+0000 7f2043600700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-11T19:48:57.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: Updating smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:57.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: Updating smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:57.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: Updating smithi097:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:57.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:57.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:57.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:57.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:48:57.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:48:57.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:58.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: Updating smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:58.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: Updating smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:58.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: Updating smithi097:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:48:58.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:58.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:48:58.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:58.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:58.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:58.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:58.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:58.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:58.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:48:58.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:48:58.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:48:58.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:58.345 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T19:48:58.346 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:48:58.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T19:48:58.435 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:48:58.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:48:58.572 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-11T19:48:58.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T19:48:58.607 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-11T19:48:58.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:48:58.677 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-11T19:48:58.678 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T19:48:58.708 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-11T19:48:58.709 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:48:58.776 INFO:tasks.cephadm:Adding mgr.x on smithi097 2024-03-11T19:48:58.776 INFO:tasks.cephadm:Adding mgr.y on smithi114 2024-03-11T19:48:58.776 INFO:tasks.cephadm:Adding mgr.z on smithi161 2024-03-11T19:48:58.776 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch apply mgr '3;smithi097=x;smithi114=y;smithi161=z' 2024-03-11T19:48:58.845 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:58 smithi097 ceph-mon[31898]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T19:48:58.845 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:58 smithi097 ceph-mon[31898]: Reconfiguring daemon mon.a on smithi097 2024-03-11T19:48:58.845 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:58 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1783410229' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:58.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:58 smithi114 ceph-mon[36376]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T19:48:58.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:58 smithi114 ceph-mon[36376]: Reconfiguring daemon mon.a on smithi097 2024-03-11T19:48:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:58 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1783410229' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:59.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:58 smithi161 ceph-mon[36332]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T19:48:59.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:58 smithi161 ceph-mon[36332]: Reconfiguring daemon mon.a on smithi097 2024-03-11T19:48:59.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:58 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1783410229' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:48:59.046 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:48:59.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:48:59 smithi161 ceph-mon[36332]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:59.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:48:59 smithi097 ceph-mon[31898]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:48:59.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:48:59 smithi114 ceph-mon[36376]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:00.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.596+0000 7f0458b7e700 1 -- 172.21.15.161:0/1317839286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0454100690 msgr2=0x7f0454100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:49:00.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.596+0000 7f0458b7e700 1 --2- 172.21.15.161:0/1317839286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0454100690 0x7f0454100ab0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0444009910 tx=0x7f0444009c20 comp rx=0 tx=0).stop 2024-03-11T19:49:00.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.596+0000 7f0458b7e700 1 -- 172.21.15.161:0/1317839286 shutdown_connections 2024-03-11T19:49:00.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.596+0000 7f0458b7e700 1 --2- 172.21.15.161:0/1317839286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0454100690 0x7f0454100ab0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:00.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.596+0000 7f0458b7e700 1 -- 172.21.15.161:0/1317839286 >> 172.21.15.161:0/1317839286 conn(0x7f04540fbc50 msgr2=0x7f04540fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:49:00.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.596+0000 7f0458b7e700 1 -- 172.21.15.161:0/1317839286 shutdown_connections 2024-03-11T19:49:00.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.596+0000 7f0458b7e700 1 -- 172.21.15.161:0/1317839286 wait complete. 2024-03-11T19:49:00.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.597+0000 7f0458b7e700 1 Processor -- start 2024-03-11T19:49:00.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.597+0000 7f0458b7e700 1 -- start start 2024-03-11T19:49:00.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.597+0000 7f0458b7e700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0454100690 0x7f045406a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:49:00.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.597+0000 7f0458b7e700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0454199b30 0x7f045419bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:49:00.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f0458b7e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045419c4b0 0x7f045419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:49:00.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f0458b7e700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045419f0f0 con 0x7f0454199b30 2024-03-11T19:49:00.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f0458b7e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045419f260 con 0x7f045419c4b0 2024-03-11T19:49:00.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f0458b7e700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045419f420 con 0x7f0454100690 2024-03-11T19:49:00.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f0452d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045419c4b0 0x7f045419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:49:00.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f045259c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0454100690 0x7f045406a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:49:00.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f0452d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045419c4b0 0x7f045419e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.161:60114/0 (socket says 172.21.15.161:60114) 2024-03-11T19:49:00.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f045259c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0454100690 0x7f045406a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38780/0 (socket says 172.21.15.161:38780) 2024-03-11T19:49:00.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f0452d9d700 1 -- 172.21.15.161:0/2217248037 learned_addr learned my addr 172.21.15.161:0/2217248037 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-11T19:49:00.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f0449d9b700 1 --2- 172.21.15.161:0/2217248037 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0454199b30 0x7f045419bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:49:00.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f045259c700 1 -- 172.21.15.161:0/2217248037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045419c4b0 msgr2=0x7f045419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:49:00.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f045259c700 1 --2- 172.21.15.161:0/2217248037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045419c4b0 0x7f045419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:00.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f045259c700 1 -- 172.21.15.161:0/2217248037 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0454199b30 msgr2=0x7f045419bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:49:00.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f045259c700 1 --2- 172.21.15.161:0/2217248037 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0454199b30 0x7f045419bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:00.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f045259c700 1 -- 172.21.15.161:0/2217248037 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04440095f0 con 0x7f0454100690 2024-03-11T19:49:00.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f045259c700 1 --2- 172.21.15.161:0/2217248037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0454100690 0x7f045406a140 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0444006010 tx=0x7f044400bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:49:00.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f044bfff700 1 -- 172.21.15.161:0/2217248037 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044401d070 con 0x7f0454100690 2024-03-11T19:49:00.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f0458b7e700 1 -- 172.21.15.161:0/2217248037 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045419f6a0 con 0x7f0454100690 2024-03-11T19:49:00.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f044bfff700 1 -- 172.21.15.161:0/2217248037 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0444004e00 con 0x7f0454100690 2024-03-11T19:49:00.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.598+0000 7f044bfff700 1 -- 172.21.15.161:0/2217248037 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0444022e60 con 0x7f0454100690 2024-03-11T19:49:00.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.599+0000 7f0458b7e700 1 -- 172.21.15.161:0/2217248037 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f045419fbc0 con 0x7f0454100690 2024-03-11T19:49:00.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.599+0000 7f044bfff700 1 -- 172.21.15.161:0/2217248037 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f044400f460 con 0x7f0454100690 2024-03-11T19:49:00.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.599+0000 7f044bfff700 1 --2- 172.21.15.161:0/2217248037 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f04400391e0 0x7f044003b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:49:00.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.599+0000 7f044bfff700 1 -- 172.21.15.161:0/2217248037 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f044404cc50 con 0x7f0454100690 2024-03-11T19:49:00.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.599+0000 7f0458b7e700 1 -- 172.21.15.161:0/2217248037 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045418f4a0 con 0x7f0454100690 2024-03-11T19:49:00.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.600+0000 7f0449d9b700 1 --2- 172.21.15.161:0/2217248037 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f04400391e0 0x7f044003b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:49:00.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.601+0000 7f0449d9b700 1 --2- 172.21.15.161:0/2217248037 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f04400391e0 0x7f044003b6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f045406a6a0 tx=0x7f043c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:49:00.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.605+0000 7f044bfff700 1 -- 172.21.15.161:0/2217248037 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0444027070 con 0x7f0454100690 2024-03-11T19:49:00.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.810+0000 7f0458b7e700 1 -- 172.21.15.161:0/2217248037 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi097=x;smithi114=y;smithi161=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f045402d870 con 0x7f04400391e0 2024-03-11T19:49:00.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.824+0000 7f044bfff700 1 -- 172.21.15.161:0/2217248037 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f045402d870 con 0x7f04400391e0 2024-03-11T19:49:00.829 INFO:teuthology.orchestra.run.smithi161.stdout:Scheduled mgr update... 2024-03-11T19:49:00.829 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:00.829 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:00.829 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:00 smithi161 ceph-mon[36332]: Reconfiguring mon.b (monmap changed)... 2024-03-11T19:49:00.829 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:49:00.829 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:49:00.829 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:00.829 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:00 smithi161 ceph-mon[36332]: Reconfiguring daemon mon.b on smithi114 2024-03-11T19:49:00.830 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:00 smithi161 ceph-mon[36332]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:00.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.827+0000 7f0458b7e700 1 -- 172.21.15.161:0/2217248037 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f04400391e0 msgr2=0x7f044003b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:49:00.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.827+0000 7f0458b7e700 1 --2- 172.21.15.161:0/2217248037 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f04400391e0 0x7f044003b6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f045406a6a0 tx=0x7f043c008040 comp rx=0 tx=0).stop 2024-03-11T19:49:00.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.827+0000 7f0458b7e700 1 -- 172.21.15.161:0/2217248037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0454100690 msgr2=0x7f045406a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:49:00.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.827+0000 7f0458b7e700 1 --2- 172.21.15.161:0/2217248037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0454100690 0x7f045406a140 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0444006010 tx=0x7f044400bd60 comp rx=0 tx=0).stop 2024-03-11T19:49:00.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.827+0000 7f0458b7e700 1 -- 172.21.15.161:0/2217248037 shutdown_connections 2024-03-11T19:49:00.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.827+0000 7f0458b7e700 1 --2- 172.21.15.161:0/2217248037 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f04400391e0 0x7f044003b6a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:00.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.827+0000 7f0458b7e700 1 --2- 172.21.15.161:0/2217248037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045419c4b0 0x7f045419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:00.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.827+0000 7f0458b7e700 1 --2- 172.21.15.161:0/2217248037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0454100690 0x7f045406a140 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:00.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.827+0000 7f0458b7e700 1 --2- 172.21.15.161:0/2217248037 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0454199b30 0x7f045419bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:00.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.827+0000 7f0458b7e700 1 -- 172.21.15.161:0/2217248037 >> 172.21.15.161:0/2217248037 conn(0x7f04540fbc50 msgr2=0x7f04540fc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:49:00.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.827+0000 7f0458b7e700 1 -- 172.21.15.161:0/2217248037 shutdown_connections 2024-03-11T19:49:00.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:49:00.827+0000 7f0458b7e700 1 -- 172.21.15.161:0/2217248037 wait complete. 2024-03-11T19:49:00.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:00.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:00.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:00 smithi114 ceph-mon[36376]: Reconfiguring mon.b (monmap changed)... 2024-03-11T19:49:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:49:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:49:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:00 smithi114 ceph-mon[36376]: Reconfiguring daemon mon.b on smithi114 2024-03-11T19:49:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:00 smithi114 ceph-mon[36376]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:01.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:01.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:01.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:00 smithi097 ceph-mon[31898]: Reconfiguring mon.b (monmap changed)... 2024-03-11T19:49:01.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:49:01.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:49:01.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:01.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:00 smithi097 ceph-mon[31898]: Reconfiguring daemon mon.b on smithi114 2024-03-11T19:49:01.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:00 smithi097 ceph-mon[31898]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:01.478 DEBUG:teuthology.orchestra.run.smithi114:mgr.y> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T19:49:01.481 DEBUG:teuthology.orchestra.run.smithi161:mgr.z> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.z.service 2024-03-11T19:49:01.483 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T19:49:01.483 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:49:01.483 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T19:49:01.542 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T19:49:01.542 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2024-03-11T19:49:01.594 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T19:49:01.594 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:49:01.594 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-03-11T19:49:01.594 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:49:01.594 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:49:01.594 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-11 19:48:00.759754094 +0000 2024-03-11T19:49:01.594 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-11 19:41:08.689075255 +0000 2024-03-11T19:49:01.594 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-11 19:41:08.689075255 +0000 2024-03-11T19:49:01.594 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-11 19:41:08.689075255 +0000 2024-03-11T19:49:01.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T19:49:01.684 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-11T19:49:01.684 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-11T19:49:01.684 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000366923 s, 1.4 MB/s 2024-03-11T19:49:01.689 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T19:49:01.744 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2024-03-11T19:49:01.794 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T19:49:01.794 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:49:01.794 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-03-11T19:49:01.794 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:49:01.794 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:49:01.794 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-11 19:48:00.861752338 +0000 2024-03-11T19:49:01.794 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-11 19:41:09.265065581 +0000 2024-03-11T19:49:01.794 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-11 19:41:09.265065581 +0000 2024-03-11T19:49:01.795 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-11 19:41:09.265065581 +0000 2024-03-11T19:49:01.795 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T19:49:01.877 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-11T19:49:01.877 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-11T19:49:01.878 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000287555 s, 1.8 MB/s 2024-03-11T19:49:01.879 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T19:49:01.932 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2024-03-11T19:49:01.978 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T19:49:01.978 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:49:01.978 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-03-11T19:49:01.978 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:49:01.978 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:49:01.978 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-11 19:48:00.930751150 +0000 2024-03-11T19:49:01.978 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-11 19:41:09.857055638 +0000 2024-03-11T19:49:01.978 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-11 19:41:09.857055638 +0000 2024-03-11T19:49:01.978 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-11 19:41:09.857055638 +0000 2024-03-11T19:49:01.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T19:49:02.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:01 smithi161 ceph-mon[36332]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi097=x;smithi114=y;smithi161=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:49:02.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:01 smithi161 ceph-mon[36332]: Saving service mgr spec with placement smithi097=x;smithi114=y;smithi161=z;count:3 2024-03-11T19:49:02.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:01 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:02.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:01 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:02.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:01 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:02.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:01 smithi161 ceph-mon[36332]: Reconfiguring mon.c (monmap changed)... 2024-03-11T19:49:02.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:01 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:49:02.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:01 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:49:02.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:01 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:02.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:01 smithi161 ceph-mon[36332]: Reconfiguring daemon mon.c on smithi161 2024-03-11T19:49:02.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:01 smithi097 ceph-mon[31898]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi097=x;smithi114=y;smithi161=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:49:02.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:01 smithi097 ceph-mon[31898]: Saving service mgr spec with placement smithi097=x;smithi114=y;smithi161=z;count:3 2024-03-11T19:49:02.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:01 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:02.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:01 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:02.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:01 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:02.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:01 smithi097 ceph-mon[31898]: Reconfiguring mon.c (monmap changed)... 2024-03-11T19:49:02.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:01 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:49:02.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:01 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:49:02.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:01 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:02.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:01 smithi097 ceph-mon[31898]: Reconfiguring daemon mon.c on smithi161 2024-03-11T19:49:02.064 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-11T19:49:02.064 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-11T19:49:02.064 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000363396 s, 1.4 MB/s 2024-03-11T19:49:02.069 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T19:49:02.121 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2024-03-11T19:49:02.169 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T19:49:02.169 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:49:02.169 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-03-11T19:49:02.169 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:49:02.169 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:49:02.169 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-11 19:48:00.987750169 +0000 2024-03-11T19:49:02.170 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-11 19:41:10.446045744 +0000 2024-03-11T19:49:02.170 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-11 19:41:10.446045744 +0000 2024-03-11T19:49:02.170 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-11 19:41:10.446045744 +0000 2024-03-11T19:49:02.170 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T19:49:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:01 smithi114 ceph-mon[36376]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi097=x;smithi114=y;smithi161=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:49:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:01 smithi114 ceph-mon[36376]: Saving service mgr spec with placement smithi097=x;smithi114=y;smithi161=z;count:3 2024-03-11T19:49:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:01 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:01 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:01 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:01 smithi114 ceph-mon[36376]: Reconfiguring mon.c (monmap changed)... 2024-03-11T19:49:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:01 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T19:49:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:01 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T19:49:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:01 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:01 smithi114 ceph-mon[36376]: Reconfiguring daemon mon.c on smithi161 2024-03-11T19:49:02.257 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-11T19:49:02.257 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-11T19:49:02.257 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000377054 s, 1.4 MB/s 2024-03-11T19:49:02.261 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T19:49:02.317 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-11T19:49:02.317 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T19:49:02.335 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T19:49:02.335 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-03-11T19:49:02.392 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T19:49:02.392 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:49:02.392 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-11T19:49:02.392 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:49:02.392 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:49:02.392 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-11 19:48:48.169384354 +0000 2024-03-11T19:49:02.392 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-11 19:40:05.478471814 +0000 2024-03-11T19:49:02.392 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-11 19:40:05.478471814 +0000 2024-03-11T19:49:02.392 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-11 19:40:05.478471814 +0000 2024-03-11T19:49:02.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T19:49:02.459 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-11T19:49:02.459 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-11T19:49:02.459 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000204576 s, 2.5 MB/s 2024-03-11T19:49:02.461 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T19:49:02.517 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-03-11T19:49:02.573 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T19:49:02.573 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:49:02.573 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-11T19:49:02.573 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:49:02.573 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:49:02.573 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-11 19:48:48.169384354 +0000 2024-03-11T19:49:02.573 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-11 19:40:05.818464312 +0000 2024-03-11T19:49:02.574 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-11 19:40:05.818464312 +0000 2024-03-11T19:49:02.574 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-11 19:40:05.818464312 +0000 2024-03-11T19:49:02.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T19:49:02.639 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-11T19:49:02.640 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-11T19:49:02.640 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000170366 s, 3.0 MB/s 2024-03-11T19:49:02.642 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T19:49:02.697 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-03-11T19:49:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T19:49:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:49:02.753 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-11T19:49:02.753 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:49:02.753 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:49:02.753 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-11 19:48:48.169384354 +0000 2024-03-11T19:49:02.753 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-11 19:40:06.143457141 +0000 2024-03-11T19:49:02.753 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-11 19:40:06.143457141 +0000 2024-03-11T19:49:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-11 19:40:06.143457141 +0000 2024-03-11T19:49:02.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T19:49:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-11T19:49:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-11T19:49:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000195321 s, 2.6 MB/s 2024-03-11T19:49:02.823 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T19:49:02.878 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-03-11T19:49:02.932 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T19:49:02.932 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:49:02.932 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-11T19:49:02.932 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:49:02.932 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:49:02.932 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-11 19:48:48.170384332 +0000 2024-03-11T19:49:02.932 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-11 19:40:06.475449816 +0000 2024-03-11T19:49:02.932 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-11 19:40:06.475449816 +0000 2024-03-11T19:49:02.933 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-11 19:40:06.475449816 +0000 2024-03-11T19:49:02.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T19:49:02.997 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-11T19:49:02.998 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-11T19:49:02.998 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000189584 s, 2.7 MB/s 2024-03-11T19:49:02.999 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T19:49:03.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:02 smithi161 ceph-mon[36332]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:03.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:02 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:03.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:02 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:03.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:02 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:03.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:02 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:03.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:02 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:03.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:02 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:03.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:02 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:49:03.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:02 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T19:49:03.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:02 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:49:03.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:02 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:03.055 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-11T19:49:03.055 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T19:49:03.070 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T19:49:03.070 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 2024-03-11T19:49:03.126 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T19:49:03.126 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:49:03.126 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-11T19:49:03.126 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:49:03.126 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:49:03.126 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-11 19:48:38.462040985 +0000 2024-03-11T19:49:03.126 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-11 19:40:26.519938123 +0000 2024-03-11T19:49:03.126 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-11 19:40:26.519938123 +0000 2024-03-11T19:49:03.126 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-11 19:40:26.519938123 +0000 2024-03-11T19:49:03.126 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T19:49:03.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:02 smithi097 ceph-mon[31898]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:03.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:02 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:03.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:02 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:03.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:02 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:03.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:02 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:03.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:02 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:03.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:02 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:03.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:02 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:49:03.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:02 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T19:49:03.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:02 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:49:03.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:02 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:03.192 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-11T19:49:03.192 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-11T19:49:03.192 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.00025114 s, 2.0 MB/s 2024-03-11T19:49:03.193 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T19:49:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:02 smithi114 ceph-mon[36376]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:02 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:03.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:02 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:03.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:02 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:03.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:02 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:03.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:02 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:03.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:02 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:03.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:02 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:49:03.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:02 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T19:49:03.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:02 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:49:03.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:02 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:03.250 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 2024-03-11T19:49:03.305 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T19:49:03.306 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:49:03.306 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-03-11T19:49:03.306 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:49:03.306 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:49:03.306 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-11 19:48:38.462040985 +0000 2024-03-11T19:49:03.306 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-11 19:40:26.861930817 +0000 2024-03-11T19:49:03.306 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-11 19:40:26.861930817 +0000 2024-03-11T19:49:03.306 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-11 19:40:26.861930817 +0000 2024-03-11T19:49:03.306 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T19:49:03.371 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-11T19:49:03.371 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-11T19:49:03.372 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000205998 s, 2.5 MB/s 2024-03-11T19:49:03.373 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T19:49:03.428 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 2024-03-11T19:49:03.483 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T19:49:03.483 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:49:03.483 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-11T19:49:03.483 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:49:03.483 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:49:03.483 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-11 19:48:38.462040985 +0000 2024-03-11T19:49:03.483 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-11 19:40:27.204923490 +0000 2024-03-11T19:49:03.483 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-11 19:40:27.204923490 +0000 2024-03-11T19:49:03.483 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-11 19:40:27.204923490 +0000 2024-03-11T19:49:03.484 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T19:49:03.548 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-11T19:49:03.548 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-11T19:49:03.548 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000172191 s, 3.0 MB/s 2024-03-11T19:49:03.549 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T19:49:03.605 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 2024-03-11T19:49:03.660 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T19:49:03.660 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T19:49:03.661 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-11T19:49:03.661 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:49:03.661 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T19:49:03.661 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-11 19:48:38.463040964 +0000 2024-03-11T19:49:03.661 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-11 19:40:27.544916227 +0000 2024-03-11T19:49:03.661 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-11 19:40:27.544916227 +0000 2024-03-11T19:49:03.661 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-11 19:40:27.544916227 +0000 2024-03-11T19:49:03.661 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T19:49:03.727 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-11T19:49:03.727 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-11T19:49:03.727 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000218444 s, 2.3 MB/s 2024-03-11T19:49:03.728 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T19:49:03.784 INFO:tasks.cephadm:Deploying osd.0 on smithi097 with /dev/vg_nvme/lv_4... 2024-03-11T19:49:03.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T19:49:04.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:03 smithi161 ceph-mon[36332]: Deploying daemon mgr.z on smithi161 2024-03-11T19:49:04.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:03 smithi097 ceph-mon[31898]: Deploying daemon mgr.z on smithi161 2024-03-11T19:49:04.218 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:49:04.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:03 smithi114 ceph-mon[36376]: Deploying daemon mgr.z on smithi161 2024-03-11T19:49:04.790 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:04 smithi161 systemd[1]: Starting Ceph mgr.z for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T19:49:04.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:04 smithi097 ceph-mon[31898]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:05.118 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:04 smithi161 ceph-mon[36332]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:05.118 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:04 smithi161 podman[38174]: 2024-03-11 19:49:04.759441935 +0000 UTC m=+0.017790718 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:49:05.118 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:04 smithi161 podman[38174]: 2024-03-11 19:49:04.885287993 +0000 UTC m=+0.143636743 container create b71e0c963bb256d1d6cf47fae870b83bb7f934641eedf81f8cd81f4fd543d3bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-11T19:49:05.118 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:05 smithi161 podman[38174]: 2024-03-11 19:49:05.087736159 +0000 UTC m=+0.346084906 container init b71e0c963bb256d1d6cf47fae870b83bb7f934641eedf81f8cd81f4fd543d3bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=) 2024-03-11T19:49:05.118 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:05 smithi161 podman[38174]: 2024-03-11 19:49:05.090345825 +0000 UTC m=+0.348694576 container start b71e0c963bb256d1d6cf47fae870b83bb7f934641eedf81f8cd81f4fd543d3bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-11T19:49:05.119 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:05 smithi161 ceph-mgr[38193]: -- 172.21.15.161:0/1547626333 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ef30aa41a0 con 0x55ef30a7a800 2024-03-11T19:49:05.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:04 smithi114 ceph-mon[36376]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:05.540 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:05 smithi161 bash[38174]: b71e0c963bb256d1d6cf47fae870b83bb7f934641eedf81f8cd81f4fd543d3bc 2024-03-11T19:49:05.540 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:05 smithi161 systemd[1]: Started Ceph mgr.z for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:49:06.039 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:05 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:05.629+0000 7f6c52690200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T19:49:06.040 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:05 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:05.722+0000 7f6c52690200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T19:49:06.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:06 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:06.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:06 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:06.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:06 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:06.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:06 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:49:06.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:06 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T19:49:06.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:06 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:49:06.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:06 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:06.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:06 smithi114 ceph-mon[36376]: Deploying daemon mgr.y on smithi114 2024-03-11T19:49:06.540 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:06 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:06.358+0000 7f6c52690200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T19:49:06.540 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:06 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:06.448+0000 7f6c52690200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T19:49:06.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:06 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:06.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:06 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:06.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:06 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:06.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:06 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:49:06.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:06 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T19:49:06.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:06 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:49:06.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:06 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:06.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:06 smithi161 ceph-mon[36332]: Deploying daemon mgr.y on smithi114 2024-03-11T19:49:06.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:06 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:06.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:06 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:06.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:06 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:06.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:06 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:49:06.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:06 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T19:49:06.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:06 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:49:06.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:06 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:06.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:06 smithi097 ceph-mon[31898]: Deploying daemon mgr.y on smithi114 2024-03-11T19:49:07.037 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:06 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:06.591+0000 7f6c52690200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T19:49:07.037 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:06 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:06.684+0000 7f6c52690200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T19:49:07.037 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:06 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:06.765+0000 7f6c52690200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T19:49:07.037 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:07 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:07.033+0000 7f6c52690200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T19:49:07.120 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T19:49:07.120 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T19:49:07.120 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-03-11T19:49:07.120 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-03-11T19:49:07.120 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0355296 s, 295 MB/s 2024-03-11T19:49:07.120 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-03-11T19:49:07.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:07 smithi161 ceph-mon[36332]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:07.290 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:07 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:07.141+0000 7f6c52690200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T19:49:07.290 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:07 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:07.226+0000 7f6c52690200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T19:49:07.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:07 smithi097 ceph-mon[31898]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:07.455 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:07 smithi114 ceph-mon[36376]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:07.742 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:07 smithi114 ceph-mgr[37767]: -- 172.21.15.114:0/142238472 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b52c7841a0 con 0x55b52c75b000 2024-03-11T19:49:07.743 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:07 smithi114 bash[37748]: 45ae7a66dab3b7dcf23002aca70a26b2d2285c855905cbecd9fccdc38cef8706 2024-03-11T19:49:07.789 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:07 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:07.322+0000 7f6c52690200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T19:49:07.790 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:07 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:07.510+0000 7f6c52690200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T19:49:07.993 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:07 smithi114 systemd[1]: Started Ceph mgr.y for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:49:07.999 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi097:vg_nvme/lv_4 2024-03-11T19:49:08.424 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:49:08.493 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:08.216+0000 7f2e8a159200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T19:49:08.494 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:08.313+0000 7f2e8a159200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T19:49:08.790 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:08.500+0000 7f6c52690200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T19:49:08.790 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:08.639+0000 7f6c52690200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T19:49:09.079 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:08 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:09.079 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:08 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:09.079 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:08 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:09.079 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:08 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:09.079 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:08 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:09.079 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:08 smithi114 ceph-mon[36376]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:09.080 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:08.977+0000 7f2e8a159200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T19:49:09.080 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:09.074+0000 7f2e8a159200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T19:49:09.098 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:08 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:09.098 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:08 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:09.098 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:08 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:09.098 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:08 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:09.099 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:08 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:09.099 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:08 smithi161 ceph-mon[36332]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:09.099 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:09.001+0000 7f6c52690200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T19:49:09.099 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:09.094+0000 7f6c52690200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T19:49:09.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:08 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:09.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:08 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:09.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:08 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:09.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:08 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:09.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:08 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:09.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:08 smithi097 ceph-mon[31898]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:09.406 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:09.216+0000 7f2e8a159200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T19:49:09.407 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:09.310+0000 7f2e8a159200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T19:49:09.407 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:09.401+0000 7f2e8a159200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T19:49:09.485 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:09.288+0000 7f6c52690200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:49:09.957 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:09.667+0000 7f2e8a159200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T19:49:09.957 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:09.776+0000 7f2e8a159200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T19:49:09.957 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:09.860+0000 7f2e8a159200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T19:49:09.957 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:09.952+0000 7f2e8a159200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T19:49:10.029 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:09.481+0000 7f6c52690200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T19:49:10.029 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:09.765+0000 7f6c52690200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:49:10.029 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:10 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:10.024+0000 7f6c52690200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T19:49:10.243 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:10 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:10.125+0000 7f2e8a159200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T19:49:10.289 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:10 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:10.256+0000 7f6c52690200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T19:49:10.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.429+0000 7efc35a00700 1 -- 172.21.15.97:0/2207826802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc30101cb0 msgr2=0x7efc301020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:49:10.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.429+0000 7efc35a00700 1 --2- 172.21.15.97:0/2207826802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc30101cb0 0x7efc301020d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efc200099c0 tx=0x7efc20009cd0 comp rx=0 tx=0).stop 2024-03-11T19:49:10.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.429+0000 7efc35a00700 1 -- 172.21.15.97:0/2207826802 shutdown_connections 2024-03-11T19:49:10.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.429+0000 7efc35a00700 1 --2- 172.21.15.97:0/2207826802 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc30101cb0 0x7efc301020d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:10.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.429+0000 7efc35a00700 1 --2- 172.21.15.97:0/2207826802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc30103850 0x7efc3010a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:10.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.429+0000 7efc35a00700 1 --2- 172.21.15.97:0/2207826802 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc30102eb0 0x7efc30103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:10.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.429+0000 7efc35a00700 1 -- 172.21.15.97:0/2207826802 >> 172.21.15.97:0/2207826802 conn(0x7efc300fd210 msgr2=0x7efc300ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:49:10.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.429+0000 7efc35a00700 1 -- 172.21.15.97:0/2207826802 shutdown_connections 2024-03-11T19:49:10.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.429+0000 7efc35a00700 1 -- 172.21.15.97:0/2207826802 wait complete. 2024-03-11T19:49:10.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.430+0000 7efc35a00700 1 Processor -- start 2024-03-11T19:49:10.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.430+0000 7efc35a00700 1 -- start start 2024-03-11T19:49:10.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.430+0000 7efc35a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc30101cb0 0x7efc30199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:49:10.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.430+0000 7efc35a00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc30102eb0 0x7efc3019a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:49:10.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.430+0000 7efc35a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc30103850 0x7efc301a14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:49:10.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.430+0000 7efc35a00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc301a1a90 con 0x7efc30103850 2024-03-11T19:49:10.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.430+0000 7efc35a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc301a1bd0 con 0x7efc30101cb0 2024-03-11T19:49:10.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.430+0000 7efc35a00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc301a1d60 con 0x7efc30102eb0 2024-03-11T19:49:10.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc2ea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc30101cb0 0x7efc30199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:49:10.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc2e000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc30102eb0 0x7efc3019a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:49:10.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc2ea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc30101cb0 0x7efc30199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:34794/0 (socket says 172.21.15.97:34794) 2024-03-11T19:49:10.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc2e000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc30102eb0 0x7efc3019a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:48722/0 (socket says 172.21.15.97:48722) 2024-03-11T19:49:10.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc2ea00700 1 -- 172.21.15.97:0/3842880976 learned_addr learned my addr 172.21.15.97:0/3842880976 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:49:10.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc2f400700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc30103850 0x7efc301a14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:49:10.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc2e000700 1 -- 172.21.15.97:0/3842880976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc30101cb0 msgr2=0x7efc30199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:49:10.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc2e000700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc30101cb0 0x7efc30199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:10.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc2e000700 1 -- 172.21.15.97:0/3842880976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc30103850 msgr2=0x7efc301a14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:49:10.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc2e000700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc30103850 0x7efc301a14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:10.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc2e000700 1 -- 172.21.15.97:0/3842880976 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc200095f0 con 0x7efc30102eb0 2024-03-11T19:49:10.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc2f400700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc30103850 0x7efc301a14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:49:10.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc2e000700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc30102eb0 0x7efc3019a490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efc1c00ea20 tx=0x7efc1c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:49:10.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.431+0000 7efc27400700 1 -- 172.21.15.97:0/3842880976 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c0096d0 con 0x7efc30102eb0 2024-03-11T19:49:10.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.432+0000 7efc27400700 1 -- 172.21.15.97:0/3842880976 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc1c00cbd0 con 0x7efc30102eb0 2024-03-11T19:49:10.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.432+0000 7efc27400700 1 -- 172.21.15.97:0/3842880976 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c018680 con 0x7efc30102eb0 2024-03-11T19:49:10.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.432+0000 7efc35a00700 1 -- 172.21.15.97:0/3842880976 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc301a1fe0 con 0x7efc30102eb0 2024-03-11T19:49:10.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.432+0000 7efc35a00700 1 -- 172.21.15.97:0/3842880976 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc301a2500 con 0x7efc30102eb0 2024-03-11T19:49:10.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.433+0000 7efc27400700 1 -- 172.21.15.97:0/3842880976 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7efc1c0187e0 con 0x7efc30102eb0 2024-03-11T19:49:10.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.433+0000 7efc35a00700 1 -- 172.21.15.97:0/3842880976 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc30068960 con 0x7efc30102eb0 2024-03-11T19:49:10.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.433+0000 7efc27400700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efc14038f90 0x7efc1403b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:49:10.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.433+0000 7efc27400700 1 -- 172.21.15.97:0/3842880976 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efc1c014070 con 0x7efc30102eb0 2024-03-11T19:49:10.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.433+0000 7efc2ea00700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efc14038f90 0x7efc1403b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:49:10.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.434+0000 7efc2ea00700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efc14038f90 0x7efc1403b450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efc20003890 tx=0x7efc200058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:49:10.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.438+0000 7efc27400700 1 -- 172.21.15.97:0/3842880976 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efc1c01f9a0 con 0x7efc30102eb0 2024-03-11T19:49:10.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:10.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:10.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.610 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:49:10.611 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:49:10.611 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:10.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:10.673+0000 7efc35a00700 1 -- 172.21.15.97:0/3842880976 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7efc300755c0 con 0x7efc14038f90 2024-03-11T19:49:10.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:10.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:10.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:49:10.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:49:10.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T19:49:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T19:49:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:11.493 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:11.085+0000 7f2e8a159200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T19:49:11.494 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:11.214+0000 7f2e8a159200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T19:49:11.494 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:11.047+0000 7f6c52690200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T19:49:11.790 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:11.615+0000 7f6c52690200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T19:49:11.790 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:11.711+0000 7f6c52690200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T19:49:11.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:11 smithi161 ceph-mon[36332]: Reconfiguring mgr.x (unknown last config time)... 2024-03-11T19:49:11.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:11 smithi161 ceph-mon[36332]: Reconfiguring daemon mgr.x on smithi097 2024-03-11T19:49:11.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:11 smithi161 ceph-mon[36332]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:11.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:11 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:49:11.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:11 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:49:11.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:11 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:11.797 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:11 smithi097 ceph-mon[31898]: Reconfiguring mgr.x (unknown last config time)... 2024-03-11T19:49:11.797 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:11 smithi097 ceph-mon[31898]: Reconfiguring daemon mgr.x on smithi097 2024-03-11T19:49:11.797 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:11 smithi097 ceph-mon[31898]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:11.797 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:11 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:49:11.797 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:11 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:49:11.797 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:11 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:11.831 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:11.545+0000 7f2e8a159200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T19:49:11.831 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:11.634+0000 7f2e8a159200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T19:49:11.831 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:11.826+0000 7f2e8a159200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:49:11.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:11 smithi114 ceph-mon[36376]: Reconfiguring mgr.x (unknown last config time)... 2024-03-11T19:49:11.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:11 smithi114 ceph-mon[36376]: Reconfiguring daemon mgr.x on smithi097 2024-03-11T19:49:11.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:11 smithi114 ceph-mon[36376]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:11.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:11 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:49:11.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:11 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:49:11.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:11 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:12.243 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:12.007+0000 7f2e8a159200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T19:49:12.289 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:49:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:49:11.922+0000 7f6c52690200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T19:49:12.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:12.502+0000 7efc27400700 1 -- 172.21.15.97:0/3842880976 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mgrmap(e 14) v1 ==== 91024+0+0 (secure 0 0 0) 0x7efc1c00cd40 con 0x7efc30102eb0 2024-03-11T19:49:12.548 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:12.286+0000 7f2e8a159200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:49:12.548 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:12.543+0000 7f2e8a159200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T19:49:12.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-mon[36376]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:49:12.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-mon[36376]: Standby manager daemon z started 2024-03-11T19:49:12.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.161:0/2493472466' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T19:49:12.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.161:0/2493472466' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:49:12.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.161:0/2493472466' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T19:49:12.549 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.161:0/2493472466' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:49:12.549 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:12.549 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:12.549 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:12.549 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:12.549 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:12.549 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:12.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:12 smithi161 ceph-mon[36332]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:49:12.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:12 smithi161 ceph-mon[36332]: Standby manager daemon z started 2024-03-11T19:49:12.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:12 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.161:0/2493472466' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T19:49:12.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:12 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.161:0/2493472466' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:49:12.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:12 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.161:0/2493472466' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T19:49:12.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:12 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.161:0/2493472466' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:49:12.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:12 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:12.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:12 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:12.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:12 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:12.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:12 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:12.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:12 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:12.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:12 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:12.899 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:12 smithi097 ceph-mon[31898]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:49:12.899 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:12 smithi097 ceph-mon[31898]: Standby manager daemon z started 2024-03-11T19:49:12.899 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:12 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.161:0/2493472466' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T19:49:12.900 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:12 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.161:0/2493472466' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:49:12.900 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:12 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.161:0/2493472466' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T19:49:12.900 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:12 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.161:0/2493472466' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:49:12.900 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:12 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:12.900 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:12 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:12.900 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:12 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:12.900 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:12 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:12.900 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:12 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:12.900 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:12 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:12.993 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:12 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:12.773+0000 7f2e8a159200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T19:49:13.595 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:13 smithi097 ceph-mon[31898]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:13.595 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:13 smithi097 ceph-mon[31898]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-11T19:49:13.595 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:13 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T19:49:13.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:13 smithi161 ceph-mon[36332]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:13.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:13 smithi161 ceph-mon[36332]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-11T19:49:13.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:13 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T19:49:13.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:13 smithi114 ceph-mon[36376]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:13.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:13 smithi114 ceph-mon[36376]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-11T19:49:13.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:13 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T19:49:13.994 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:13 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:13.560+0000 7f2e8a159200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T19:49:14.454 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:14 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:14.139+0000 7f2e8a159200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T19:49:14.454 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:14 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:14.235+0000 7f2e8a159200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T19:49:14.454 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:49:14 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:49:14.449+0000 7f2e8a159200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T19:49:14.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:14.537+0000 7efc27400700 1 -- 172.21.15.97:0/3842880976 <== mon.1 v2:172.21.15.161:3300/0 8 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7efc1c017750 con 0x7efc30102eb0 2024-03-11T19:49:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:14 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.114:0/3620791872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T19:49:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:14 smithi114 ceph-mon[36376]: Standby manager daemon y started 2024-03-11T19:49:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:14 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.114:0/3620791872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:49:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:14 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.114:0/3620791872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T19:49:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:14 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.114:0/3620791872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:49:14.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:14 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.114:0/3620791872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T19:49:14.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:14 smithi161 ceph-mon[36332]: Standby manager daemon y started 2024-03-11T19:49:14.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:14 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.114:0/3620791872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:49:14.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:14 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.114:0/3620791872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T19:49:14.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:14 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.114:0/3620791872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:49:14.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:14 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.114:0/3620791872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T19:49:14.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:14 smithi097 ceph-mon[31898]: Standby manager daemon y started 2024-03-11T19:49:14.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:14 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.114:0/3620791872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:49:14.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:14 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.114:0/3620791872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T19:49:14.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:14 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.114:0/3620791872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:49:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:15 smithi161 ceph-mon[36332]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:15 smithi161 ceph-mon[36332]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-11T19:49:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:15 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T19:49:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:15 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2883072926' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaa3b17b-f7a0-4cf1-857c-5e3415f224a9"}]: dispatch 2024-03-11T19:49:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:15 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaa3b17b-f7a0-4cf1-857c-5e3415f224a9"}]: dispatch 2024-03-11T19:49:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:15 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaa3b17b-f7a0-4cf1-857c-5e3415f224a9"}]': finished 2024-03-11T19:49:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:15 smithi161 ceph-mon[36332]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T19:49:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:15 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:15.824 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:15 smithi097 ceph-mon[31898]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:15.824 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:15 smithi097 ceph-mon[31898]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-11T19:49:15.824 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:15 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T19:49:15.824 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:15 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2883072926' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaa3b17b-f7a0-4cf1-857c-5e3415f224a9"}]: dispatch 2024-03-11T19:49:15.824 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:15 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaa3b17b-f7a0-4cf1-857c-5e3415f224a9"}]: dispatch 2024-03-11T19:49:15.825 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:15 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaa3b17b-f7a0-4cf1-857c-5e3415f224a9"}]': finished 2024-03-11T19:49:15.825 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:15 smithi097 ceph-mon[31898]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T19:49:15.825 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:15 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:15.897 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:15 smithi114 ceph-mon[36376]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:15.898 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:15 smithi114 ceph-mon[36376]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-11T19:49:15.898 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:15 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T19:49:15.898 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:15 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2883072926' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaa3b17b-f7a0-4cf1-857c-5e3415f224a9"}]: dispatch 2024-03-11T19:49:15.898 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:15 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaa3b17b-f7a0-4cf1-857c-5e3415f224a9"}]: dispatch 2024-03-11T19:49:15.898 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:15 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaa3b17b-f7a0-4cf1-857c-5e3415f224a9"}]': finished 2024-03-11T19:49:15.898 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:15 smithi114 ceph-mon[36376]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T19:49:15.898 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:15 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:16.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:16 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2389109930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:49:16.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:16 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2389109930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:49:17.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:16 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2389109930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:49:17.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:17 smithi097 ceph-mon[31898]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:17 smithi114 ceph-mon[36376]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:18.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:17 smithi161 ceph-mon[36332]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:19.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:19 smithi097 ceph-mon[31898]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:19.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:19 smithi114 ceph-mon[36376]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:20.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:19 smithi161 ceph-mon[36332]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:21.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:21 smithi097 ceph-mon[31898]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:21.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:21 smithi114 ceph-mon[36376]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:22.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:21 smithi161 ceph-mon[36332]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:23.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:23 smithi097 ceph-mon[31898]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:23.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:23 smithi114 ceph-mon[36376]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:24.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:23 smithi161 ceph-mon[36332]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:24.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:24 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T19:49:24.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:24 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:24.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:24 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T19:49:24.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:24 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:25.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:24 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T19:49:25.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:24 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:25.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:25 smithi097 ceph-mon[31898]: Deploying daemon osd.0 on smithi097 2024-03-11T19:49:25.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:25 smithi097 ceph-mon[31898]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:25 smithi114 ceph-mon[36376]: Deploying daemon osd.0 on smithi097 2024-03-11T19:49:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:25 smithi114 ceph-mon[36376]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:26.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:25 smithi161 ceph-mon[36332]: Deploying daemon osd.0 on smithi097 2024-03-11T19:49:26.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:25 smithi161 ceph-mon[36332]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:27.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:27 smithi097 ceph-mon[31898]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:27.842 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:27 smithi161 ceph-mon[36332]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:27.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:27 smithi114 ceph-mon[36376]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:29.741 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:29 smithi097 ceph-mon[31898]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:29.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:29 smithi114 ceph-mon[36376]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:30.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:29 smithi161 ceph-mon[36332]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:31.878 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:31 smithi097 ceph-mon[31898]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:31.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:31 smithi114 ceph-mon[36376]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:32.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:31 smithi161 ceph-mon[36332]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:33.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:33 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:33.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:33 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:33.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:33 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:33.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:33 smithi097 ceph-mon[31898]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:33.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:33 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:33.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:33 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:33.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:33 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:33.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:33 smithi114 ceph-mon[36376]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:33.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:33 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:33.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:33 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:33.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:33 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:33.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:33 smithi161 ceph-mon[36332]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:35.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:35 smithi097 ceph-mon[31898]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:35.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:35 smithi114 ceph-mon[36376]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:35.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:35 smithi161 ceph-mon[36332]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:36 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:36 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:36 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:36 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:36 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:36.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:36 smithi097 ceph-mon[31898]: from='osd.0 [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T19:49:36.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:36 smithi097 ceph-mon[31898]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T19:49:36.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:36 smithi097 ceph-mon[31898]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:36.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:36 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:36 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:36 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:36 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:36 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:36 smithi114 ceph-mon[36376]: from='osd.0 [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T19:49:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:36 smithi114 ceph-mon[36376]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T19:49:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:36 smithi114 ceph-mon[36376]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:36 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:36 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:36 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:36 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:36 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:36 smithi161 ceph-mon[36332]: from='osd.0 [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T19:49:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:36 smithi161 ceph-mon[36332]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T19:49:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:36 smithi161 ceph-mon[36332]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:37.068 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 0 on host 'smithi097' 2024-03-11T19:49:37.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:37.063+0000 7efc27400700 1 -- 172.21.15.97:0/3842880976 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efc300755c0 con 0x7efc14038f90 2024-03-11T19:49:37.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:37.067+0000 7efc35a00700 1 -- 172.21.15.97:0/3842880976 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efc14038f90 msgr2=0x7efc1403b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:49:37.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:37.067+0000 7efc35a00700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efc14038f90 0x7efc1403b450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efc20003890 tx=0x7efc200058e0 comp rx=0 tx=0).stop 2024-03-11T19:49:37.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:37.067+0000 7efc35a00700 1 -- 172.21.15.97:0/3842880976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc30102eb0 msgr2=0x7efc3019a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:49:37.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:37.067+0000 7efc35a00700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc30102eb0 0x7efc3019a490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efc1c00ea20 tx=0x7efc1c00ed30 comp rx=0 tx=0).stop 2024-03-11T19:49:37.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:37.067+0000 7efc35a00700 1 -- 172.21.15.97:0/3842880976 shutdown_connections 2024-03-11T19:49:37.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:37.067+0000 7efc35a00700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efc14038f90 0x7efc1403b450 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:37.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:37.067+0000 7efc35a00700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efc30103850 0x7efc301a14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:37.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:37.067+0000 7efc35a00700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efc30101cb0 0x7efc30199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:37.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:37.067+0000 7efc35a00700 1 --2- 172.21.15.97:0/3842880976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc30102eb0 0x7efc3019a490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:37.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:37.067+0000 7efc35a00700 1 -- 172.21.15.97:0/3842880976 >> 172.21.15.97:0/3842880976 conn(0x7efc300fd210 msgr2=0x7efc30073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:49:37.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:37.067+0000 7efc35a00700 1 -- 172.21.15.97:0/3842880976 shutdown_connections 2024-03-11T19:49:37.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:49:37.067+0000 7efc35a00700 1 -- 172.21.15.97:0/3842880976 wait complete. 2024-03-11T19:49:37.754 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:37 smithi097 ceph-mon[31898]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T19:49:37.755 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:37 smithi097 ceph-mon[31898]: from='osd.0 [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:49:37.755 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:37 smithi097 ceph-mon[31898]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T19:49:37.755 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:37 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:37.755 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:37 smithi097 ceph-mon[31898]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:49:37.755 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:37 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:37.909 DEBUG:teuthology.orchestra.run.smithi097:osd.0> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T19:49:37.914 INFO:tasks.cephadm:Deploying osd.1 on smithi114 with /dev/vg_nvme/lv_4... 2024-03-11T19:49:37.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T19:49:38.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:37 smithi161 ceph-mon[36332]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T19:49:38.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:37 smithi161 ceph-mon[36332]: from='osd.0 [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:49:38.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:37 smithi161 ceph-mon[36332]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T19:49:38.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:37 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:38.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:37 smithi161 ceph-mon[36332]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:49:38.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:37 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:38.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:37 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:38.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:37 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:38.160 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:49:38.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:37 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:38.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:37 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:37 smithi114 ceph-mon[36376]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T19:49:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:37 smithi114 ceph-mon[36376]: from='osd.0 [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:49:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:37 smithi114 ceph-mon[36376]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T19:49:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:37 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:37 smithi114 ceph-mon[36376]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:49:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:37 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:37 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:37 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:38.765 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 19:49:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0[47090]: 2024-03-11T19:49:38.560+0000 7faea6000700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T19:49:38.765 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 19:49:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0[47090]: 2024-03-11T19:49:38.574+0000 7fae9be00700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:49:38.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:38 smithi114 ceph-mon[36376]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T19:49:38.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:38 smithi114 ceph-mon[36376]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T19:49:38.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:38 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:38.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:38 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:38.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:38 smithi114 ceph-mon[36376]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:38.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:38 smithi114 ceph-mon[36376]: from='osd.0 ' entity='osd.0' 2024-03-11T19:49:39.020 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:38 smithi097 ceph-mon[31898]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T19:49:39.021 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:38 smithi097 ceph-mon[31898]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T19:49:39.021 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:38 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:39.021 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:38 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:39.021 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:38 smithi097 ceph-mon[31898]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:39.021 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:38 smithi097 ceph-mon[31898]: from='osd.0 ' entity='osd.0' 2024-03-11T19:49:39.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:38 smithi161 ceph-mon[36332]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T19:49:39.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:38 smithi161 ceph-mon[36332]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T19:49:39.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:38 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:39.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:38 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:39.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:38 smithi161 ceph-mon[36332]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T19:49:39.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:38 smithi161 ceph-mon[36332]: from='osd.0 ' entity='osd.0' 2024-03-11T19:49:39.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:39 smithi097 ceph-mon[31898]: purged_snaps scrub starts 2024-03-11T19:49:39.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:39 smithi097 ceph-mon[31898]: purged_snaps scrub ok 2024-03-11T19:49:39.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:39.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:39 smithi097 ceph-mon[31898]: osd.0 [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520] boot 2024-03-11T19:49:39.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:39 smithi097 ceph-mon[31898]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T19:49:39.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:39.923 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T19:49:39.923 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T19:49:39.923 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-03-11T19:49:39.923 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-03-11T19:49:39.923 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122621 s, 855 MB/s 2024-03-11T19:49:39.923 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-03-11T19:49:40.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:39 smithi161 ceph-mon[36332]: purged_snaps scrub starts 2024-03-11T19:49:40.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:39 smithi161 ceph-mon[36332]: purged_snaps scrub ok 2024-03-11T19:49:40.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:40.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:39 smithi161 ceph-mon[36332]: osd.0 [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520] boot 2024-03-11T19:49:40.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:39 smithi161 ceph-mon[36332]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T19:49:40.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:39 smithi114 ceph-mon[36376]: purged_snaps scrub starts 2024-03-11T19:49:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:39 smithi114 ceph-mon[36376]: purged_snaps scrub ok 2024-03-11T19:49:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:39 smithi114 ceph-mon[36376]: osd.0 [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520] boot 2024-03-11T19:49:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:39 smithi114 ceph-mon[36376]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T19:49:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:49:40.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi114:vg_nvme/lv_4 2024-03-11T19:49:40.682 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:49:41.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:40 smithi161 ceph-mon[36332]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:41.083 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:40 smithi114 ceph-mon[36376]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:41.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:40 smithi097 ceph-mon[31898]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:42.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:41 smithi161 ceph-mon[36332]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T19:49:42.064 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:41 smithi114 ceph-mon[36376]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T19:49:42.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:41 smithi097 ceph-mon[31898]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T19:49:42.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.163+0000 7f89e0cb1700 1 -- 172.21.15.114:0/1672277393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89dc101da0 msgr2=0x7f89dc102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:49:42.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.163+0000 7f89e0cb1700 1 --2- 172.21.15.114:0/1672277393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89dc101da0 0x7f89dc102200 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f89d00099c0 tx=0x7f89d0009cd0 comp rx=0 tx=0).stop 2024-03-11T19:49:42.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.163+0000 7f89e0cb1700 1 -- 172.21.15.114:0/1672277393 shutdown_connections 2024-03-11T19:49:42.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.163+0000 7f89e0cb1700 1 --2- 172.21.15.114:0/1672277393 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89dc100ba0 0x7f89dc100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:42.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.163+0000 7f89e0cb1700 1 --2- 172.21.15.114:0/1672277393 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc102740 0x7f89dc108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:42.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.163+0000 7f89e0cb1700 1 --2- 172.21.15.114:0/1672277393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89dc101da0 0x7f89dc102200 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:42.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.163+0000 7f89e0cb1700 1 -- 172.21.15.114:0/1672277393 >> 172.21.15.114:0/1672277393 conn(0x7f89dc0fc160 msgr2=0x7f89dc0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:49:42.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.163+0000 7f89e0cb1700 1 -- 172.21.15.114:0/1672277393 shutdown_connections 2024-03-11T19:49:42.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.163+0000 7f89e0cb1700 1 -- 172.21.15.114:0/1672277393 wait complete. 2024-03-11T19:49:42.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.163+0000 7f89e0cb1700 1 Processor -- start 2024-03-11T19:49:42.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.163+0000 7f89e0cb1700 1 -- start start 2024-03-11T19:49:42.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89e0cb1700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc100ba0 0x7f89dc197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:49:42.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89e0cb1700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89dc101da0 0x7f89dc198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:49:42.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89e0cb1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89dc102740 0x7f89dc19f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:49:42.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89e0cb1700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89dc19f840 con 0x7f89dc101da0 2024-03-11T19:49:42.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89e0cb1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89dc19f980 con 0x7f89dc102740 2024-03-11T19:49:42.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89e0cb1700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89dc19fb40 con 0x7f89dc100ba0 2024-03-11T19:49:42.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89dad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89dc102740 0x7f89dc19f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:49:42.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89dad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89dc102740 0x7f89dc19f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50638/0 (socket says 172.21.15.114:50638) 2024-03-11T19:49:42.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89dad9d700 1 -- 172.21.15.114:0/4027774578 learned_addr learned my addr 172.21.15.114:0/4027774578 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-11T19:49:42.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89da59c700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc100ba0 0x7f89dc197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:49:42.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89dad9d700 1 -- 172.21.15.114:0/4027774578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc100ba0 msgr2=0x7f89dc197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:49:42.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89dad9d700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc100ba0 0x7f89dc197cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:42.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89dad9d700 1 -- 172.21.15.114:0/4027774578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89dc101da0 msgr2=0x7f89dc198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:49:42.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89dad9d700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89dc101da0 0x7f89dc198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:49:42.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89dad9d700 1 -- 172.21.15.114:0/4027774578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d00095f0 con 0x7f89dc102740 2024-03-11T19:49:42.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89da59c700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc100ba0 0x7f89dc197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:49:42.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89dad9d700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89dc102740 0x7f89dc19f270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f89d400ada0 tx=0x7f89d400c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:49:42.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89cb7fe700 1 -- 172.21.15.114:0/4027774578 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d400f070 con 0x7f89dc102740 2024-03-11T19:49:42.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89cb7fe700 1 -- 172.21.15.114:0/4027774578 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89d4007b30 con 0x7f89dc102740 2024-03-11T19:49:42.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89e0cb1700 1 -- 172.21.15.114:0/4027774578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89dc19fe20 con 0x7f89dc102740 2024-03-11T19:49:42.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89cb7fe700 1 -- 172.21.15.114:0/4027774578 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4005500 con 0x7f89dc102740 2024-03-11T19:49:42.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.164+0000 7f89e0cb1700 1 -- 172.21.15.114:0/4027774578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89dc1a0260 con 0x7f89dc102740 2024-03-11T19:49:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.165+0000 7f89e0cb1700 1 -- 172.21.15.114:0/4027774578 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89dc10c580 con 0x7f89dc102740 2024-03-11T19:49:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.166+0000 7f89cb7fe700 1 -- 172.21.15.114:0/4027774578 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7f89d4004ad0 con 0x7f89dc102740 2024-03-11T19:49:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.167+0000 7f89cb7fe700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f89c40a1a30 0x7f89c40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:49:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.167+0000 7f89cb7fe700 1 -- 172.21.15.114:0/4027774578 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f89d40cb070 con 0x7f89dc102740 2024-03-11T19:49:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.167+0000 7f89da59c700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f89c40a1a30 0x7f89c40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:49:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.168+0000 7f89da59c700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f89c40a1a30 0x7f89c40a3ef0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89dc1991d0 tx=0x7f89cc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:49:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.171+0000 7f89cb7fe700 1 -- 172.21.15.114:0/4027774578 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f89d4095d80 con 0x7f89dc102740 2024-03-11T19:49:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:49:42.373+0000 7f89e0cb1700 1 -- 172.21.15.114:0/4027774578 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f89dc0fe340 con 0x7f89c40a1a30 2024-03-11T19:49:42.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:49:42.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:49:42.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:49:42.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:42.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:42.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: Detected new or changed devices on smithi097 2024-03-11T19:49:42.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:42.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:42.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:49:42.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi097 to 17091M 2024-03-11T19:49:42.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:42.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:42.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:42.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:43.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:49:43.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:49:43.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:49:43.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:43.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:43.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: Detected new or changed devices on smithi097 2024-03-11T19:49:43.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:43.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:43.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:49:43.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi097 to 17091M 2024-03-11T19:49:43.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:43.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:43.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:43.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:43.074 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:49:43.075 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:49:43.075 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:49:43.075 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:43.075 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:43.075 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: Detected new or changed devices on smithi097 2024-03-11T19:49:43.075 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:43.075 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:43.075 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:49:43.075 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi097 to 17091M 2024-03-11T19:49:43.075 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:43.075 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:43.075 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:49:43.075 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:45 smithi114 ceph-mon[36376]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:45 smithi114 ceph-mon[36376]: from='client.? 172.21.15.114:0/2462433340' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "259f580e-e340-4048-a461-5da3946ed1d7"}]: dispatch 2024-03-11T19:49:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:45 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "259f580e-e340-4048-a461-5da3946ed1d7"}]: dispatch 2024-03-11T19:49:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:45 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "259f580e-e340-4048-a461-5da3946ed1d7"}]': finished 2024-03-11T19:49:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:45 smithi114 ceph-mon[36376]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T19:49:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:49:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:45 smithi114 ceph-mon[36376]: from='client.? 172.21.15.114:0/438928335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:49:45.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:45 smithi161 ceph-mon[36332]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:45.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:45 smithi161 ceph-mon[36332]: from='client.? 172.21.15.114:0/2462433340' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "259f580e-e340-4048-a461-5da3946ed1d7"}]: dispatch 2024-03-11T19:49:45.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:45 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "259f580e-e340-4048-a461-5da3946ed1d7"}]: dispatch 2024-03-11T19:49:45.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:45 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "259f580e-e340-4048-a461-5da3946ed1d7"}]': finished 2024-03-11T19:49:45.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:45 smithi161 ceph-mon[36332]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T19:49:45.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:49:45.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:45 smithi161 ceph-mon[36332]: from='client.? 172.21.15.114:0/438928335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:49:45.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:45 smithi097 ceph-mon[31898]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:45 smithi097 ceph-mon[31898]: from='client.? 172.21.15.114:0/2462433340' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "259f580e-e340-4048-a461-5da3946ed1d7"}]: dispatch 2024-03-11T19:49:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:45 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "259f580e-e340-4048-a461-5da3946ed1d7"}]: dispatch 2024-03-11T19:49:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:45 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "259f580e-e340-4048-a461-5da3946ed1d7"}]': finished 2024-03-11T19:49:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:45 smithi097 ceph-mon[31898]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T19:49:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:49:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:45 smithi097 ceph-mon[31898]: from='client.? 172.21.15.114:0/438928335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:49:47.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:47 smithi097 ceph-mon[31898]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:47.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:47 smithi114 ceph-mon[36376]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:47.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:47 smithi161 ceph-mon[36332]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:49.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:49 smithi114 ceph-mon[36376]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:49.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:49 smithi161 ceph-mon[36332]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:49.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:49 smithi097 ceph-mon[31898]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:51.732 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:51 smithi114 ceph-mon[36376]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:51.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:51 smithi161 ceph-mon[36332]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:51.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:51 smithi097 ceph-mon[31898]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:52.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:52 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T19:49:52.651 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:52 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:52.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:52 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T19:49:52.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:52 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:52.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:52 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T19:49:52.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:52 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:49:53.579 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:53 smithi114 ceph-mon[36376]: Deploying daemon osd.1 on smithi114 2024-03-11T19:49:53.579 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:53 smithi114 ceph-mon[36376]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:53.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:53 smithi161 ceph-mon[36332]: Deploying daemon osd.1 on smithi114 2024-03-11T19:49:53.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:53 smithi161 ceph-mon[36332]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:53.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:53 smithi097 ceph-mon[31898]: Deploying daemon osd.1 on smithi114 2024-03-11T19:49:53.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:53 smithi097 ceph-mon[31898]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:55.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:55 smithi114 ceph-mon[36376]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:55.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:55 smithi161 ceph-mon[36332]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:55.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:55 smithi097 ceph-mon[31898]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:57.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:57 smithi097 ceph-mon[31898]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:57.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:57.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:57.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:57.680 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:57 smithi114 ceph-mon[36376]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:57.680 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:57.680 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:57.680 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:57.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:57 smithi161 ceph-mon[36332]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:57.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:57.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:49:57.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:49:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:49:59 smithi114 ceph-mon[36376]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:59.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:49:59 smithi161 ceph-mon[36332]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:49:59.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:49:59 smithi097 ceph-mon[31898]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:50:00.163 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 1 on host 'smithi114' 2024-03-11T19:50:00.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:50:00.158+0000 7f89cb7fe700 1 -- 172.21.15.114:0/4027774578 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f89dc0fe340 con 0x7f89c40a1a30 2024-03-11T19:50:00.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:50:00.160+0000 7f89e0cb1700 1 -- 172.21.15.114:0/4027774578 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f89c40a1a30 msgr2=0x7f89c40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:50:00.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:50:00.160+0000 7f89e0cb1700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f89c40a1a30 0x7f89c40a3ef0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89dc1991d0 tx=0x7f89cc008040 comp rx=0 tx=0).stop 2024-03-11T19:50:00.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:50:00.160+0000 7f89e0cb1700 1 -- 172.21.15.114:0/4027774578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89dc102740 msgr2=0x7f89dc19f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:50:00.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:50:00.160+0000 7f89e0cb1700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89dc102740 0x7f89dc19f270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f89d400ada0 tx=0x7f89d400c330 comp rx=0 tx=0).stop 2024-03-11T19:50:00.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:50:00.160+0000 7f89e0cb1700 1 -- 172.21.15.114:0/4027774578 shutdown_connections 2024-03-11T19:50:00.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:50:00.160+0000 7f89e0cb1700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f89c40a1a30 0x7f89c40a3ef0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:00.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:50:00.160+0000 7f89e0cb1700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89dc102740 0x7f89dc19f270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:00.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:50:00.160+0000 7f89e0cb1700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89dc100ba0 0x7f89dc197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:00.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:50:00.160+0000 7f89e0cb1700 1 --2- 172.21.15.114:0/4027774578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f89dc101da0 0x7f89dc198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:00.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:50:00.160+0000 7f89e0cb1700 1 -- 172.21.15.114:0/4027774578 >> 172.21.15.114:0/4027774578 conn(0x7f89dc0fc160 msgr2=0x7f89dc0fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:50:00.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:50:00.160+0000 7f89e0cb1700 1 -- 172.21.15.114:0/4027774578 shutdown_connections 2024-03-11T19:50:00.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:50:00.160+0000 7f89e0cb1700 1 -- 172.21.15.114:0/4027774578 wait complete. 2024-03-11T19:50:00.750 DEBUG:teuthology.orchestra.run.smithi114:osd.1> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T19:50:00.752 INFO:tasks.cephadm:Deploying osd.2 on smithi161 with /dev/vg_nvme/lv_4... 2024-03-11T19:50:00.753 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T19:50:00.776 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.776 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.776 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:00.776 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:00.776 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.776 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:00 smithi114 ceph-mon[36376]: overall HEALTH_OK 2024-03-11T19:50:00.776 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.776 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.776 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:00.797 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.797 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.797 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:00.797 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:00.797 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.797 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:00 smithi161 ceph-mon[36332]: overall HEALTH_OK 2024-03-11T19:50:00.797 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.798 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.798 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:00.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:00.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:00.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:00 smithi097 ceph-mon[31898]: overall HEALTH_OK 2024-03-11T19:50:00.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:00.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:01.029 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:50:01.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 19:50:00 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1[40946]: 2024-03-11T19:50:00.858+0000 7f59c3b667c0 -1 osd.1 0 log_to_monitors true 2024-03-11T19:50:01.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:01 smithi161 ceph-mon[36332]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:50:01.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:01 smithi161 ceph-mon[36332]: from='osd.1 [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T19:50:01.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:01 smithi161 ceph-mon[36332]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T19:50:01.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:01 smithi097 ceph-mon[31898]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:50:01.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:01 smithi097 ceph-mon[31898]: from='osd.1 [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T19:50:01.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:01 smithi097 ceph-mon[31898]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T19:50:01.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:01 smithi114 ceph-mon[36376]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:50:01.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:01 smithi114 ceph-mon[36376]: from='osd.1 [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T19:50:01.922 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:01 smithi114 ceph-mon[36376]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T19:50:02.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:02 smithi161 ceph-mon[36332]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T19:50:02.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:02 smithi161 ceph-mon[36332]: from='osd.1 [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:50:02.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:02 smithi161 ceph-mon[36332]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T19:50:02.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:02 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:50:02.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:02 smithi161 ceph-mon[36332]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:50:02.896 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T19:50:02.896 INFO:teuthology.orchestra.run.smithi161.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T19:50:02.896 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10+0 records in 2024-03-11T19:50:02.896 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2024-03-11T19:50:02.896 INFO:teuthology.orchestra.run.smithi161.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124799 s, 840 MB/s 2024-03-11T19:50:02.897 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 2024-03-11T19:50:02.897 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping successful for: 2024-03-11T19:50:02.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:02 smithi097 ceph-mon[31898]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T19:50:02.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:02 smithi097 ceph-mon[31898]: from='osd.1 [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:50:02.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:02 smithi097 ceph-mon[31898]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T19:50:02.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:02 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:50:02.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:02 smithi097 ceph-mon[31898]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:50:02.993 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 19:50:02 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1[40946]: 2024-03-11T19:50:02.571+0000 7f59b91cb700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T19:50:02.993 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 19:50:02 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1[40946]: 2024-03-11T19:50:02.574+0000 7f59b4fbe700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:50:02.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:02 smithi114 ceph-mon[36376]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T19:50:02.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:02 smithi114 ceph-mon[36376]: from='osd.1 [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:50:02.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:02 smithi114 ceph-mon[36376]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T19:50:02.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:02 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:50:02.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:02 smithi114 ceph-mon[36376]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:50:03.531 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi161:vg_nvme/lv_4 2024-03-11T19:50:03.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:03 smithi114 ceph-mon[36376]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:50:03.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:03 smithi114 ceph-mon[36376]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-11T19:50:03.653 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:03 smithi114 ceph-mon[36376]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T19:50:03.654 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:03 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:50:03.783 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:50:03.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:03 smithi161 ceph-mon[36332]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:50:03.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:03 smithi161 ceph-mon[36332]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-11T19:50:03.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:03 smithi161 ceph-mon[36332]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T19:50:03.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:03 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:50:03.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:03 smithi097 ceph-mon[31898]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T19:50:03.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:03 smithi097 ceph-mon[31898]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-11T19:50:03.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:03 smithi097 ceph-mon[31898]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T19:50:03.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:50:04.832 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:04 smithi161 ceph-mon[36332]: purged_snaps scrub starts 2024-03-11T19:50:04.833 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:04 smithi161 ceph-mon[36332]: purged_snaps scrub ok 2024-03-11T19:50:04.833 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:04 smithi161 ceph-mon[36332]: osd.1 [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844] boot 2024-03-11T19:50:04.833 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:04 smithi161 ceph-mon[36332]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T19:50:04.833 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:04 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:50:04.833 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:04 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:04.833 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:04 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:04.833 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:04 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:50:04.833 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:04 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:04.833 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:04 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:04.833 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:04 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:04.833 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:04 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:04.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:04 smithi097 ceph-mon[31898]: purged_snaps scrub starts 2024-03-11T19:50:04.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:04 smithi097 ceph-mon[31898]: purged_snaps scrub ok 2024-03-11T19:50:04.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:04 smithi097 ceph-mon[31898]: osd.1 [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844] boot 2024-03-11T19:50:04.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:04 smithi097 ceph-mon[31898]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T19:50:04.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:04 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:50:04.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:04 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:04.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:04 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:04.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:04 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:50:04.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:04 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:04.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:04 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:04.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:04 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:04.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:04 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:04.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:04 smithi114 ceph-mon[36376]: purged_snaps scrub starts 2024-03-11T19:50:04.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:04 smithi114 ceph-mon[36376]: purged_snaps scrub ok 2024-03-11T19:50:04.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:04 smithi114 ceph-mon[36376]: osd.1 [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844] boot 2024-03-11T19:50:04.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:04 smithi114 ceph-mon[36376]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T19:50:04.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:04 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:50:04.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:04 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:04.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:04 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:04.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:04 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:50:04.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:04 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:04.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:04 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:04.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:04 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:04.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:04 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:05.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.334+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/2550991890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc4102e90 msgr2=0x7f8cc41032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:50:05.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.334+0000 7f8ccc7f2700 1 --2- 172.21.15.161:0/2550991890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc4102e90 0x7f8cc41032f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009990 tx=0x7f8cc0009ca0 comp rx=0 tx=0).stop 2024-03-11T19:50:05.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.334+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/2550991890 shutdown_connections 2024-03-11T19:50:05.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.334+0000 7f8ccc7f2700 1 --2- 172.21.15.161:0/2550991890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc4103830 0x7f8cc410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:05.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.334+0000 7f8ccc7f2700 1 --2- 172.21.15.161:0/2550991890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc4101c90 0x7f8cc41020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:05.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.334+0000 7f8ccc7f2700 1 --2- 172.21.15.161:0/2550991890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc4102e90 0x7f8cc41032f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:05.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.334+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/2550991890 >> 172.21.15.161:0/2550991890 conn(0x7f8cc40fd250 msgr2=0x7f8cc40ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:50:05.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.334+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/2550991890 shutdown_connections 2024-03-11T19:50:05.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.334+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/2550991890 wait complete. 2024-03-11T19:50:05.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.334+0000 7f8ccc7f2700 1 Processor -- start 2024-03-11T19:50:05.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.334+0000 7f8ccc7f2700 1 -- start start 2024-03-11T19:50:05.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.334+0000 7f8ccc7f2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc4101c90 0x7f8cc4197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:50:05.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccc7f2700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc4102e90 0x7f8cc41981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:50:05.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccc7f2700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc4103830 0x7f8cc419f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:50:05.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccc7f2700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc419f820 con 0x7f8cc4102e90 2024-03-11T19:50:05.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccc7f2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc419f960 con 0x7f8cc4101c90 2024-03-11T19:50:05.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccc7f2700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc419fb20 con 0x7f8cc4103830 2024-03-11T19:50:05.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8cca58e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc4101c90 0x7f8cc4197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:50:05.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccad8f700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc4103830 0x7f8cc419f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:50:05.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8cca58e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc4101c90 0x7f8cc4197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.161:41052/0 (socket says 172.21.15.161:41052) 2024-03-11T19:50:05.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8cca58e700 1 -- 172.21.15.161:0/3063889713 learned_addr learned my addr 172.21.15.161:0/3063889713 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-11T19:50:05.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccad8f700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc4103830 0x7f8cc419f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:59680/0 (socket says 172.21.15.161:59680) 2024-03-11T19:50:05.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccad8f700 1 -- 172.21.15.161:0/3063889713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc4101c90 msgr2=0x7f8cc4197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:50:05.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccad8f700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc4101c90 0x7f8cc4197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:05.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccad8f700 1 -- 172.21.15.161:0/3063889713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc4102e90 msgr2=0x7f8cc41981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:50:05.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8cc9d8d700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc4102e90 0x7f8cc41981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:50:05.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccad8f700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc4102e90 0x7f8cc41981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:05.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccad8f700 1 -- 172.21.15.161:0/3063889713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc00095f0 con 0x7f8cc4103830 2024-03-11T19:50:05.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8cca58e700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc4101c90 0x7f8cc4197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:50:05.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccad8f700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc4103830 0x7f8cc419f250 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00be20 tx=0x7f8cbc009ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:50:05.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8cbb7fe700 1 -- 172.21.15.161:0/3063889713 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc00cc10 con 0x7f8cc4103830 2024-03-11T19:50:05.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/3063889713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc419fe00 con 0x7f8cc4103830 2024-03-11T19:50:05.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8cbb7fe700 1 -- 172.21.15.161:0/3063889713 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8cbc00cd70 con 0x7f8cc4103830 2024-03-11T19:50:05.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.335+0000 7f8cbb7fe700 1 -- 172.21.15.161:0/3063889713 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc012730 con 0x7f8cc4103830 2024-03-11T19:50:05.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.336+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/3063889713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc41a02f0 con 0x7f8cc4103830 2024-03-11T19:50:05.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.337+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/3063889713 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc4109050 con 0x7f8cc4103830 2024-03-11T19:50:05.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.337+0000 7f8cbb7fe700 1 -- 172.21.15.161:0/3063889713 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7f8cbc014a90 con 0x7f8cc4103830 2024-03-11T19:50:05.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.338+0000 7f8cbb7fe700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8cb00a1ab0 0x7f8cb00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:50:05.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.338+0000 7f8cbb7fe700 1 -- 172.21.15.161:0/3063889713 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f8cbc006e40 con 0x7f8cc4103830 2024-03-11T19:50:05.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.339+0000 7f8cca58e700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8cb00a1ab0 0x7f8cb00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:50:05.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.339+0000 7f8cca58e700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8cb00a1ab0 0x7f8cb00a3f70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4198b80 tx=0x7f8cb400b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:50:05.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.343+0000 7f8cbb7fe700 1 -- 172.21.15.161:0/3063889713 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8cbc095e50 con 0x7f8cc4103830 2024-03-11T19:50:05.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:05.543+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/3063889713 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cc402d8a0 con 0x7f8cb00a1ab0 2024-03-11T19:50:05.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:05 smithi114 ceph-mon[36376]: Detected new or changed devices on smithi114 2024-03-11T19:50:05.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:05 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi114 to 17080M 2024-03-11T19:50:05.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:05 smithi114 ceph-mon[36376]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:05.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:05 smithi114 ceph-mon[36376]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T19:50:05.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:50:05.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:50:05.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:05 smithi097 ceph-mon[31898]: Detected new or changed devices on smithi114 2024-03-11T19:50:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:05 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi114 to 17080M 2024-03-11T19:50:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:05 smithi097 ceph-mon[31898]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:05 smithi097 ceph-mon[31898]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T19:50:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:50:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:50:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:06.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:05 smithi161 ceph-mon[36332]: Detected new or changed devices on smithi114 2024-03-11T19:50:06.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:05 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi114 to 17080M 2024-03-11T19:50:06.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:05 smithi161 ceph-mon[36332]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:06.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:05 smithi161 ceph-mon[36332]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T19:50:06.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:50:06.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:50:06.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:06.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:06 smithi161 ceph-mon[36332]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:50:06.908 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:06 smithi114 ceph-mon[36376]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:50:06.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:06 smithi097 ceph-mon[31898]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:50:07.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:07 smithi097 ceph-mon[31898]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:07.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:07 smithi114 ceph-mon[36376]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:08.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:07 smithi161 ceph-mon[36332]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:08.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:08 smithi161 ceph-mon[36332]: from='client.? 172.21.15.161:0/3877781142' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64012698-9df0-4d30-b4e4-1bebdbde0a0c"}]: dispatch 2024-03-11T19:50:08.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:08 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64012698-9df0-4d30-b4e4-1bebdbde0a0c"}]: dispatch 2024-03-11T19:50:08.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:08 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64012698-9df0-4d30-b4e4-1bebdbde0a0c"}]': finished 2024-03-11T19:50:08.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:08 smithi161 ceph-mon[36332]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T19:50:08.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:08 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:08.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:08 smithi097 ceph-mon[31898]: from='client.? 172.21.15.161:0/3877781142' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64012698-9df0-4d30-b4e4-1bebdbde0a0c"}]: dispatch 2024-03-11T19:50:08.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:08 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64012698-9df0-4d30-b4e4-1bebdbde0a0c"}]: dispatch 2024-03-11T19:50:08.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:08 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64012698-9df0-4d30-b4e4-1bebdbde0a0c"}]': finished 2024-03-11T19:50:08.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:08 smithi097 ceph-mon[31898]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T19:50:08.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:08 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:08.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:08 smithi114 ceph-mon[36376]: from='client.? 172.21.15.161:0/3877781142' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64012698-9df0-4d30-b4e4-1bebdbde0a0c"}]: dispatch 2024-03-11T19:50:08.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:08 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64012698-9df0-4d30-b4e4-1bebdbde0a0c"}]: dispatch 2024-03-11T19:50:08.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:08 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64012698-9df0-4d30-b4e4-1bebdbde0a0c"}]': finished 2024-03-11T19:50:08.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:08 smithi114 ceph-mon[36376]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T19:50:08.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:08 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:09.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:09 smithi097 ceph-mon[31898]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:09.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:09 smithi097 ceph-mon[31898]: from='client.? 172.21.15.161:0/3556789458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:50:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:09 smithi114 ceph-mon[36376]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:09 smithi114 ceph-mon[36376]: from='client.? 172.21.15.161:0/3556789458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:50:10.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:09 smithi161 ceph-mon[36332]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:10.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:09 smithi161 ceph-mon[36332]: from='client.? 172.21.15.161:0/3556789458' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:50:11.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:11 smithi097 ceph-mon[31898]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:11.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:11 smithi114 ceph-mon[36376]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:12.006 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:11 smithi161 ceph-mon[36332]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:13.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:13 smithi161 ceph-mon[36332]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:13.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:13 smithi097 ceph-mon[31898]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:13.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:13 smithi114 ceph-mon[36376]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:15.871 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:15 smithi161 ceph-mon[36332]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:15.871 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:15 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T19:50:15.871 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:15 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:15.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:15 smithi097 ceph-mon[31898]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:15.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:15 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T19:50:15.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:15 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:15.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:15 smithi114 ceph-mon[36376]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:15.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:15 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T19:50:15.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:15 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:16.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:16 smithi097 ceph-mon[31898]: Deploying daemon osd.2 on smithi161 2024-03-11T19:50:16.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:16 smithi161 ceph-mon[36332]: Deploying daemon osd.2 on smithi161 2024-03-11T19:50:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:16 smithi114 ceph-mon[36376]: Deploying daemon osd.2 on smithi161 2024-03-11T19:50:17.846 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:17 smithi161 ceph-mon[36332]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:17.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:17 smithi097 ceph-mon[31898]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:17 smithi114 ceph-mon[36376]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:19.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:19 smithi097 ceph-mon[31898]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:19.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:19 smithi114 ceph-mon[36376]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:20.029 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:19 smithi161 ceph-mon[36332]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:21.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:21 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:21.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:21 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:21.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:21 smithi097 ceph-mon[31898]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:21.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:21 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:21.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:21 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:21.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:21 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:21.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:21 smithi114 ceph-mon[36376]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:21.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:21 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:21.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:21 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:21.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:21 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:21.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:21 smithi161 ceph-mon[36332]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:21.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:21 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:23.614 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:23 smithi161 ceph-mon[36332]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:23.614 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:23.614 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:23.614 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:23.614 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:23.614 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:23.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:23.673+0000 7f8cbb7fe700 1 -- 172.21.15.161:0/3063889713 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8cc402d8a0 con 0x7f8cb00a1ab0 2024-03-11T19:50:23.678 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 2 on host 'smithi161' 2024-03-11T19:50:23.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:23.676+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/3063889713 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8cb00a1ab0 msgr2=0x7f8cb00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:50:23.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:23.676+0000 7f8ccc7f2700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8cb00a1ab0 0x7f8cb00a3f70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4198b80 tx=0x7f8cb400b4a0 comp rx=0 tx=0).stop 2024-03-11T19:50:23.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:23.676+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/3063889713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc4103830 msgr2=0x7f8cc419f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:50:23.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:23.676+0000 7f8ccc7f2700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc4103830 0x7f8cc419f250 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc00be20 tx=0x7f8cbc009ed0 comp rx=0 tx=0).stop 2024-03-11T19:50:23.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:23.676+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/3063889713 shutdown_connections 2024-03-11T19:50:23.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:23.676+0000 7f8ccc7f2700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8cb00a1ab0 0x7f8cb00a3f70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:23.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:23.676+0000 7f8ccc7f2700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc4103830 0x7f8cc419f250 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:23.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:23.676+0000 7f8ccc7f2700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc4101c90 0x7f8cc4197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:23.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:23.676+0000 7f8ccc7f2700 1 --2- 172.21.15.161:0/3063889713 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc4102e90 0x7f8cc41981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:23.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:23.676+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/3063889713 >> 172.21.15.161:0/3063889713 conn(0x7f8cc40fd250 msgr2=0x7f8cc4106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:50:23.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:23.676+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/3063889713 shutdown_connections 2024-03-11T19:50:23.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:50:23.676+0000 7f8ccc7f2700 1 -- 172.21.15.161:0/3063889713 wait complete. 2024-03-11T19:50:23.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:23 smithi114 ceph-mon[36376]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:23.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:23.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:23.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:23.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:23.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:23 smithi097 ceph-mon[31898]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:24.272 DEBUG:teuthology.orchestra.run.smithi161:osd.2> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T19:50:24.274 INFO:tasks.cephadm:Deploying osd.3 on smithi097 with /dev/vg_nvme/lv_3... 2024-03-11T19:50:24.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T19:50:24.717 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:50:24.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:24 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:24.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:24 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:24.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:24 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:24.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:24 smithi161 ceph-mon[36332]: from='osd.2 [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T19:50:24.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:24 smithi161 ceph-mon[36332]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T19:50:24.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:24 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:24.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:24 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:24.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:24 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:24.908 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:24 smithi097 ceph-mon[31898]: from='osd.2 [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T19:50:24.908 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:24 smithi097 ceph-mon[31898]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T19:50:24.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:24 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:24.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:24 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:24.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:24 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:24.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:24 smithi114 ceph-mon[36376]: from='osd.2 [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T19:50:24.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:24 smithi114 ceph-mon[36376]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T19:50:25.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:25 smithi114 ceph-mon[36376]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:25 smithi114 ceph-mon[36376]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T19:50:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:25 smithi114 ceph-mon[36376]: from='osd.2 [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:50:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:25 smithi114 ceph-mon[36376]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T19:50:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:25 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:25 smithi114 ceph-mon[36376]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:50:26.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:25 smithi161 ceph-mon[36332]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:26.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:25 smithi161 ceph-mon[36332]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T19:50:26.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:25 smithi161 ceph-mon[36332]: from='osd.2 [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:50:26.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:25 smithi161 ceph-mon[36332]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T19:50:26.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:25 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:26.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:25 smithi161 ceph-mon[36332]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:50:26.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:25 smithi097 ceph-mon[31898]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:26.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:25 smithi097 ceph-mon[31898]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T19:50:26.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:25 smithi097 ceph-mon[31898]: from='osd.2 [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:50:26.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:25 smithi097 ceph-mon[31898]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T19:50:26.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:25 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:26.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:25 smithi097 ceph-mon[31898]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:50:26.690 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 19:50:26 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2[41453]: 2024-03-11T19:50:26.431+0000 7f81e0f15700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T19:50:26.691 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 19:50:26 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2[41453]: 2024-03-11T19:50:26.436+0000 7f81d84ff700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:50:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:26 smithi114 ceph-mon[36376]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-03-11T19:50:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:26 smithi114 ceph-mon[36376]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T19:50:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:26 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:26 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:26 smithi114 ceph-mon[36376]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:26 smithi114 ceph-mon[36376]: from='osd.2 ' entity='osd.2' 2024-03-11T19:50:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:26 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:26.995 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:26 smithi161 ceph-mon[36332]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-03-11T19:50:26.995 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:26 smithi161 ceph-mon[36332]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T19:50:26.996 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:26 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:26.996 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:26 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:26.996 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:26 smithi161 ceph-mon[36332]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:26.996 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:26 smithi161 ceph-mon[36332]: from='osd.2 ' entity='osd.2' 2024-03-11T19:50:26.996 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:26 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:27.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:26 smithi097 ceph-mon[31898]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-03-11T19:50:27.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:26 smithi097 ceph-mon[31898]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T19:50:27.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:26 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:27.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:26 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:27.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:26 smithi097 ceph-mon[31898]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T19:50:27.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:26 smithi097 ceph-mon[31898]: from='osd.2 ' entity='osd.2' 2024-03-11T19:50:27.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:26 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:27.570 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T19:50:27.570 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T19:50:27.570 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-03-11T19:50:27.570 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-03-11T19:50:27.570 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0312208 s, 336 MB/s 2024-03-11T19:50:27.570 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-03-11T19:50:27.704 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: purged_snaps scrub starts 2024-03-11T19:50:27.704 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: purged_snaps scrub ok 2024-03-11T19:50:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: purged_snaps scrub starts 2024-03-11T19:50:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: purged_snaps scrub ok 2024-03-11T19:50:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: Detected new or changed devices on smithi161 2024-03-11T19:50:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:50:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi161 to 17080M 2024-03-11T19:50:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:27.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:27.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:27.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: osd.2 [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993] boot 2024-03-11T19:50:27.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T19:50:27.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:27 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:28.010 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: Detected new or changed devices on smithi161 2024-03-11T19:50:28.010 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:28.010 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:28.010 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:50:28.010 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi161 to 17080M 2024-03-11T19:50:28.010 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:28.010 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:28.010 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:28.010 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:28.010 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: osd.2 [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993] boot 2024-03-11T19:50:28.010 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T19:50:28.010 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:27 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:28.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: purged_snaps scrub starts 2024-03-11T19:50:28.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: purged_snaps scrub ok 2024-03-11T19:50:28.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: Detected new or changed devices on smithi161 2024-03-11T19:50:28.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:28.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:28.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:50:28.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi161 to 17080M 2024-03-11T19:50:28.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:28.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:28.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:28.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:28.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: osd.2 [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993] boot 2024-03-11T19:50:28.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T19:50:28.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:27 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:50:28.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi097:vg_nvme/lv_3 2024-03-11T19:50:28.859 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:50:29.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:29 smithi161 ceph-mon[36332]: pgmap v90: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:29.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:50:29.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T19:50:29.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:29 smithi161 ceph-mon[36332]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T19:50:29.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:50:29.308 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:29 smithi097 ceph-mon[31898]: pgmap v90: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:29.308 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:50:29.308 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T19:50:29.308 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:29 smithi097 ceph-mon[31898]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T19:50:29.308 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:50:29.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:29 smithi114 ceph-mon[36376]: pgmap v90: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:29.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:50:29.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T19:50:29.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:29 smithi114 ceph-mon[36376]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T19:50:29.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T19:50:30.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.945+0000 7fae38600700 1 -- 172.21.15.97:0/30041036 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae30105090 msgr2=0x7fae30105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:50:30.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.945+0000 7fae38600700 1 --2- 172.21.15.97:0/30041036 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae30105090 0x7fae30105510 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fae1c009990 tx=0x7fae1c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:50:30.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.946+0000 7fae38600700 1 -- 172.21.15.97:0/30041036 shutdown_connections 2024-03-11T19:50:30.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.946+0000 7fae38600700 1 --2- 172.21.15.97:0/30041036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae30103ea0 0x7fae301042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:30.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.946+0000 7fae38600700 1 --2- 172.21.15.97:0/30041036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae30105a50 0x7fae3010c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:30.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.946+0000 7fae38600700 1 --2- 172.21.15.97:0/30041036 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae30105090 0x7fae30105510 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:30.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.946+0000 7fae38600700 1 -- 172.21.15.97:0/30041036 >> 172.21.15.97:0/30041036 conn(0x7fae300ff490 msgr2=0x7fae301018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:50:30.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.946+0000 7fae38600700 1 -- 172.21.15.97:0/30041036 shutdown_connections 2024-03-11T19:50:30.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.946+0000 7fae38600700 1 -- 172.21.15.97:0/30041036 wait complete. 2024-03-11T19:50:30.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.947+0000 7fae38600700 1 Processor -- start 2024-03-11T19:50:30.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.947+0000 7fae38600700 1 -- start start 2024-03-11T19:50:30.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.948+0000 7fae38600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae30103ea0 0x7fae30199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:50:30.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.948+0000 7fae38600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae30105090 0x7fae3019a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:50:30.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.948+0000 7fae38600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae30105a50 0x7fae301a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:50:30.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.948+0000 7fae38600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae301a1980 con 0x7fae30103ea0 2024-03-11T19:50:30.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.948+0000 7fae38600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae301a1ac0 con 0x7fae30105090 2024-03-11T19:50:30.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.948+0000 7fae38600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae301a1c50 con 0x7fae30105a50 2024-03-11T19:50:30.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.948+0000 7fae36600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae30105a50 0x7fae301a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:50:30.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.948+0000 7fae35c00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae30103ea0 0x7fae30199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:50:30.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.948+0000 7fae36600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae30105a50 0x7fae301a13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:60876/0 (socket says 172.21.15.97:60876) 2024-03-11T19:50:30.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.948+0000 7fae36600700 1 -- 172.21.15.97:0/1997690815 learned_addr learned my addr 172.21.15.97:0/1997690815 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:50:30.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.948+0000 7fae35200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae30105090 0x7fae3019a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:50:30.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.949+0000 7fae36600700 1 -- 172.21.15.97:0/1997690815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae30105090 msgr2=0x7fae3019a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:50:30.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.949+0000 7fae36600700 1 --2- 172.21.15.97:0/1997690815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae30105090 0x7fae3019a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:30.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.949+0000 7fae36600700 1 -- 172.21.15.97:0/1997690815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae30103ea0 msgr2=0x7fae30199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:50:30.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.949+0000 7fae36600700 1 --2- 172.21.15.97:0/1997690815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae30103ea0 0x7fae30199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:30.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.949+0000 7fae36600700 1 -- 172.21.15.97:0/1997690815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae28009520 con 0x7fae30105a50 2024-03-11T19:50:30.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.949+0000 7fae35200700 1 --2- 172.21.15.97:0/1997690815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae30105090 0x7fae3019a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:50:30.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.949+0000 7fae36600700 1 --2- 172.21.15.97:0/1997690815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae30105a50 0x7fae301a13b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fae2800eab0 tx=0x7fae2800edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:50:30.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.949+0000 7fae2e000700 1 -- 172.21.15.97:0/1997690815 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae280098a0 con 0x7fae30105a50 2024-03-11T19:50:30.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.949+0000 7fae2e000700 1 -- 172.21.15.97:0/1997690815 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae2800ce30 con 0x7fae30105a50 2024-03-11T19:50:30.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.949+0000 7fae2e000700 1 -- 172.21.15.97:0/1997690815 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae28003e00 con 0x7fae30105a50 2024-03-11T19:50:30.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.949+0000 7fae38600700 1 -- 172.21.15.97:0/1997690815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1c0095f0 con 0x7fae30105a50 2024-03-11T19:50:30.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.950+0000 7fae38600700 1 -- 172.21.15.97:0/1997690815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae301a21f0 con 0x7fae30105a50 2024-03-11T19:50:30.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.951+0000 7fae38600700 1 -- 172.21.15.97:0/1997690815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae30068930 con 0x7fae30105a50 2024-03-11T19:50:30.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.953+0000 7fae2e000700 1 -- 172.21.15.97:0/1997690815 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7fae2800c950 con 0x7fae30105a50 2024-03-11T19:50:30.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.954+0000 7fae2e000700 1 --2- 172.21.15.97:0/1997690815 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fae180a1a30 0x7fae180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:50:30.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.954+0000 7fae2e000700 1 -- 172.21.15.97:0/1997690815 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3237+0+0 (secure 0 0 0) 0x7fae28014070 con 0x7fae30105a50 2024-03-11T19:50:30.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.954+0000 7fae35c00700 1 --2- 172.21.15.97:0/1997690815 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fae180a1a30 0x7fae180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:50:30.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.955+0000 7fae35c00700 1 --2- 172.21.15.97:0/1997690815 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fae180a1a30 0x7fae180a3ef0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fae30104c00 tx=0x7fae2400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:50:30.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:30.958+0000 7fae2e000700 1 -- 172.21.15.97:0/1997690815 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fae28096830 con 0x7fae30105a50 2024-03-11T19:50:31.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:31.207+0000 7fae38600700 1 -- 172.21.15.97:0/1997690815 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fae3002d990 con 0x7fae180a1a30 2024-03-11T19:50:31.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:31 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T19:50:31.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:31 smithi097 ceph-mon[31898]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T19:50:31.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:31 smithi097 ceph-mon[31898]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:31.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:31 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T19:50:31.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:31 smithi114 ceph-mon[36376]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T19:50:31.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:31 smithi114 ceph-mon[36376]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:31.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:31 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T19:50:31.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:31 smithi161 ceph-mon[36332]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T19:50:31.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:31 smithi161 ceph-mon[36332]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:32.080 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 19:50:31 smithi097 sudo[51413]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T19:50:32.080 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 19:50:31 smithi097 sudo[51413]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:50:32.080 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 19:50:31 smithi097 sudo[51413]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:50:32.081 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 19:50:31 smithi097 sudo[51413]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:50:32.352 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:32 smithi097 ceph-mon[31898]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:50:32.353 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:32 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:50:32.353 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:32 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:50:32.353 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:32 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:32.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:32 smithi114 ceph-mon[36376]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:50:32.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:32 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:50:32.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:32 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:50:32.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:32 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:32.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:32 smithi161 ceph-mon[36332]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:50:32.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:32 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:50:32.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:32 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:50:32.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:32 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:32.672 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 19:50:32 smithi097 sudo[51473]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T19:50:32.672 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 19:50:32 smithi097 sudo[51473]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:50:32.672 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 19:50:32 smithi097 sudo[51473]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:50:32.672 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 19:50:32 smithi097 sudo[51473]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:50:32.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 19:50:32 smithi114 sudo[44957]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T19:50:32.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 19:50:32 smithi114 sudo[44957]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:50:32.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 19:50:32 smithi114 sudo[44957]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:50:32.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 19:50:32 smithi114 sudo[44957]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:50:33.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 19:50:32 smithi114 sudo[44960]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T19:50:33.244 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 19:50:32 smithi114 sudo[44960]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:50:33.244 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 19:50:32 smithi114 sudo[44960]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:50:33.244 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 19:50:32 smithi114 sudo[44960]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:50:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:33 smithi114 ceph-mon[36376]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T19:50:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:33 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:33 smithi114 ceph-mon[36376]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:33 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:33.278 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:33 smithi097 ceph-mon[31898]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T19:50:33.278 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:33 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:33.278 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:33 smithi097 ceph-mon[31898]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:33.278 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:33 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:33.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:33.408+0000 7fae2e000700 1 -- 172.21.15.97:0/1997690815 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fae28096c00 con 0x7fae30105a50 2024-03-11T19:50:33.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:33 smithi161 ceph-mon[36332]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T19:50:33.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:33 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:33.450 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:33 smithi161 ceph-mon[36332]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:33.451 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:33 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:33.451 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 19:50:33 smithi161 sudo[45556]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T19:50:33.451 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 19:50:33 smithi161 sudo[45556]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:50:33.451 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 19:50:33 smithi161 sudo[45556]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:50:33.451 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 19:50:33 smithi161 sudo[45556]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:50:33.790 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 19:50:33 smithi161 sudo[45559]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T19:50:33.790 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 19:50:33 smithi161 sudo[45559]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:50:33.790 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 19:50:33 smithi161 sudo[45559]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:50:33.790 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 19:50:33 smithi161 sudo[45559]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:50:34.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:34 smithi097 sudo[51520]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T19:50:34.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:34 smithi097 sudo[51520]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:50:34.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:34 smithi097 sudo[51520]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:50:34.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:34 smithi097 sudo[51520]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:50:34.881 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:34 smithi161 ceph-mon[36332]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-11T19:50:34.881 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:34 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:34.881 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:34 smithi161 ceph-mon[36332]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:50:34.882 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:34 smithi161 sudo[45562]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T19:50:34.882 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:34 smithi161 sudo[45562]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:50:34.882 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:34 smithi161 sudo[45562]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:50:34.882 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:34 smithi161 sudo[45562]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:50:34.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:34 smithi097 sudo[51542]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T19:50:34.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:34 smithi097 sudo[51542]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:50:34.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:34 smithi097 sudo[51542]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:50:34.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:34 smithi097 sudo[51542]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:50:34.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:34 smithi097 ceph-mon[31898]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-11T19:50:34.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:34 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:34.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:34 smithi097 ceph-mon[31898]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:50:34.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:34 smithi114 ceph-mon[36376]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-11T19:50:34.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:34 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:34.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:34 smithi114 ceph-mon[36376]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:50:35.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 sudo[45565]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T19:50:35.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 sudo[45565]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:50:35.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 sudo[45565]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:50:35.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 sudo[45565]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:50:35.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 sudo[44963]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T19:50:35.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 sudo[44963]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:50:35.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 sudo[44963]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:50:35.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 sudo[44963]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:50:35.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:35.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:50:35.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:50:35.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:50:35.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:50:35.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:50:35.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:50:35.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:50:35.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:50:35.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:50:35.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:50:35.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:50:35.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:50:35.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:50:35.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1308208059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "794e7a63-96da-4d6e-b020-29d61db19359"}]: dispatch 2024-03-11T19:50:35.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "794e7a63-96da-4d6e-b020-29d61db19359"}]: dispatch 2024-03-11T19:50:35.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:35 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "794e7a63-96da-4d6e-b020-29d61db19359"}]': finished 2024-03-11T19:50:35.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 sudo[44966]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T19:50:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 sudo[44966]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T19:50:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 sudo[44966]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T19:50:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 sudo[44966]: pam_unix(sudo:session): session closed for user root 2024-03-11T19:50:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:50:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:50:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:50:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:50:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:50:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:50:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:50:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:50:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:50:35.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:50:35.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:50:35.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:50:35.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:50:35.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1308208059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "794e7a63-96da-4d6e-b020-29d61db19359"}]: dispatch 2024-03-11T19:50:35.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "794e7a63-96da-4d6e-b020-29d61db19359"}]: dispatch 2024-03-11T19:50:35.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:35 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "794e7a63-96da-4d6e-b020-29d61db19359"}]': finished 2024-03-11T19:50:36.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:36.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:50:36.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:50:36.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:50:36.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:50:36.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:50:36.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:50:36.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:50:36.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:50:36.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:50:36.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T19:50:36.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:50:36.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:50:36.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:50:36.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1308208059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "794e7a63-96da-4d6e-b020-29d61db19359"}]: dispatch 2024-03-11T19:50:36.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "794e7a63-96da-4d6e-b020-29d61db19359"}]: dispatch 2024-03-11T19:50:36.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:35 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "794e7a63-96da-4d6e-b020-29d61db19359"}]': finished 2024-03-11T19:50:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:36 smithi097 ceph-mon[31898]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:50:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:36 smithi097 ceph-mon[31898]: osdmap e22: 4 total, 3 up, 4 in 2024-03-11T19:50:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:36 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:50:36.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:36 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/797596693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:50:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:36 smithi114 ceph-mon[36376]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:50:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:36 smithi114 ceph-mon[36376]: osdmap e22: 4 total, 3 up, 4 in 2024-03-11T19:50:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:36 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:50:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:36 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/797596693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:50:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:36 smithi161 ceph-mon[36332]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T19:50:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:36 smithi161 ceph-mon[36332]: osdmap e22: 4 total, 3 up, 4 in 2024-03-11T19:50:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:36 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:50:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:36 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/797596693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:50:37.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:37 smithi097 ceph-mon[31898]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:37.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:37 smithi114 ceph-mon[36376]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:38.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:37 smithi161 ceph-mon[36332]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:39.907 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:39 smithi161 ceph-mon[36332]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:39.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:39 smithi097 ceph-mon[31898]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:39.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:39 smithi114 ceph-mon[36376]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:41.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:41 smithi097 ceph-mon[31898]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-11T19:50:41.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:41 smithi114 ceph-mon[36376]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-11T19:50:42.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:41 smithi161 ceph-mon[36332]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-11T19:50:43.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:43 smithi097 ceph-mon[31898]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-11T19:50:43.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:43 smithi114 ceph-mon[36376]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-11T19:50:44.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:43 smithi161 ceph-mon[36332]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-11T19:50:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:45 smithi097 ceph-mon[31898]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-11T19:50:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T19:50:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:45 smithi114 ceph-mon[36376]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-11T19:50:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T19:50:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:46.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:45 smithi161 ceph-mon[36332]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-11T19:50:46.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T19:50:46.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:46.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:46 smithi097 ceph-mon[31898]: Deploying daemon osd.3 on smithi097 2024-03-11T19:50:46.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:46 smithi114 ceph-mon[36376]: Deploying daemon osd.3 on smithi097 2024-03-11T19:50:47.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:46 smithi161 ceph-mon[36332]: Deploying daemon osd.3 on smithi097 2024-03-11T19:50:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:47 smithi114 ceph-mon[36376]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 949 B/s rd, 24 KiB/s wr, 2 op/s 2024-03-11T19:50:48.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:47 smithi161 ceph-mon[36332]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 949 B/s rd, 24 KiB/s wr, 2 op/s 2024-03-11T19:50:48.045 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:47 smithi097 ceph-mon[31898]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 949 B/s rd, 24 KiB/s wr, 2 op/s 2024-03-11T19:50:49.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:49 smithi097 ceph-mon[31898]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-11T19:50:49.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:49 smithi114 ceph-mon[36376]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-11T19:50:50.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:49 smithi161 ceph-mon[36332]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-11T19:50:51.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:51 smithi114 ceph-mon[36376]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-11T19:50:52.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:51 smithi161 ceph-mon[36332]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-11T19:50:52.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:51 smithi097 ceph-mon[31898]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-11T19:50:53.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:53 smithi097 ceph-mon[31898]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:53.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:53.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:53.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:53.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:53 smithi114 ceph-mon[36376]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:53.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:53.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:53.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:54.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:53 smithi161 ceph-mon[36332]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:55.010 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:54 smithi097 ceph-mon[31898]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:55.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:54 smithi161 ceph-mon[36332]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:55.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:54 smithi114 ceph-mon[36376]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:57.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:57 smithi097 ceph-mon[31898]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:57.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:57 smithi097 ceph-mon[31898]: from='osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T19:50:57.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:57.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:57.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:57.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:57.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:57.743 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:57 smithi161 ceph-mon[36332]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:57.743 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:57 smithi161 ceph-mon[36332]: from='osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T19:50:57.743 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:57.743 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:57.744 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:57.744 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:57.744 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:57 smithi114 ceph-mon[36376]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:57 smithi114 ceph-mon[36376]: from='osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T19:50:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:50:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:50:57.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:58.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:58.299+0000 7fae2e000700 1 -- 172.21.15.97:0/1997690815 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fae3002d990 con 0x7fae180a1a30 2024-03-11T19:50:58.304 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 3 on host 'smithi097' 2024-03-11T19:50:58.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:58.302+0000 7fae38600700 1 -- 172.21.15.97:0/1997690815 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fae180a1a30 msgr2=0x7fae180a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:50:58.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:58.302+0000 7fae38600700 1 --2- 172.21.15.97:0/1997690815 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fae180a1a30 0x7fae180a3ef0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fae30104c00 tx=0x7fae2400b040 comp rx=0 tx=0).stop 2024-03-11T19:50:58.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:58.302+0000 7fae38600700 1 -- 172.21.15.97:0/1997690815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae30105a50 msgr2=0x7fae301a13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:50:58.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:58.302+0000 7fae38600700 1 --2- 172.21.15.97:0/1997690815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae30105a50 0x7fae301a13b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fae2800eab0 tx=0x7fae2800edc0 comp rx=0 tx=0).stop 2024-03-11T19:50:58.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:58.303+0000 7fae38600700 1 -- 172.21.15.97:0/1997690815 shutdown_connections 2024-03-11T19:50:58.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:58.303+0000 7fae38600700 1 --2- 172.21.15.97:0/1997690815 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fae180a1a30 0x7fae180a3ef0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:58.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:58.303+0000 7fae38600700 1 --2- 172.21.15.97:0/1997690815 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae30103ea0 0x7fae30199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:58.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:58.303+0000 7fae38600700 1 --2- 172.21.15.97:0/1997690815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae30105a50 0x7fae301a13b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:58.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:58.303+0000 7fae38600700 1 --2- 172.21.15.97:0/1997690815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae30105090 0x7fae3019a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:50:58.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:58.303+0000 7fae38600700 1 -- 172.21.15.97:0/1997690815 >> 172.21.15.97:0/1997690815 conn(0x7fae300ff490 msgr2=0x7fae301083c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:50:58.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:58.303+0000 7fae38600700 1 -- 172.21.15.97:0/1997690815 shutdown_connections 2024-03-11T19:50:58.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:50:58.303+0000 7fae38600700 1 -- 172.21.15.97:0/1997690815 wait complete. 2024-03-11T19:50:58.742 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:58 smithi097 ceph-mon[31898]: from='osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T19:50:58.742 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:58 smithi097 ceph-mon[31898]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T19:50:58.742 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:58 smithi097 ceph-mon[31898]: from='osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:50:58.742 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:58 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:50:58.742 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:58 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:58.742 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:58 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:58.742 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:58 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:58.742 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:58 smithi097 ceph-mon[31898]: from='osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T19:50:58.743 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:58 smithi097 ceph-mon[31898]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T19:50:58.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:58 smithi114 ceph-mon[36376]: from='osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T19:50:58.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:58 smithi114 ceph-mon[36376]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T19:50:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:58 smithi114 ceph-mon[36376]: from='osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:50:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:58 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:50:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:58 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:58 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:58 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:58 smithi114 ceph-mon[36376]: from='osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T19:50:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:58 smithi114 ceph-mon[36376]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T19:50:58.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:58 smithi161 ceph-mon[36332]: from='osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T19:50:58.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:58 smithi161 ceph-mon[36332]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T19:50:58.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:58 smithi161 ceph-mon[36332]: from='osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:50:58.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:58 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:50:58.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:58 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:58.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:58 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:50:58.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:58 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:50:58.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:58 smithi161 ceph-mon[36332]: from='osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T19:50:58.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:58 smithi161 ceph-mon[36332]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T19:50:59.188 DEBUG:teuthology.orchestra.run.smithi097:osd.3> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.3.service 2024-03-11T19:50:59.198 INFO:tasks.cephadm:Deploying osd.4 on smithi114 with /dev/vg_nvme/lv_3... 2024-03-11T19:50:59.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T19:50:59.443 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:50:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:59 smithi114 ceph-mon[36376]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:59 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:50:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:59 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:50:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:50:59 smithi114 ceph-mon[36376]: osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214] boot 2024-03-11T19:50:59.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:59 smithi161 ceph-mon[36332]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:59.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:59 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:50:59.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:59 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:50:59.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:50:59 smithi161 ceph-mon[36332]: osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214] boot 2024-03-11T19:50:59.836 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:59 smithi097 ceph-mon[31898]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T19:50:59.836 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:59 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:50:59.836 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:59 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:50:59.837 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:50:59 smithi097 ceph-mon[31898]: osd.3 [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214] boot 2024-03-11T19:51:00.540 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:00 smithi114 ceph-mon[36376]: purged_snaps scrub starts 2024-03-11T19:51:00.540 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:00 smithi114 ceph-mon[36376]: purged_snaps scrub ok 2024-03-11T19:51:00.540 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:00 smithi114 ceph-mon[36376]: osdmap e25: 4 total, 4 up, 4 in 2024-03-11T19:51:00.540 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:51:00.540 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:00 smithi114 ceph-mon[36376]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T19:51:00.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:00 smithi097 ceph-mon[31898]: purged_snaps scrub starts 2024-03-11T19:51:00.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:00 smithi097 ceph-mon[31898]: purged_snaps scrub ok 2024-03-11T19:51:00.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:00 smithi097 ceph-mon[31898]: osdmap e25: 4 total, 4 up, 4 in 2024-03-11T19:51:00.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:51:00.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:00 smithi097 ceph-mon[31898]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T19:51:00.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:00 smithi161 ceph-mon[36332]: purged_snaps scrub starts 2024-03-11T19:51:00.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:00 smithi161 ceph-mon[36332]: purged_snaps scrub ok 2024-03-11T19:51:00.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:00 smithi161 ceph-mon[36332]: osdmap e25: 4 total, 4 up, 4 in 2024-03-11T19:51:00.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:51:00.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:00 smithi161 ceph-mon[36332]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T19:51:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T19:51:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T19:51:01.408 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-03-11T19:51:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-03-11T19:51:01.408 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012312 s, 852 MB/s 2024-03-11T19:51:01.408 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-03-11T19:51:01.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:01 smithi161 ceph-mon[36332]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:01.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:01 smithi161 ceph-mon[36332]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T19:51:01.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:01 smithi097 ceph-mon[31898]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:01.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:01 smithi097 ceph-mon[31898]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T19:51:01.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:01 smithi114 ceph-mon[36376]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:01.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:01 smithi114 ceph-mon[36376]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T19:51:02.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi114:vg_nvme/lv_3 2024-03-11T19:51:02.311 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:51:03.726 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:03 smithi114 ceph-mon[36376]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:03.726 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:03 smithi114 ceph-mon[36376]: osdmap e28: 4 total, 4 up, 4 in 2024-03-11T19:51:03.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:03 smithi161 ceph-mon[36332]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:03.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:03 smithi161 ceph-mon[36332]: osdmap e28: 4 total, 4 up, 4 in 2024-03-11T19:51:03.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:03 smithi097 ceph-mon[31898]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:03.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:03 smithi097 ceph-mon[31898]: osdmap e28: 4 total, 4 up, 4 in 2024-03-11T19:51:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.963+0000 7f395343d700 1 -- 172.21.15.114:0/4271595226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f394c1079f0 msgr2=0x7f394c109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:51:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.963+0000 7f395343d700 1 --2- 172.21.15.114:0/4271595226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f394c1079f0 0x7f394c109e00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f394400b0a0 tx=0x7f394400b3b0 comp rx=0 tx=0).stop 2024-03-11T19:51:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.963+0000 7f395343d700 1 -- 172.21.15.114:0/4271595226 shutdown_connections 2024-03-11T19:51:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.963+0000 7f395343d700 1 --2- 172.21.15.114:0/4271595226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f394c1079f0 0x7f394c109e00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.963+0000 7f395343d700 1 --2- 172.21.15.114:0/4271595226 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f394c102790 0x7f394c104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.963+0000 7f395343d700 1 --2- 172.21.15.114:0/4271595226 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f394c1050c0 0x7f394c1074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.963+0000 7f395343d700 1 -- 172.21.15.114:0/4271595226 >> 172.21.15.114:0/4271595226 conn(0x7f394c0fc160 msgr2=0x7f394c0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:51:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.963+0000 7f395343d700 1 -- 172.21.15.114:0/4271595226 shutdown_connections 2024-03-11T19:51:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.963+0000 7f395343d700 1 -- 172.21.15.114:0/4271595226 wait complete. 2024-03-11T19:51:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.964+0000 7f395343d700 1 Processor -- start 2024-03-11T19:51:03.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.964+0000 7f395343d700 1 -- start start 2024-03-11T19:51:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.964+0000 7f395343d700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f394c102790 0x7f394c1020f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:51:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.964+0000 7f395343d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f394c1050c0 0x7f394c100740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:51:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.964+0000 7f395343d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f394c1079f0 0x7f394c100c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:51:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.964+0000 7f395343d700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c101400 con 0x7f394c102790 2024-03-11T19:51:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.964+0000 7f395343d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c101570 con 0x7f394c1050c0 2024-03-11T19:51:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.964+0000 7f395343d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c1a1f00 con 0x7f394c1079f0 2024-03-11T19:51:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.964+0000 7f39509d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f394c1050c0 0x7f394c100740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:51:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.964+0000 7f39509d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f394c1050c0 0x7f394c100740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58392/0 (socket says 172.21.15.114:58392) 2024-03-11T19:51:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.964+0000 7f39509d8700 1 -- 172.21.15.114:0/3735029239 learned_addr learned my addr 172.21.15.114:0/3735029239 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-11T19:51:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.964+0000 7f39519da700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f394c1079f0 0x7f394c100c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:51:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f39511d9700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f394c102790 0x7f394c1020f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:51:03.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f39509d8700 1 -- 172.21.15.114:0/3735029239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f394c1079f0 msgr2=0x7f394c100c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:51:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f39509d8700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f394c1079f0 0x7f394c100c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f39509d8700 1 -- 172.21.15.114:0/3735029239 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f394c102790 msgr2=0x7f394c1020f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:51:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f39509d8700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f394c102790 0x7f394c1020f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f39509d8700 1 -- 172.21.15.114:0/3735029239 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3944009d00 con 0x7f394c1050c0 2024-03-11T19:51:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f39519da700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f394c1079f0 0x7f394c100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:51:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f39511d9700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f394c102790 0x7f394c1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:51:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f39509d8700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f394c1050c0 0x7f394c100740 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f394800b530 tx=0x7f394800b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:51:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f393e7fc700 1 -- 172.21.15.114:0/3735029239 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3948011830 con 0x7f394c1050c0 2024-03-11T19:51:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f393e7fc700 1 -- 172.21.15.114:0/3735029239 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3948011e70 con 0x7f394c1050c0 2024-03-11T19:51:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f395343d700 1 -- 172.21.15.114:0/3735029239 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394c1a21e0 con 0x7f394c1050c0 2024-03-11T19:51:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f393e7fc700 1 -- 172.21.15.114:0/3735029239 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394800f2e0 con 0x7f394c1050c0 2024-03-11T19:51:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.965+0000 7f395343d700 1 -- 172.21.15.114:0/3735029239 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394c1a2760 con 0x7f394c1050c0 2024-03-11T19:51:03.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.966+0000 7f395343d700 1 -- 172.21.15.114:0/3735029239 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394c0fdce0 con 0x7f394c1050c0 2024-03-11T19:51:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.967+0000 7f393e7fc700 1 -- 172.21.15.114:0/3735029239 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f39480105f0 con 0x7f394c1050c0 2024-03-11T19:51:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.968+0000 7f393e7fc700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39380a1ac0 0x7f39380a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:51:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.968+0000 7f393e7fc700 1 -- 172.21.15.114:0/3735029239 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3669+0+0 (secure 0 0 0) 0x7f39480cc1b0 con 0x7f394c1050c0 2024-03-11T19:51:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.968+0000 7f39511d9700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39380a1ac0 0x7f39380a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:51:03.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.969+0000 7f39511d9700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39380a1ac0 0x7f39380a3f80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3940007900 tx=0x7f3940008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:51:03.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:03.972+0000 7f393e7fc700 1 -- 172.21.15.114:0/3735029239 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f39480965e0 con 0x7f394c1050c0 2024-03-11T19:51:04.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:04.174+0000 7f395343d700 1 -- 172.21.15.114:0/3735029239 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f394c02d7d0 con 0x7f39380a1ac0 2024-03-11T19:51:04.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:04 smithi161 ceph-mon[36332]: osdmap e29: 4 total, 4 up, 4 in 2024-03-11T19:51:04.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:04 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:51:04.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:04 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:51:04.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:04 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:04.802 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:04 smithi114 ceph-mon[36376]: osdmap e29: 4 total, 4 up, 4 in 2024-03-11T19:51:04.802 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:04 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:51:04.802 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:04 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:51:04.802 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:04 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:04.822 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:04 smithi097 ceph-mon[31898]: osdmap e29: 4 total, 4 up, 4 in 2024-03-11T19:51:04.823 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:04 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:51:04.823 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:04 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:51:04.823 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:04 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:05.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:05 smithi114 ceph-mon[36376]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:51:05.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:05 smithi114 ceph-mon[36376]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T19:51:05.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:05.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:05.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:05.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:05.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:05.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:05.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:05.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:05.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:05 smithi161 ceph-mon[36332]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:51:05.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:05 smithi161 ceph-mon[36332]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T19:51:05.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:05.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:05.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:05.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:05.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:05.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:05.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:05.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:05 smithi097 ceph-mon[31898]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:51:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:05 smithi097 ceph-mon[31898]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T19:51:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:05.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:05.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:05.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:05.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:05.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:06.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:06 smithi097 ceph-mon[31898]: Detected new or changed devices on smithi097 2024-03-11T19:51:06.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:06 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi097 to 8545M 2024-03-11T19:51:06.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:06 smithi114 ceph-mon[36376]: Detected new or changed devices on smithi097 2024-03-11T19:51:06.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:06 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi097 to 8545M 2024-03-11T19:51:07.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:06 smithi161 ceph-mon[36332]: Detected new or changed devices on smithi097 2024-03-11T19:51:07.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:06 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi097 to 8545M 2024-03-11T19:51:07.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:07 smithi097 ceph-mon[31898]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T19:51:07.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:07 smithi097 ceph-mon[31898]: from='client.? 172.21.15.114:0/3607048424' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "601a3bf7-cfee-4e6d-a838-7f04eccdb73f"}]: dispatch 2024-03-11T19:51:07.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:07 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "601a3bf7-cfee-4e6d-a838-7f04eccdb73f"}]: dispatch 2024-03-11T19:51:07.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:07 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "601a3bf7-cfee-4e6d-a838-7f04eccdb73f"}]': finished 2024-03-11T19:51:07.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:07 smithi097 ceph-mon[31898]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T19:51:07.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:07 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:07.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:07 smithi114 ceph-mon[36376]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T19:51:07.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:07 smithi114 ceph-mon[36376]: from='client.? 172.21.15.114:0/3607048424' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "601a3bf7-cfee-4e6d-a838-7f04eccdb73f"}]: dispatch 2024-03-11T19:51:07.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:07 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "601a3bf7-cfee-4e6d-a838-7f04eccdb73f"}]: dispatch 2024-03-11T19:51:07.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:07 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "601a3bf7-cfee-4e6d-a838-7f04eccdb73f"}]': finished 2024-03-11T19:51:07.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:07 smithi114 ceph-mon[36376]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T19:51:07.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:07 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:08.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:07 smithi161 ceph-mon[36332]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T19:51:08.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:07 smithi161 ceph-mon[36332]: from='client.? 172.21.15.114:0/3607048424' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "601a3bf7-cfee-4e6d-a838-7f04eccdb73f"}]: dispatch 2024-03-11T19:51:08.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:07 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "601a3bf7-cfee-4e6d-a838-7f04eccdb73f"}]: dispatch 2024-03-11T19:51:08.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:07 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "601a3bf7-cfee-4e6d-a838-7f04eccdb73f"}]': finished 2024-03-11T19:51:08.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:07 smithi161 ceph-mon[36332]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T19:51:08.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:07 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:08.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:08 smithi161 ceph-mon[36332]: from='client.? 172.21.15.114:0/2240971946' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:51:08.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:08 smithi097 ceph-mon[31898]: from='client.? 172.21.15.114:0/2240971946' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:51:08.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:08 smithi114 ceph-mon[36376]: from='client.? 172.21.15.114:0/2240971946' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:51:09.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:09 smithi097 ceph-mon[31898]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:51:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:09 smithi114 ceph-mon[36376]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:51:10.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:09 smithi161 ceph-mon[36332]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:51:11.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:11 smithi097 ceph-mon[31898]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T19:51:11.931 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:11 smithi114 ceph-mon[36376]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T19:51:12.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:11 smithi161 ceph-mon[36332]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T19:51:13.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:13 smithi097 ceph-mon[31898]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T19:51:13.934 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:13 smithi114 ceph-mon[36376]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T19:51:14.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:13 smithi161 ceph-mon[36332]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T19:51:14.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:14 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T19:51:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:14 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:14.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:14 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T19:51:14.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:14 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:15.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:14 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T19:51:15.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:14 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:15.847 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:15 smithi097 ceph-mon[31898]: Deploying daemon osd.4 on smithi114 2024-03-11T19:51:15.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:15 smithi097 ceph-mon[31898]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:15.958 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:15 smithi114 ceph-mon[36376]: Deploying daemon osd.4 on smithi114 2024-03-11T19:51:15.958 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:15 smithi114 ceph-mon[36376]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:16.042 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:15 smithi161 ceph-mon[36332]: Deploying daemon osd.4 on smithi114 2024-03-11T19:51:16.042 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:15 smithi161 ceph-mon[36332]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:17.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:17 smithi097 ceph-mon[31898]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:17.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:17 smithi114 ceph-mon[36376]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:18.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:17 smithi161 ceph-mon[36332]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:19.702 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:19 smithi114 ceph-mon[36376]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:19.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:19 smithi097 ceph-mon[31898]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:20.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:19 smithi161 ceph-mon[36332]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:21.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:20 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:21.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:20 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:21.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:20 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:51:21.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:20 smithi097 ceph-mon[31898]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:21.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:20 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:21.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:20 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:21.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:20 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:51:21.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:20 smithi114 ceph-mon[36376]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:21.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:20 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:21.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:20 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:21.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:20 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:51:21.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:20 smithi161 ceph-mon[36332]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:23.360 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 4 on host 'smithi114' 2024-03-11T19:51:23.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:23.356+0000 7f393e7fc700 1 -- 172.21.15.114:0/3735029239 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f394c02d7d0 con 0x7f39380a1ac0 2024-03-11T19:51:23.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:23.359+0000 7f395343d700 1 -- 172.21.15.114:0/3735029239 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39380a1ac0 msgr2=0x7f39380a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:51:23.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:23.359+0000 7f395343d700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39380a1ac0 0x7f39380a3f80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3940007900 tx=0x7f3940008040 comp rx=0 tx=0).stop 2024-03-11T19:51:23.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:23.359+0000 7f395343d700 1 -- 172.21.15.114:0/3735029239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f394c1050c0 msgr2=0x7f394c100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:51:23.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:23.359+0000 7f395343d700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f394c1050c0 0x7f394c100740 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f394800b530 tx=0x7f394800b8f0 comp rx=0 tx=0).stop 2024-03-11T19:51:23.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:23.359+0000 7f395343d700 1 -- 172.21.15.114:0/3735029239 shutdown_connections 2024-03-11T19:51:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:23.359+0000 7f395343d700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39380a1ac0 0x7f39380a3f80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:23.359+0000 7f395343d700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f394c102790 0x7f394c1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:23.359+0000 7f395343d700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f394c1079f0 0x7f394c100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:23.359+0000 7f395343d700 1 --2- 172.21.15.114:0/3735029239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f394c1050c0 0x7f394c100740 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:23.359+0000 7f395343d700 1 -- 172.21.15.114:0/3735029239 >> 172.21.15.114:0/3735029239 conn(0x7f394c0fc160 msgr2=0x7f394c1030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:51:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:23.359+0000 7f395343d700 1 -- 172.21.15.114:0/3735029239 shutdown_connections 2024-03-11T19:51:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:51:23.359+0000 7f395343d700 1 -- 172.21.15.114:0/3735029239 wait complete. 2024-03-11T19:51:23.680 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:23 smithi114 ceph-mon[36376]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:23.680 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:23.680 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:23.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:23 smithi161 ceph-mon[36332]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:23.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:23.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:23 smithi097 ceph-mon[31898]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:23.965 DEBUG:teuthology.orchestra.run.smithi114:osd.4> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.4.service 2024-03-11T19:51:23.967 INFO:tasks.cephadm:Deploying osd.5 on smithi161 with /dev/vg_nvme/lv_3... 2024-03-11T19:51:23.968 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T19:51:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:51:24.587 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:24 smithi161 ceph-mon[36332]: from='osd.4 [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T19:51:24.587 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:24 smithi161 ceph-mon[36332]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T19:51:24.587 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:24 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:24.587 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:24 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:24.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:24 smithi114 ceph-mon[36376]: from='osd.4 [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T19:51:24.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:24 smithi114 ceph-mon[36376]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T19:51:24.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:24 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:24.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:24 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:24.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:24 smithi097 ceph-mon[31898]: from='osd.4 [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T19:51:24.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:24 smithi097 ceph-mon[31898]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T19:51:24.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:24 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:24.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:24 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:25.676 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:25 smithi161 ceph-mon[36332]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:25.676 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:25 smithi161 ceph-mon[36332]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T19:51:25.676 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:25 smithi161 ceph-mon[36332]: from='osd.4 [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:51:25.676 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:25 smithi161 ceph-mon[36332]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T19:51:25.676 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:25 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:25.676 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:25 smithi161 ceph-mon[36332]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:51:25.786 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:25 smithi114 ceph-mon[36376]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:25.786 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:25 smithi114 ceph-mon[36376]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T19:51:25.786 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:25 smithi114 ceph-mon[36376]: from='osd.4 [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:51:25.786 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:25 smithi114 ceph-mon[36376]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T19:51:25.786 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:25 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:25.786 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:25 smithi114 ceph-mon[36376]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:51:25.787 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 19:51:25 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4[47754]: 2024-03-11T19:51:25.493+0000 7fb4ac7ee700 -1 osd.4 0 waiting for initial osdmap 2024-03-11T19:51:25.787 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 19:51:25 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4[47754]: 2024-03-11T19:51:25.500+0000 7fb4a85e1700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:51:25.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:25 smithi097 ceph-mon[31898]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:25.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:25 smithi097 ceph-mon[31898]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T19:51:25.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:25 smithi097 ceph-mon[31898]: from='osd.4 [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:51:25.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:25 smithi097 ceph-mon[31898]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T19:51:25.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:25 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:25.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:25 smithi097 ceph-mon[31898]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:51:26.280 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T19:51:26.280 INFO:teuthology.orchestra.run.smithi161.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T19:51:26.280 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10+0 records in 2024-03-11T19:51:26.280 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2024-03-11T19:51:26.280 INFO:teuthology.orchestra.run.smithi161.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013219 s, 793 MB/s 2024-03-11T19:51:26.280 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping successful for: 2024-03-11T19:51:26.649 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:26 smithi161 ceph-mon[36332]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-11T19:51:26.649 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:26 smithi161 ceph-mon[36332]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T19:51:26.649 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:26 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:26.836 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:26 smithi097 ceph-mon[31898]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-11T19:51:26.837 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:26 smithi097 ceph-mon[31898]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T19:51:26.837 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:26 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:26.865 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:26 smithi114 ceph-mon[36376]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-11T19:51:26.865 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:26 smithi114 ceph-mon[36376]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T19:51:26.865 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:26 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:26.985 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi161:vg_nvme/lv_3 2024-03-11T19:51:27.228 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:51:27.515 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:27 smithi161 ceph-mon[36332]: purged_snaps scrub starts 2024-03-11T19:51:27.516 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:27 smithi161 ceph-mon[36332]: purged_snaps scrub ok 2024-03-11T19:51:27.516 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:27 smithi161 ceph-mon[36332]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:27.516 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:27 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:27.516 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:27 smithi161 ceph-mon[36332]: osd.4 [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177] boot 2024-03-11T19:51:27.516 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:27 smithi161 ceph-mon[36332]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T19:51:27.516 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:27 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:27.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:27 smithi097 ceph-mon[31898]: purged_snaps scrub starts 2024-03-11T19:51:27.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:27 smithi097 ceph-mon[31898]: purged_snaps scrub ok 2024-03-11T19:51:27.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:27 smithi097 ceph-mon[31898]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:27.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:27 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:27.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:27 smithi097 ceph-mon[31898]: osd.4 [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177] boot 2024-03-11T19:51:27.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:27 smithi097 ceph-mon[31898]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T19:51:27.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:27 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:27 smithi114 ceph-mon[36376]: purged_snaps scrub starts 2024-03-11T19:51:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:27 smithi114 ceph-mon[36376]: purged_snaps scrub ok 2024-03-11T19:51:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:27 smithi114 ceph-mon[36376]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-11T19:51:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:27 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:27 smithi114 ceph-mon[36376]: osd.4 [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177] boot 2024-03-11T19:51:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:27 smithi114 ceph-mon[36376]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T19:51:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:27 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:51:28.676 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T19:51:28.676 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: Detected new or changed devices on smithi114 2024-03-11T19:51:28.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:28.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:28.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi114 to 8540M 2024-03-11T19:51:28.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:28.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:28.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:51:28.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:28.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:28.677 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:28 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T19:51:28.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: Detected new or changed devices on smithi114 2024-03-11T19:51:28.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:28.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:28.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi114 to 8540M 2024-03-11T19:51:28.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:28.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:28.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:51:28.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:28.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:28.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:28 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 -- 172.21.15.161:0/1302640023 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f961c109b70 msgr2=0x7f961c10c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:51:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 --2- 172.21.15.161:0/1302640023 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f961c109b70 0x7f961c10c000 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f961400b0a0 tx=0x7f961400b3b0 comp rx=0 tx=0).stop 2024-03-11T19:51:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 -- 172.21.15.161:0/1302640023 shutdown_connections 2024-03-11T19:51:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 --2- 172.21.15.161:0/1302640023 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f961c109b70 0x7f961c10c000 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 --2- 172.21.15.161:0/1302640023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f961c075ca0 0x7f961c074100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 --2- 172.21.15.161:0/1302640023 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f961c0746d0 0x7f961c074b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 -- 172.21.15.161:0/1302640023 >> 172.21.15.161:0/1302640023 conn(0x7f961c0fd280 msgr2=0x7f961c0ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:51:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 -- 172.21.15.161:0/1302640023 shutdown_connections 2024-03-11T19:51:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 -- 172.21.15.161:0/1302640023 wait complete. 2024-03-11T19:51:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 Processor -- start 2024-03-11T19:51:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 -- start start 2024-03-11T19:51:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f961c0746d0 0x7f961c199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:51:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f961c075ca0 0x7f961c19a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:51:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f961c109b70 0x7f961c1a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:51:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c1a1b80 con 0x7f961c075ca0 2024-03-11T19:51:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c1a1cc0 con 0x7f961c109b70 2024-03-11T19:51:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f9622c5f700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c1a1e50 con 0x7f961c0746d0 2024-03-11T19:51:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.913+0000 7f96209fb700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f961c0746d0 0x7f961c199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:51:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f96209fb700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f961c0746d0 0x7f961c199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:56536/0 (socket says 172.21.15.161:56536) 2024-03-11T19:51:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f96209fb700 1 -- 172.21.15.161:0/4185404113 learned_addr learned my addr 172.21.15.161:0/4185404113 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-11T19:51:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f96211fc700 1 --2- 172.21.15.161:0/4185404113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f961c109b70 0x7f961c1a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:51:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f96209fb700 1 -- 172.21.15.161:0/4185404113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f961c109b70 msgr2=0x7f961c1a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:51:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f96209fb700 1 --2- 172.21.15.161:0/4185404113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f961c109b70 0x7f961c1a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f96209fb700 1 -- 172.21.15.161:0/4185404113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f961c075ca0 msgr2=0x7f961c19a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:51:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f96209fb700 1 --2- 172.21.15.161:0/4185404113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f961c075ca0 0x7f961c19a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f96209fb700 1 -- 172.21.15.161:0/4185404113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9614009d00 con 0x7f961c0746d0 2024-03-11T19:51:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f96209fb700 1 --2- 172.21.15.161:0/4185404113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f961c0746d0 0x7f961c199ff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f961000c920 tx=0x7f961000cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:51:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f9619ffb700 1 -- 172.21.15.161:0/4185404113 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610007ac0 con 0x7f961c0746d0 2024-03-11T19:51:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f9622c5f700 1 -- 172.21.15.161:0/4185404113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961c1a2130 con 0x7f961c0746d0 2024-03-11T19:51:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f9619ffb700 1 -- 172.21.15.161:0/4185404113 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f961000ce70 con 0x7f961c0746d0 2024-03-11T19:51:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f9619ffb700 1 -- 172.21.15.161:0/4185404113 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610018660 con 0x7f961c0746d0 2024-03-11T19:51:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.914+0000 7f9622c5f700 1 -- 172.21.15.161:0/4185404113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961c1a26b0 con 0x7f961c0746d0 2024-03-11T19:51:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.915+0000 7f9622c5f700 1 -- 172.21.15.161:0/4185404113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961c108930 con 0x7f961c0746d0 2024-03-11T19:51:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.916+0000 7f9619ffb700 1 -- 172.21.15.161:0/4185404113 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f961000f450 con 0x7f961c0746d0 2024-03-11T19:51:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.917+0000 7f9619ffb700 1 --2- 172.21.15.161:0/4185404113 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f960c0a1ac0 0x7f960c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:51:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.917+0000 7f9619ffb700 1 -- 172.21.15.161:0/4185404113 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f96100cc240 con 0x7f961c0746d0 2024-03-11T19:51:28.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.917+0000 7f961bfff700 1 --2- 172.21.15.161:0/4185404113 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f960c0a1ac0 0x7f960c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:51:28.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.918+0000 7f961bfff700 1 --2- 172.21.15.161:0/4185404113 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f960c0a1ac0 0x7f960c0a3f80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f961c0759d0 tx=0x7f9608006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:51:28.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:28.921+0000 7f9619ffb700 1 -- 172.21.15.161:0/4185404113 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f96100964c0 con 0x7f961c0746d0 2024-03-11T19:51:28.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T19:51:28.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: Detected new or changed devices on smithi114 2024-03-11T19:51:28.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:28.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:28.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi114 to 8540M 2024-03-11T19:51:28.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:28.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:28.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:28.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:51:28.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:28.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:28.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:28 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:29.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:29.121+0000 7f9622c5f700 1 -- 172.21.15.161:0/4185404113 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f961c02d8d0 con 0x7f960c0a1ac0 2024-03-11T19:51:29.786 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:29 smithi161 ceph-mon[36332]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:29.786 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:51:29.786 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:51:29.786 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:29.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:29 smithi097 ceph-mon[31898]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:29.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:51:29.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:51:29.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:29.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:29 smithi114 ceph-mon[36376]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:29.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:51:29.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:51:29.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:30.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:30 smithi161 ceph-mon[36332]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:51:30.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:30 smithi097 ceph-mon[31898]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:51:30.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:30 smithi114 ceph-mon[36376]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:51:31.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:31 smithi097 ceph-mon[31898]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:31.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:31 smithi114 ceph-mon[36376]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:32.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:31 smithi161 ceph-mon[36332]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:32.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:32 smithi097 ceph-mon[31898]: from='client.? 172.21.15.161:0/273421579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ce520f9-2aef-4901-aa82-edd7ca858028"}]: dispatch 2024-03-11T19:51:32.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:32 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ce520f9-2aef-4901-aa82-edd7ca858028"}]: dispatch 2024-03-11T19:51:32.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:32 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ce520f9-2aef-4901-aa82-edd7ca858028"}]': finished 2024-03-11T19:51:32.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:32 smithi097 ceph-mon[31898]: osdmap e35: 6 total, 5 up, 6 in 2024-03-11T19:51:32.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:32 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:51:32.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:32 smithi097 ceph-mon[31898]: from='client.? 172.21.15.161:0/1931714245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:51:32.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:32 smithi114 ceph-mon[36376]: from='client.? 172.21.15.161:0/273421579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ce520f9-2aef-4901-aa82-edd7ca858028"}]: dispatch 2024-03-11T19:51:32.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:32 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ce520f9-2aef-4901-aa82-edd7ca858028"}]: dispatch 2024-03-11T19:51:32.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:32 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ce520f9-2aef-4901-aa82-edd7ca858028"}]': finished 2024-03-11T19:51:32.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:32 smithi114 ceph-mon[36376]: osdmap e35: 6 total, 5 up, 6 in 2024-03-11T19:51:32.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:32 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:51:32.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:32 smithi114 ceph-mon[36376]: from='client.? 172.21.15.161:0/1931714245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:51:33.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:32 smithi161 ceph-mon[36332]: from='client.? 172.21.15.161:0/273421579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ce520f9-2aef-4901-aa82-edd7ca858028"}]: dispatch 2024-03-11T19:51:33.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:32 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ce520f9-2aef-4901-aa82-edd7ca858028"}]: dispatch 2024-03-11T19:51:33.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:32 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ce520f9-2aef-4901-aa82-edd7ca858028"}]': finished 2024-03-11T19:51:33.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:32 smithi161 ceph-mon[36332]: osdmap e35: 6 total, 5 up, 6 in 2024-03-11T19:51:33.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:32 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:51:33.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:32 smithi161 ceph-mon[36332]: from='client.? 172.21.15.161:0/1931714245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:51:33.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:33 smithi114 ceph-mon[36376]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:34.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:33 smithi161 ceph-mon[36332]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:34.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:33 smithi097 ceph-mon[31898]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:35.967 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:35 smithi161 ceph-mon[36332]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:35 smithi114 ceph-mon[36376]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:36.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:35 smithi097 ceph-mon[31898]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:37.884 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:37 smithi161 ceph-mon[36332]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:37.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:37 smithi114 ceph-mon[36376]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:38.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:37 smithi097 ceph-mon[31898]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:39.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:39 smithi161 ceph-mon[36332]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:39.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T19:51:39.917 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:39.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:39 smithi114 ceph-mon[36376]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:39.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T19:51:39.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:40.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:39 smithi097 ceph-mon[31898]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:40.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T19:51:40.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:40.963 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:40 smithi161 ceph-mon[36332]: Deploying daemon osd.5 on smithi161 2024-03-11T19:51:40.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:40 smithi114 ceph-mon[36376]: Deploying daemon osd.5 on smithi161 2024-03-11T19:51:41.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:40 smithi097 ceph-mon[31898]: Deploying daemon osd.5 on smithi161 2024-03-11T19:51:41.943 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:41 smithi161 ceph-mon[36332]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:41.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:41 smithi114 ceph-mon[36376]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:42.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:41 smithi097 ceph-mon[31898]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:42.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:42 smithi114 ceph-mon[36376]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:43.006 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:42 smithi161 ceph-mon[36332]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:43.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:42 smithi097 ceph-mon[31898]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:45 smithi097 ceph-mon[31898]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:51:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:45 smithi114 ceph-mon[36376]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:45.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:45.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:51:45.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:46.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:45 smithi161 ceph-mon[36332]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:46.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:46.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:51:46.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:47.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:47 smithi097 ceph-mon[31898]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:47 smithi114 ceph-mon[36376]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:48.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:47 smithi161 ceph-mon[36332]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:48.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:48.206+0000 7f9619ffb700 1 -- 172.21.15.161:0/4185404113 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f961c02d8d0 con 0x7f960c0a1ac0 2024-03-11T19:51:48.211 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 5 on host 'smithi161' 2024-03-11T19:51:48.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:48.209+0000 7f9622c5f700 1 -- 172.21.15.161:0/4185404113 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f960c0a1ac0 msgr2=0x7f960c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:51:48.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:48.209+0000 7f9622c5f700 1 --2- 172.21.15.161:0/4185404113 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f960c0a1ac0 0x7f960c0a3f80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f961c0759d0 tx=0x7f9608006d20 comp rx=0 tx=0).stop 2024-03-11T19:51:48.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:48.209+0000 7f9622c5f700 1 -- 172.21.15.161:0/4185404113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f961c0746d0 msgr2=0x7f961c199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:51:48.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:48.209+0000 7f9622c5f700 1 --2- 172.21.15.161:0/4185404113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f961c0746d0 0x7f961c199ff0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f961000c920 tx=0x7f961000cc30 comp rx=0 tx=0).stop 2024-03-11T19:51:48.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:48.209+0000 7f9622c5f700 1 -- 172.21.15.161:0/4185404113 shutdown_connections 2024-03-11T19:51:48.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:48.209+0000 7f9622c5f700 1 --2- 172.21.15.161:0/4185404113 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f960c0a1ac0 0x7f960c0a3f80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:48.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:48.209+0000 7f9622c5f700 1 --2- 172.21.15.161:0/4185404113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f961c109b70 0x7f961c1a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:48.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:48.209+0000 7f9622c5f700 1 --2- 172.21.15.161:0/4185404113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f961c0746d0 0x7f961c199ff0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:48.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:48.209+0000 7f9622c5f700 1 --2- 172.21.15.161:0/4185404113 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f961c075ca0 0x7f961c19a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:48.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:48.209+0000 7f9622c5f700 1 -- 172.21.15.161:0/4185404113 >> 172.21.15.161:0/4185404113 conn(0x7f961c0fd280 msgr2=0x7f961c105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:51:48.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:48.209+0000 7f9622c5f700 1 -- 172.21.15.161:0/4185404113 shutdown_connections 2024-03-11T19:51:48.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:51:48.209+0000 7f9622c5f700 1 -- 172.21.15.161:0/4185404113 wait complete. 2024-03-11T19:51:48.786 DEBUG:teuthology.orchestra.run.smithi161:osd.5> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.5.service 2024-03-11T19:51:48.788 INFO:tasks.cephadm:Deploying osd.6 on smithi097 with /dev/vg_nvme/lv_2... 2024-03-11T19:51:48.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T19:51:49.230 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:51:49.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:49 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:49.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:49 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:49.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:49 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:49.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:49 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:49.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:49 smithi097 ceph-mon[31898]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:49.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:49 smithi097 ceph-mon[31898]: from='osd.5 [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T19:51:49.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:49 smithi097 ceph-mon[31898]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T19:51:49.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:49 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:49.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:49 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:49.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:49 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:49.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:49 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:49.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:49 smithi114 ceph-mon[36376]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:49.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:49 smithi114 ceph-mon[36376]: from='osd.5 [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T19:51:49.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:49 smithi114 ceph-mon[36376]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T19:51:49.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:49 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:49.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:49 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:49.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:49 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:49.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:49 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:49.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:49 smithi161 ceph-mon[36332]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:49.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:49 smithi161 ceph-mon[36332]: from='osd.5 [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T19:51:49.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:49 smithi161 ceph-mon[36332]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T19:51:50.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:50 smithi114 ceph-mon[36376]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T19:51:50.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:50 smithi114 ceph-mon[36376]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T19:51:50.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:50 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:51:50.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:50 smithi114 ceph-mon[36376]: from='osd.5 [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:51:50.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:50 smithi114 ceph-mon[36376]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:51:50.511 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:50 smithi161 ceph-mon[36332]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T19:51:50.511 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:50 smithi161 ceph-mon[36332]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T19:51:50.511 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:50 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:51:50.512 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:50 smithi161 ceph-mon[36332]: from='osd.5 [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:51:50.512 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:50 smithi161 ceph-mon[36332]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:51:50.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:50 smithi097 ceph-mon[31898]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T19:51:50.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:50 smithi097 ceph-mon[31898]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T19:51:50.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:50 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:51:50.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:50 smithi097 ceph-mon[31898]: from='osd.5 [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:51:50.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:50 smithi097 ceph-mon[31898]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:51:51.182 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 19:51:50 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T19:51:50.917+0000 7f33e76a0700 -1 osd.5 0 waiting for initial osdmap 2024-03-11T19:51:51.182 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 19:51:50 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T19:51:50.924+0000 7f33df48b700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:51:51.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:51 smithi097 ceph-mon[31898]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-03-11T19:51:51.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:51 smithi097 ceph-mon[31898]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T19:51:51.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:51 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:51:51.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:51 smithi097 ceph-mon[31898]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:51.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:51 smithi097 ceph-mon[31898]: from='osd.5 ' entity='osd.5' 2024-03-11T19:51:51.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:51 smithi097 ceph-mon[31898]: osd.5 [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465] boot 2024-03-11T19:51:51.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:51 smithi097 ceph-mon[31898]: osdmap e38: 6 total, 6 up, 6 in 2024-03-11T19:51:51.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:51 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:51:51.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:51 smithi161 ceph-mon[36332]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-03-11T19:51:51.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:51 smithi161 ceph-mon[36332]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T19:51:51.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:51 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:51:51.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:51 smithi161 ceph-mon[36332]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:51.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:51 smithi161 ceph-mon[36332]: from='osd.5 ' entity='osd.5' 2024-03-11T19:51:51.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:51 smithi161 ceph-mon[36332]: osd.5 [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465] boot 2024-03-11T19:51:51.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:51 smithi161 ceph-mon[36332]: osdmap e38: 6 total, 6 up, 6 in 2024-03-11T19:51:51.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:51 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:51:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:51 smithi114 ceph-mon[36376]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-03-11T19:51:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:51 smithi114 ceph-mon[36376]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T19:51:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:51 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:51:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:51 smithi114 ceph-mon[36376]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-11T19:51:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:51 smithi114 ceph-mon[36376]: from='osd.5 ' entity='osd.5' 2024-03-11T19:51:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:51 smithi114 ceph-mon[36376]: osd.5 [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465] boot 2024-03-11T19:51:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:51 smithi114 ceph-mon[36376]: osdmap e38: 6 total, 6 up, 6 in 2024-03-11T19:51:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:51 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:51:52.233 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T19:51:52.233 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T19:51:52.234 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-03-11T19:51:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-03-11T19:51:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0544935 s, 192 MB/s 2024-03-11T19:51:52.234 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-03-11T19:51:52.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:52 smithi161 ceph-mon[36332]: purged_snaps scrub starts 2024-03-11T19:51:52.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:52 smithi161 ceph-mon[36332]: purged_snaps scrub ok 2024-03-11T19:51:52.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:52 smithi161 ceph-mon[36332]: osdmap e39: 6 total, 6 up, 6 in 2024-03-11T19:51:52.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:52 smithi097 ceph-mon[31898]: purged_snaps scrub starts 2024-03-11T19:51:52.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:52 smithi097 ceph-mon[31898]: purged_snaps scrub ok 2024-03-11T19:51:52.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:52 smithi097 ceph-mon[31898]: osdmap e39: 6 total, 6 up, 6 in 2024-03-11T19:51:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:52 smithi114 ceph-mon[36376]: purged_snaps scrub starts 2024-03-11T19:51:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:52 smithi114 ceph-mon[36376]: purged_snaps scrub ok 2024-03-11T19:51:52.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:52 smithi114 ceph-mon[36376]: osdmap e39: 6 total, 6 up, 6 in 2024-03-11T19:51:53.165 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi097:vg_nvme/lv_2 2024-03-11T19:51:53.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: Detected new or changed devices on smithi161 2024-03-11T19:51:53.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi161 to 8540M 2024-03-11T19:51:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 510 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:51:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:51:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:53 smithi097 ceph-mon[31898]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T19:51:53.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: Detected new or changed devices on smithi161 2024-03-11T19:51:53.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:53.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:53.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi161 to 8540M 2024-03-11T19:51:53.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:53.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:53.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 510 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:51:53.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:51:53.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:53.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:53.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:53 smithi161 ceph-mon[36332]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T19:51:53.614 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:51:53.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: Detected new or changed devices on smithi161 2024-03-11T19:51:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:51:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi161 to 8540M 2024-03-11T19:51:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 510 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:51:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:51:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:53.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:51:53.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:51:53.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:53 smithi114 ceph-mon[36376]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T19:51:55.384 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:55 smithi097 ceph-mon[31898]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T19:51:55.384 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:55 smithi097 ceph-mon[31898]: pgmap v156: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 510 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T19:51:55.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:55 smithi114 ceph-mon[36376]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T19:51:55.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:55 smithi114 ceph-mon[36376]: pgmap v156: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 510 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T19:51:55.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:55 smithi161 ceph-mon[36332]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T19:51:55.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:55 smithi161 ceph-mon[36332]: pgmap v156: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 510 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T19:51:55.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.834+0000 7f45e3e00700 1 -- 172.21.15.97:0/4061992296 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45dc105060 msgr2=0x7f45dc107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:51:55.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.834+0000 7f45e3e00700 1 --2- 172.21.15.97:0/4061992296 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45dc105060 0x7f45dc107450 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009990 tx=0x7f45c8009ca0 comp rx=0 tx=0).stop 2024-03-11T19:51:55.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.834+0000 7f45e3e00700 1 -- 172.21.15.97:0/4061992296 shutdown_connections 2024-03-11T19:51:55.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.834+0000 7f45e3e00700 1 --2- 172.21.15.97:0/4061992296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45dc107990 0x7f45dc109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:55.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.834+0000 7f45e3e00700 1 --2- 172.21.15.97:0/4061992296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45dc102730 0x7f45dc104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:55.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.834+0000 7f45e3e00700 1 --2- 172.21.15.97:0/4061992296 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45dc105060 0x7f45dc107450 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:55.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.834+0000 7f45e3e00700 1 -- 172.21.15.97:0/4061992296 >> 172.21.15.97:0/4061992296 conn(0x7f45dc0fc140 msgr2=0x7f45dc0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:51:55.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.834+0000 7f45e3e00700 1 -- 172.21.15.97:0/4061992296 shutdown_connections 2024-03-11T19:51:55.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.834+0000 7f45e3e00700 1 -- 172.21.15.97:0/4061992296 wait complete. 2024-03-11T19:51:55.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.836+0000 7f45e3e00700 1 Processor -- start 2024-03-11T19:51:55.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.836+0000 7f45e3e00700 1 -- start start 2024-03-11T19:51:55.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.836+0000 7f45e3e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45dc102730 0x7f45dc102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:51:55.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.836+0000 7f45e3e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45dc105060 0x7f45dc1006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:51:55.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.836+0000 7f45e3e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45dc107990 0x7f45dc100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:51:55.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.836+0000 7f45e3e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1013a0 con 0x7f45dc107990 2024-03-11T19:51:55.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.836+0000 7f45e3e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc101510 con 0x7f45dc102730 2024-03-11T19:51:55.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.836+0000 7f45e3e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1a1d20 con 0x7f45dc105060 2024-03-11T19:51:55.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.837+0000 7f45e1400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45dc102730 0x7f45dc102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:51:55.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.837+0000 7f45e1e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45dc107990 0x7f45dc100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:51:55.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.837+0000 7f45e1400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45dc102730 0x7f45dc102090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:40442/0 (socket says 172.21.15.97:40442) 2024-03-11T19:51:55.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.837+0000 7f45e1400700 1 -- 172.21.15.97:0/512696585 learned_addr learned my addr 172.21.15.97:0/512696585 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:51:55.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.837+0000 7f45e0a00700 1 --2- 172.21.15.97:0/512696585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45dc105060 0x7f45dc1006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:51:55.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.837+0000 7f45e1400700 1 -- 172.21.15.97:0/512696585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45dc105060 msgr2=0x7f45dc1006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:51:55.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.837+0000 7f45e1400700 1 --2- 172.21.15.97:0/512696585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45dc105060 0x7f45dc1006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:55.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.837+0000 7f45e1400700 1 -- 172.21.15.97:0/512696585 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45dc107990 msgr2=0x7f45dc100c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:51:55.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.837+0000 7f45e1400700 1 --2- 172.21.15.97:0/512696585 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45dc107990 0x7f45dc100c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:51:55.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.837+0000 7f45e1400700 1 -- 172.21.15.97:0/512696585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c80095f0 con 0x7f45dc102730 2024-03-11T19:51:55.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.837+0000 7f45e1400700 1 --2- 172.21.15.97:0/512696585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45dc102730 0x7f45dc102090 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f45d000e9a0 tx=0x7f45d000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:51:55.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.838+0000 7f45da000700 1 -- 172.21.15.97:0/512696585 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0009790 con 0x7f45dc102730 2024-03-11T19:51:55.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.838+0000 7f45da000700 1 -- 172.21.15.97:0/512696585 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45d0010430 con 0x7f45dc102730 2024-03-11T19:51:55.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.838+0000 7f45e3e00700 1 -- 172.21.15.97:0/512696585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45dc1a1fa0 con 0x7f45dc102730 2024-03-11T19:51:55.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.838+0000 7f45da000700 1 -- 172.21.15.97:0/512696585 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0018680 con 0x7f45dc102730 2024-03-11T19:51:55.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.838+0000 7f45e3e00700 1 -- 172.21.15.97:0/512696585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45dc1a2410 con 0x7f45dc102730 2024-03-11T19:51:55.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.839+0000 7f45e3e00700 1 -- 172.21.15.97:0/512696585 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45dc0fdcc0 con 0x7f45dc102730 2024-03-11T19:51:55.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.841+0000 7f45da000700 1 -- 172.21.15.97:0/512696585 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f45d0010910 con 0x7f45dc102730 2024-03-11T19:51:55.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.842+0000 7f45da000700 1 --2- 172.21.15.97:0/512696585 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f45c40a1ac0 0x7f45c40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:51:55.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.842+0000 7f45da000700 1 -- 172.21.15.97:0/512696585 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4533+0+0 (secure 0 0 0) 0x7f45d0014070 con 0x7f45dc102730 2024-03-11T19:51:55.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.843+0000 7f45e0a00700 1 --2- 172.21.15.97:0/512696585 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f45c40a1ac0 0x7f45c40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:51:55.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.843+0000 7f45e0a00700 1 --2- 172.21.15.97:0/512696585 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f45c40a1ac0 0x7f45c40a3f80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f45c800b6f0 tx=0x7f45c800b7c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:51:55.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:55.845+0000 7f45da000700 1 -- 172.21.15.97:0/512696585 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f45d0096c60 con 0x7f45dc102730 2024-03-11T19:51:56.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:51:56.087+0000 7f45e3e00700 1 -- 172.21.15.97:0/512696585 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f45dc02d920 con 0x7f45c40a1ac0 2024-03-11T19:51:56.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:56 smithi097 ceph-mon[31898]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T19:51:56.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:56 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:51:56.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:56 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:51:56.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:56 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:56.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:56 smithi114 ceph-mon[36376]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T19:51:56.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:56 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:51:56.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:56 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:51:56.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:56 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:56.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:56 smithi161 ceph-mon[36332]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T19:51:56.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:56 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:51:56.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:56 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:51:56.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:56 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:51:57.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:57 smithi097 ceph-mon[31898]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:51:57.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:57 smithi097 ceph-mon[31898]: pgmap v158: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-11T19:51:57.691 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:57 smithi161 ceph-mon[36332]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:51:57.691 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:57 smithi161 ceph-mon[36332]: pgmap v158: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-11T19:51:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:57 smithi114 ceph-mon[36376]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:51:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:57 smithi114 ceph-mon[36376]: pgmap v158: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-11T19:51:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:51:59 smithi114 ceph-mon[36376]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:51:59.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:51:59 smithi161 ceph-mon[36332]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:51:59.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:51:59 smithi097 ceph-mon[31898]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:52:00.740 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:00 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2228142833' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "609b0dd7-7cff-4f8e-b656-2a11e7652150"}]: dispatch 2024-03-11T19:52:00.741 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:00 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "609b0dd7-7cff-4f8e-b656-2a11e7652150"}]: dispatch 2024-03-11T19:52:00.741 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:00 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "609b0dd7-7cff-4f8e-b656-2a11e7652150"}]': finished 2024-03-11T19:52:00.741 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:00 smithi097 ceph-mon[31898]: osdmap e43: 7 total, 6 up, 7 in 2024-03-11T19:52:00.741 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:00 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2228142833' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "609b0dd7-7cff-4f8e-b656-2a11e7652150"}]: dispatch 2024-03-11T19:52:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:00 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "609b0dd7-7cff-4f8e-b656-2a11e7652150"}]: dispatch 2024-03-11T19:52:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:00 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "609b0dd7-7cff-4f8e-b656-2a11e7652150"}]': finished 2024-03-11T19:52:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:00 smithi114 ceph-mon[36376]: osdmap e43: 7 total, 6 up, 7 in 2024-03-11T19:52:00.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:00.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:00 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2228142833' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "609b0dd7-7cff-4f8e-b656-2a11e7652150"}]: dispatch 2024-03-11T19:52:00.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:00 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "609b0dd7-7cff-4f8e-b656-2a11e7652150"}]: dispatch 2024-03-11T19:52:00.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:00 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "609b0dd7-7cff-4f8e-b656-2a11e7652150"}]': finished 2024-03-11T19:52:00.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:00 smithi161 ceph-mon[36332]: osdmap e43: 7 total, 6 up, 7 in 2024-03-11T19:52:00.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:01 smithi114 ceph-mon[36376]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:01.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:01 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/4179278364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:52:01.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:01 smithi161 ceph-mon[36332]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:01.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:01 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/4179278364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:52:01.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:01 smithi097 ceph-mon[31898]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:01.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:01 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/4179278364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:52:03.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:03 smithi161 ceph-mon[36332]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:03.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:03 smithi097 ceph-mon[31898]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:03.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:03 smithi114 ceph-mon[36376]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:05.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:05 smithi161 ceph-mon[36332]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:05.796 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:05 smithi114 ceph-mon[36376]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:05 smithi097 ceph-mon[31898]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:07.756 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:07 smithi097 ceph-mon[31898]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:07.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:07 smithi161 ceph-mon[36332]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:07.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:07 smithi114 ceph-mon[36376]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:09.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:09 smithi161 ceph-mon[36332]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:09.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:09 smithi097 ceph-mon[31898]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:09.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:09 smithi114 ceph-mon[36376]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T19:52:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:10.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:10 smithi161 ceph-mon[36332]: Deploying daemon osd.6 on smithi097 2024-03-11T19:52:10.823 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T19:52:10.823 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:10.823 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:10 smithi097 ceph-mon[31898]: Deploying daemon osd.6 on smithi097 2024-03-11T19:52:10.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T19:52:10.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:10.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:10 smithi114 ceph-mon[36376]: Deploying daemon osd.6 on smithi097 2024-03-11T19:52:11.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:11 smithi097 ceph-mon[31898]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:11.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:11 smithi161 ceph-mon[36332]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:11.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:11 smithi114 ceph-mon[36376]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:13.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:13 smithi097 ceph-mon[31898]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:13.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:13 smithi161 ceph-mon[36332]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:13.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:13 smithi114 ceph-mon[36376]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:15 smithi161 ceph-mon[36332]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:15.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:15 smithi097 ceph-mon[31898]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:15.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:15 smithi114 ceph-mon[36376]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:17.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:17 smithi161 ceph-mon[36332]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:17.877 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:17 smithi097 ceph-mon[31898]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:17 smithi114 ceph-mon[36376]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:19.418 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:19.419 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:52:19.419 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:19.419 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:19 smithi097 ceph-mon[31898]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:19.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:19.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:52:19.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:19.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:19 smithi161 ceph-mon[36332]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:19.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:52:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:19 smithi114 ceph-mon[36376]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:21.533 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:21 smithi097 ceph-mon[31898]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:21.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:21 smithi114 ceph-mon[36376]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:21.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:21 smithi161 ceph-mon[36332]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:23.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:52:23.291+0000 7f45da000700 1 -- 172.21.15.97:0/512696585 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f45dc02d920 con 0x7f45c40a1ac0 2024-03-11T19:52:23.296 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 6 on host 'smithi097' 2024-03-11T19:52:23.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:52:23.295+0000 7f45e3e00700 1 -- 172.21.15.97:0/512696585 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f45c40a1ac0 msgr2=0x7f45c40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:52:23.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:52:23.295+0000 7f45e3e00700 1 --2- 172.21.15.97:0/512696585 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f45c40a1ac0 0x7f45c40a3f80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f45c800b6f0 tx=0x7f45c800b7c0 comp rx=0 tx=0).stop 2024-03-11T19:52:23.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:52:23.295+0000 7f45e3e00700 1 -- 172.21.15.97:0/512696585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45dc102730 msgr2=0x7f45dc102090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:52:23.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:52:23.295+0000 7f45e3e00700 1 --2- 172.21.15.97:0/512696585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45dc102730 0x7f45dc102090 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f45d000e9a0 tx=0x7f45d000ecb0 comp rx=0 tx=0).stop 2024-03-11T19:52:23.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:52:23.295+0000 7f45e3e00700 1 -- 172.21.15.97:0/512696585 shutdown_connections 2024-03-11T19:52:23.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:52:23.295+0000 7f45e3e00700 1 --2- 172.21.15.97:0/512696585 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f45c40a1ac0 0x7f45c40a3f80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:23.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:52:23.295+0000 7f45e3e00700 1 --2- 172.21.15.97:0/512696585 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45dc107990 0x7f45dc100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:23.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:52:23.295+0000 7f45e3e00700 1 --2- 172.21.15.97:0/512696585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45dc102730 0x7f45dc102090 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:23.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:52:23.295+0000 7f45e3e00700 1 --2- 172.21.15.97:0/512696585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f45dc105060 0x7f45dc1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:23.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:52:23.295+0000 7f45e3e00700 1 -- 172.21.15.97:0/512696585 >> 172.21.15.97:0/512696585 conn(0x7f45dc0fc140 msgr2=0x7f45dc1031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:52:23.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:52:23.295+0000 7f45e3e00700 1 -- 172.21.15.97:0/512696585 shutdown_connections 2024-03-11T19:52:23.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:52:23.296+0000 7f45e3e00700 1 -- 172.21.15.97:0/512696585 wait complete. 2024-03-11T19:52:23.367 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:23.368 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:23.368 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:23 smithi097 ceph-mon[31898]: from='osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T19:52:23.368 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:23 smithi097 ceph-mon[31898]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:23.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:23.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:23.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:23 smithi114 ceph-mon[36376]: from='osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T19:52:23.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:23 smithi114 ceph-mon[36376]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:23.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:23.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:23.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:23 smithi161 ceph-mon[36332]: from='osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T19:52:23.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:23 smithi161 ceph-mon[36332]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:24.139 DEBUG:teuthology.orchestra.run.smithi097:osd.6> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.6.service 2024-03-11T19:52:24.143 INFO:tasks.cephadm:Deploying osd.7 on smithi114 with /dev/vg_nvme/lv_2... 2024-03-11T19:52:24.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T19:52:24.375 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:24 smithi097 ceph-mon[31898]: from='osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T19:52:24.375 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:24 smithi097 ceph-mon[31898]: osdmap e44: 7 total, 6 up, 7 in 2024-03-11T19:52:24.375 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:24 smithi097 ceph-mon[31898]: from='osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:52:24.375 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:24 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:24.375 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:24 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:24.375 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:24 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:24.387 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:52:24.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:24 smithi114 ceph-mon[36376]: from='osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T19:52:24.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:24 smithi114 ceph-mon[36376]: osdmap e44: 7 total, 6 up, 7 in 2024-03-11T19:52:24.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:24 smithi114 ceph-mon[36376]: from='osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:52:24.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:24 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:24.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:24 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:24.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:24 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:24.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:24 smithi161 ceph-mon[36332]: from='osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T19:52:24.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:24 smithi161 ceph-mon[36332]: osdmap e44: 7 total, 6 up, 7 in 2024-03-11T19:52:24.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:24 smithi161 ceph-mon[36332]: from='osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:52:24.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:24 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:24.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:24 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:24.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:24 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:25.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:25 smithi097 ceph-mon[31898]: purged_snaps scrub starts 2024-03-11T19:52:25.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:25 smithi097 ceph-mon[31898]: purged_snaps scrub ok 2024-03-11T19:52:25.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:25 smithi097 ceph-mon[31898]: from='osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T19:52:25.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:25 smithi097 ceph-mon[31898]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T19:52:25.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:25 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:25.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:25 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:25.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:25 smithi097 ceph-mon[31898]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:25.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:25 smithi114 ceph-mon[36376]: purged_snaps scrub starts 2024-03-11T19:52:25.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:25 smithi114 ceph-mon[36376]: purged_snaps scrub ok 2024-03-11T19:52:25.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:25 smithi114 ceph-mon[36376]: from='osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T19:52:25.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:25 smithi114 ceph-mon[36376]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T19:52:25.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:25 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:25.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:25 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:25.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:25 smithi114 ceph-mon[36376]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:25.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:25 smithi161 ceph-mon[36332]: purged_snaps scrub starts 2024-03-11T19:52:25.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:25 smithi161 ceph-mon[36332]: purged_snaps scrub ok 2024-03-11T19:52:25.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:25 smithi161 ceph-mon[36332]: from='osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T19:52:25.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:25 smithi161 ceph-mon[36332]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T19:52:25.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:25 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:25.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:25 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:25.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:25 smithi161 ceph-mon[36332]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T19:52:26.363 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:26 smithi114 ceph-mon[36376]: osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517] boot 2024-03-11T19:52:26.363 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:26 smithi114 ceph-mon[36376]: osdmap e46: 7 total, 7 up, 7 in 2024-03-11T19:52:26.363 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:26 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:26.374 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:26 smithi097 ceph-mon[31898]: osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517] boot 2024-03-11T19:52:26.375 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:26 smithi097 ceph-mon[31898]: osdmap e46: 7 total, 7 up, 7 in 2024-03-11T19:52:26.375 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:26 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:26.422 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T19:52:26.422 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T19:52:26.422 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-03-11T19:52:26.422 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-03-11T19:52:26.422 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127265 s, 824 MB/s 2024-03-11T19:52:26.422 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-03-11T19:52:26.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:26 smithi161 ceph-mon[36332]: osd.6 [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517] boot 2024-03-11T19:52:26.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:26 smithi161 ceph-mon[36332]: osdmap e46: 7 total, 7 up, 7 in 2024-03-11T19:52:26.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:26 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:52:27.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi114:vg_nvme/lv_2 2024-03-11T19:52:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:52:27.324 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:27 smithi114 ceph-mon[36376]: osdmap e47: 7 total, 7 up, 7 in 2024-03-11T19:52:27.324 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:27 smithi114 ceph-mon[36376]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:27.441 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:27 smithi161 ceph-mon[36332]: osdmap e47: 7 total, 7 up, 7 in 2024-03-11T19:52:27.442 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:27 smithi161 ceph-mon[36332]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:27.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:27 smithi097 ceph-mon[31898]: osdmap e47: 7 total, 7 up, 7 in 2024-03-11T19:52:27.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:27 smithi097 ceph-mon[31898]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:28.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.934+0000 7f21eb084700 1 -- 172.21.15.114:0/2070739067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21e4107aa0 msgr2=0x7f21e4109e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:52:28.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.934+0000 7f21eb084700 1 --2- 172.21.15.114:0/2070739067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21e4107aa0 0x7f21e4109e90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f21dc00b0a0 tx=0x7f21dc00b3b0 comp rx=0 tx=0).stop 2024-03-11T19:52:28.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.934+0000 7f21eb084700 1 -- 172.21.15.114:0/2070739067 shutdown_connections 2024-03-11T19:52:28.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.934+0000 7f21eb084700 1 --2- 172.21.15.114:0/2070739067 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21e4102840 0x7f21e4104c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:28.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.934+0000 7f21eb084700 1 --2- 172.21.15.114:0/2070739067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21e4107aa0 0x7f21e4109e90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:28.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.934+0000 7f21eb084700 1 --2- 172.21.15.114:0/2070739067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21e4105170 0x7f21e4107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:28.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.934+0000 7f21eb084700 1 -- 172.21.15.114:0/2070739067 >> 172.21.15.114:0/2070739067 conn(0x7f21e40fc1d0 msgr2=0x7f21e40fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:52:28.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.934+0000 7f21eb084700 1 -- 172.21.15.114:0/2070739067 shutdown_connections 2024-03-11T19:52:28.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.934+0000 7f21eb084700 1 -- 172.21.15.114:0/2070739067 wait complete. 2024-03-11T19:52:28.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.935+0000 7f21eb084700 1 Processor -- start 2024-03-11T19:52:28.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.935+0000 7f21eb084700 1 -- start start 2024-03-11T19:52:28.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.935+0000 7f21eb084700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21e4102840 0x7f21e4197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:52:28.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.935+0000 7f21eb084700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21e4105170 0x7f21e4198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:52:28.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.935+0000 7f21eb084700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21e4107aa0 0x7f21e419f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:52:28.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.935+0000 7f21eb084700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e419f890 con 0x7f21e4105170 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.935+0000 7f21eb084700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e419f9d0 con 0x7f21e4107aa0 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.935+0000 7f21eb084700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e419fb90 con 0x7f21e4102840 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.935+0000 7f21e9621700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21e4107aa0 0x7f21e419f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.935+0000 7f21e9621700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21e4107aa0 0x7f21e419f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46182/0 (socket says 172.21.15.114:46182) 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.935+0000 7f21e9621700 1 -- 172.21.15.114:0/4145586976 learned_addr learned my addr 172.21.15.114:0/4145586976 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.935+0000 7f21e8e20700 1 --2- 172.21.15.114:0/4145586976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21e4102840 0x7f21e4197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.936+0000 7f21e9621700 1 -- 172.21.15.114:0/4145586976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21e4102840 msgr2=0x7f21e4197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.936+0000 7f21e9621700 1 --2- 172.21.15.114:0/4145586976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21e4102840 0x7f21e4197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.936+0000 7f21e9621700 1 -- 172.21.15.114:0/4145586976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21e4105170 msgr2=0x7f21e4198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.936+0000 7f21e9621700 1 --2- 172.21.15.114:0/4145586976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21e4105170 0x7f21e4198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.936+0000 7f21e9621700 1 -- 172.21.15.114:0/4145586976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc009d00 con 0x7f21e4107aa0 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.936+0000 7f21e9621700 1 --2- 172.21.15.114:0/4145586976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21e4107aa0 0x7f21e419f2c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0094c0 tx=0x7f21dc0094f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.936+0000 7f21e1ffb700 1 -- 172.21.15.114:0/4145586976 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc00e040 con 0x7f21e4107aa0 2024-03-11T19:52:28.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.936+0000 7f21eb084700 1 -- 172.21.15.114:0/4145586976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e419fe10 con 0x7f21e4107aa0 2024-03-11T19:52:28.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.936+0000 7f21e1ffb700 1 -- 172.21.15.114:0/4145586976 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21dc01d960 con 0x7f21e4107aa0 2024-03-11T19:52:28.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.936+0000 7f21e1ffb700 1 -- 172.21.15.114:0/4145586976 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc01bac0 con 0x7f21e4107aa0 2024-03-11T19:52:28.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.936+0000 7f21eb084700 1 -- 172.21.15.114:0/4145586976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e41a0290 con 0x7f21e4107aa0 2024-03-11T19:52:28.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.937+0000 7f21eb084700 1 -- 172.21.15.114:0/4145586976 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e40fdd50 con 0x7f21e4107aa0 2024-03-11T19:52:28.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.938+0000 7f21e1ffb700 1 -- 172.21.15.114:0/4145586976 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f21dc01dad0 con 0x7f21e4107aa0 2024-03-11T19:52:28.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.939+0000 7f21e1ffb700 1 --2- 172.21.15.114:0/4145586976 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f21d40a1ac0 0x7f21d40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:52:28.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.939+0000 7f21e1ffb700 1 -- 172.21.15.114:0/4145586976 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f21dc01de40 con 0x7f21e4107aa0 2024-03-11T19:52:28.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.939+0000 7f21e8e20700 1 --2- 172.21.15.114:0/4145586976 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f21d40a1ac0 0x7f21d40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:52:28.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.940+0000 7f21e8e20700 1 --2- 172.21.15.114:0/4145586976 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f21d40a1ac0 0x7f21d40a3f80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f21e4199220 tx=0x7f21d8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:52:28.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:28.943+0000 7f21e1ffb700 1 -- 172.21.15.114:0/4145586976 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f21dc098d70 con 0x7f21e4107aa0 2024-03-11T19:52:29.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:29.143+0000 7f21eb084700 1 -- 172.21.15.114:0/4145586976 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f21e402d8e0 con 0x7f21d40a1ac0 2024-03-11T19:52:29.788 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:29 smithi114 ceph-mon[36376]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:29.788 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:29.788 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:29.788 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:52:29.788 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:29.788 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:29.788 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:52:29.788 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:29.788 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:29.789 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:29.789 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:29.789 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:52:29.789 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:29.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:29 smithi161 ceph-mon[36332]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:29.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:29.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:29.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:52:29.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:29.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:29.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:52:29.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:29.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:29.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:29.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:29.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:52:29.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:29.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:29 smithi097 ceph-mon[31898]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:29.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:29.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:29.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:52:29.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:29.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:29.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:52:29.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:29.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:29.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:29.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:29.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:52:29.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:30.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:30 smithi161 ceph-mon[36332]: Detected new or changed devices on smithi097 2024-03-11T19:52:30.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:30 smithi161 ceph-mon[36332]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:52:30.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:30 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi097 to 5697M 2024-03-11T19:52:30.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:30 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:52:30.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:30 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:30.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:30 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:52:30.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:30 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:30.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:30 smithi114 ceph-mon[36376]: Detected new or changed devices on smithi097 2024-03-11T19:52:30.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:30 smithi114 ceph-mon[36376]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:52:30.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:30 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi097 to 5697M 2024-03-11T19:52:30.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:30 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:52:30.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:30 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:30.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:30 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:52:30.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:30 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:30.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:30 smithi097 ceph-mon[31898]: Detected new or changed devices on smithi097 2024-03-11T19:52:30.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:30 smithi097 ceph-mon[31898]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:52:30.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:30 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi097 to 5697M 2024-03-11T19:52:30.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:30 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:52:30.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:30 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:30.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:30 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:52:30.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:30 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:31.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:31 smithi161 ceph-mon[36332]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:31.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:31 smithi097 ceph-mon[31898]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:31 smithi114 ceph-mon[36376]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:32.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:32 smithi114 ceph-mon[36376]: from='client.? 172.21.15.114:0/1096282925' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a0d4de9-3628-4421-91b0-0c8cb7cdcf93"}]: dispatch 2024-03-11T19:52:32.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:32 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a0d4de9-3628-4421-91b0-0c8cb7cdcf93"}]: dispatch 2024-03-11T19:52:32.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:32 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a0d4de9-3628-4421-91b0-0c8cb7cdcf93"}]': finished 2024-03-11T19:52:32.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:32 smithi114 ceph-mon[36376]: osdmap e48: 8 total, 7 up, 8 in 2024-03-11T19:52:32.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:32 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:52:32.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:32 smithi161 ceph-mon[36332]: from='client.? 172.21.15.114:0/1096282925' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a0d4de9-3628-4421-91b0-0c8cb7cdcf93"}]: dispatch 2024-03-11T19:52:32.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:32 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a0d4de9-3628-4421-91b0-0c8cb7cdcf93"}]: dispatch 2024-03-11T19:52:32.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:32 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a0d4de9-3628-4421-91b0-0c8cb7cdcf93"}]': finished 2024-03-11T19:52:32.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:32 smithi161 ceph-mon[36332]: osdmap e48: 8 total, 7 up, 8 in 2024-03-11T19:52:32.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:32 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:52:32.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:32 smithi097 ceph-mon[31898]: from='client.? 172.21.15.114:0/1096282925' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a0d4de9-3628-4421-91b0-0c8cb7cdcf93"}]: dispatch 2024-03-11T19:52:32.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:32 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a0d4de9-3628-4421-91b0-0c8cb7cdcf93"}]: dispatch 2024-03-11T19:52:32.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:32 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a0d4de9-3628-4421-91b0-0c8cb7cdcf93"}]': finished 2024-03-11T19:52:32.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:32 smithi097 ceph-mon[31898]: osdmap e48: 8 total, 7 up, 8 in 2024-03-11T19:52:32.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:32 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:52:33.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:33 smithi161 ceph-mon[36332]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:33.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:33 smithi161 ceph-mon[36332]: from='client.? 172.21.15.114:0/3117590792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:52:33.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:33 smithi097 ceph-mon[31898]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:33.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:33 smithi097 ceph-mon[31898]: from='client.? 172.21.15.114:0/3117590792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:52:33.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:33 smithi114 ceph-mon[36376]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:33.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:33 smithi114 ceph-mon[36376]: from='client.? 172.21.15.114:0/3117590792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:52:35.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:35 smithi161 ceph-mon[36332]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:35.795 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:35 smithi114 ceph-mon[36376]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:35.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:35 smithi097 ceph-mon[31898]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:37.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:37 smithi161 ceph-mon[36332]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:37.799 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:37 smithi114 ceph-mon[36376]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:37.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:37 smithi097 ceph-mon[31898]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:39.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:39 smithi097 ceph-mon[31898]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:39.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T19:52:39.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:39.930 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:39 smithi114 ceph-mon[36376]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:39.931 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T19:52:39.931 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:40.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:39 smithi161 ceph-mon[36332]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:40.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T19:52:40.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:40.906 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:40 smithi114 ceph-mon[36376]: Deploying daemon osd.7 on smithi114 2024-03-11T19:52:40.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:40 smithi097 ceph-mon[31898]: Deploying daemon osd.7 on smithi114 2024-03-11T19:52:41.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:40 smithi161 ceph-mon[36332]: Deploying daemon osd.7 on smithi114 2024-03-11T19:52:41.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:41 smithi097 ceph-mon[31898]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:41.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:41 smithi114 ceph-mon[36376]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:42.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:41 smithi161 ceph-mon[36332]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:43.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:43 smithi097 ceph-mon[31898]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:43.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:43 smithi114 ceph-mon[36376]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:44.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:43 smithi161 ceph-mon[36332]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:45.862 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:45 smithi114 ceph-mon[36376]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:45.862 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:45.862 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:45.862 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:52:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:45 smithi097 ceph-mon[31898]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:45.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:52:46.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:45 smithi161 ceph-mon[36332]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:46.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:46.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:46.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:52:47.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:47 smithi097 ceph-mon[31898]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:47.815 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:47 smithi114 ceph-mon[36376]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:48.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:47 smithi161 ceph-mon[36332]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:48.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:48.530+0000 7f21e1ffb700 1 -- 172.21.15.114:0/4145586976 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f21e402d8e0 con 0x7f21d40a1ac0 2024-03-11T19:52:48.535 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 7 on host 'smithi114' 2024-03-11T19:52:48.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:48.532+0000 7f21eb084700 1 -- 172.21.15.114:0/4145586976 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f21d40a1ac0 msgr2=0x7f21d40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:52:48.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:48.532+0000 7f21eb084700 1 --2- 172.21.15.114:0/4145586976 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f21d40a1ac0 0x7f21d40a3f80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f21e4199220 tx=0x7f21d8009190 comp rx=0 tx=0).stop 2024-03-11T19:52:48.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:48.532+0000 7f21eb084700 1 -- 172.21.15.114:0/4145586976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21e4107aa0 msgr2=0x7f21e419f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:52:48.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:48.532+0000 7f21eb084700 1 --2- 172.21.15.114:0/4145586976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21e4107aa0 0x7f21e419f2c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0094c0 tx=0x7f21dc0094f0 comp rx=0 tx=0).stop 2024-03-11T19:52:48.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:48.532+0000 7f21eb084700 1 -- 172.21.15.114:0/4145586976 shutdown_connections 2024-03-11T19:52:48.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:48.532+0000 7f21eb084700 1 --2- 172.21.15.114:0/4145586976 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f21d40a1ac0 0x7f21d40a3f80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:48.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:48.532+0000 7f21eb084700 1 --2- 172.21.15.114:0/4145586976 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f21e4107aa0 0x7f21e419f2c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:48.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:48.532+0000 7f21eb084700 1 --2- 172.21.15.114:0/4145586976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21e4102840 0x7f21e4197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:48.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:48.532+0000 7f21eb084700 1 --2- 172.21.15.114:0/4145586976 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21e4105170 0x7f21e4198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:48.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:48.532+0000 7f21eb084700 1 -- 172.21.15.114:0/4145586976 >> 172.21.15.114:0/4145586976 conn(0x7f21e40fc1d0 msgr2=0x7f21e4103180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:52:48.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:48.532+0000 7f21eb084700 1 -- 172.21.15.114:0/4145586976 shutdown_connections 2024-03-11T19:52:48.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:52:48.532+0000 7f21eb084700 1 -- 172.21.15.114:0/4145586976 wait complete. 2024-03-11T19:52:49.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:48 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:48 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:48 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:49.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:48 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:52:49.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:48 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:48 smithi161 ceph-mon[36332]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:49.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:48 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:48 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:48 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:52:49.106 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.106 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.106 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:49.106 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:52:49.107 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.107 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:48 smithi114 ceph-mon[36376]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:49.107 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.107 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.107 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:52:49.134 DEBUG:teuthology.orchestra.run.smithi114:osd.7> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.7.service 2024-03-11T19:52:49.137 INFO:tasks.cephadm:Deploying osd.8 on smithi161 with /dev/vg_nvme/lv_2... 2024-03-11T19:52:49.137 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T19:52:49.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:49.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:52:49.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:48 smithi097 ceph-mon[31898]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:49.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:49.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:52:49.388 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:52:50.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:49 smithi161 ceph-mon[36332]: from='osd.7 [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T19:52:50.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:49 smithi161 ceph-mon[36332]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T19:52:50.129 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:49 smithi114 ceph-mon[36376]: from='osd.7 [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T19:52:50.129 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:49 smithi114 ceph-mon[36376]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T19:52:50.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:49 smithi097 ceph-mon[31898]: from='osd.7 [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T19:52:50.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:49 smithi097 ceph-mon[31898]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T19:52:51.114 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:50 smithi161 ceph-mon[36332]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T19:52:51.114 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:50 smithi161 ceph-mon[36332]: from='osd.7 [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:52:51.115 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:50 smithi161 ceph-mon[36332]: osdmap e49: 8 total, 7 up, 8 in 2024-03-11T19:52:51.115 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:50 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:52:51.115 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:50 smithi161 ceph-mon[36332]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:52:51.115 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:50 smithi161 ceph-mon[36332]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:51.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:50 smithi097 ceph-mon[31898]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T19:52:51.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:50 smithi097 ceph-mon[31898]: from='osd.7 [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:52:51.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:50 smithi097 ceph-mon[31898]: osdmap e49: 8 total, 7 up, 8 in 2024-03-11T19:52:51.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:50 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:52:51.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:50 smithi097 ceph-mon[31898]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:52:51.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:50 smithi097 ceph-mon[31898]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:51.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:50 smithi114 ceph-mon[36376]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T19:52:51.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:50 smithi114 ceph-mon[36376]: from='osd.7 [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:52:51.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:50 smithi114 ceph-mon[36376]: osdmap e49: 8 total, 7 up, 8 in 2024-03-11T19:52:51.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:50 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:52:51.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:50 smithi114 ceph-mon[36376]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:52:51.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:50 smithi114 ceph-mon[36376]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:51.482 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T19:52:51.482 INFO:teuthology.orchestra.run.smithi161.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T19:52:51.482 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10+0 records in 2024-03-11T19:52:51.483 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2024-03-11T19:52:51.483 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0150981 s, 695 MB/s 2024-03-11T19:52:51.483 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping successful for: 2024-03-11T19:52:51.703 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 19:52:51 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T19:52:51.519+0000 7f6257508700 -1 osd.7 0 waiting for initial osdmap 2024-03-11T19:52:51.703 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 19:52:51 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T19:52:51.528+0000 7f624f2f3700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:52:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:51 smithi114 ceph-mon[36376]: purged_snaps scrub starts 2024-03-11T19:52:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:51 smithi114 ceph-mon[36376]: purged_snaps scrub ok 2024-03-11T19:52:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:51 smithi114 ceph-mon[36376]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-11T19:52:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:51 smithi114 ceph-mon[36376]: osdmap e50: 8 total, 7 up, 8 in 2024-03-11T19:52:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:51 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:52:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:51 smithi114 ceph-mon[36376]: from='osd.7 ' entity='osd.7' 2024-03-11T19:52:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:51 smithi114 ceph-mon[36376]: osd.7 [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963] boot 2024-03-11T19:52:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:51 smithi114 ceph-mon[36376]: osdmap e51: 8 total, 8 up, 8 in 2024-03-11T19:52:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:51 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:52:52.143 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:51 smithi161 ceph-mon[36332]: purged_snaps scrub starts 2024-03-11T19:52:52.144 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:51 smithi161 ceph-mon[36332]: purged_snaps scrub ok 2024-03-11T19:52:52.144 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:51 smithi161 ceph-mon[36332]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-11T19:52:52.144 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:51 smithi161 ceph-mon[36332]: osdmap e50: 8 total, 7 up, 8 in 2024-03-11T19:52:52.144 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:51 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:52:52.144 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:51 smithi161 ceph-mon[36332]: from='osd.7 ' entity='osd.7' 2024-03-11T19:52:52.144 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:51 smithi161 ceph-mon[36332]: osd.7 [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963] boot 2024-03-11T19:52:52.144 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:51 smithi161 ceph-mon[36332]: osdmap e51: 8 total, 8 up, 8 in 2024-03-11T19:52:52.144 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:51 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:52:52.170 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi161:vg_nvme/lv_2 2024-03-11T19:52:52.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:51 smithi097 ceph-mon[31898]: purged_snaps scrub starts 2024-03-11T19:52:52.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:51 smithi097 ceph-mon[31898]: purged_snaps scrub ok 2024-03-11T19:52:52.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:51 smithi097 ceph-mon[31898]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-11T19:52:52.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:51 smithi097 ceph-mon[31898]: osdmap e50: 8 total, 7 up, 8 in 2024-03-11T19:52:52.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:51 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:52:52.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:51 smithi097 ceph-mon[31898]: from='osd.7 ' entity='osd.7' 2024-03-11T19:52:52.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:51 smithi097 ceph-mon[31898]: osd.7 [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963] boot 2024-03-11T19:52:52.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:51 smithi097 ceph-mon[31898]: osdmap e51: 8 total, 8 up, 8 in 2024-03-11T19:52:52.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:51 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:52:52.409 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:52:53.192 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:53 smithi161 ceph-mon[36332]: Detected new or changed devices on smithi114 2024-03-11T19:52:53.192 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:53.192 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:53.192 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:53.193 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:53.193 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:53.193 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:53 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi114 to 5693M 2024-03-11T19:52:53.193 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:53.193 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:53.193 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:52:53.193 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:53 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:53.193 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:53 smithi161 ceph-mon[36332]: pgmap v195: 1 pgs: 1 unknown; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:53.193 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:53 smithi161 ceph-mon[36332]: osdmap e52: 8 total, 8 up, 8 in 2024-03-11T19:52:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:53 smithi097 ceph-mon[31898]: Detected new or changed devices on smithi114 2024-03-11T19:52:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:53 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi114 to 5693M 2024-03-11T19:52:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:52:53.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:53 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:53.424 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:53 smithi097 ceph-mon[31898]: pgmap v195: 1 pgs: 1 unknown; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:53.424 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:53 smithi097 ceph-mon[31898]: osdmap e52: 8 total, 8 up, 8 in 2024-03-11T19:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:53 smithi114 ceph-mon[36376]: Detected new or changed devices on smithi114 2024-03-11T19:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:53 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi114 to 5693M 2024-03-11T19:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:53.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:52:53.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:53 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:52:53.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:53 smithi114 ceph-mon[36376]: pgmap v195: 1 pgs: 1 unknown; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T19:52:53.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:53 smithi114 ceph-mon[36376]: osdmap e52: 8 total, 8 up, 8 in 2024-03-11T19:52:54.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.079+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245376121 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed0101da0 msgr2=0x7f8ed0102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:52:54.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.079+0000 7f8ed85df700 1 --2- 172.21.15.161:0/2245376121 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed0101da0 0x7f8ed0102200 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0009990 tx=0x7f8ec0009ca0 comp rx=0 tx=0).stop 2024-03-11T19:52:54.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245376121 shutdown_connections 2024-03-11T19:52:54.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 --2- 172.21.15.161:0/2245376121 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed0100ba0 0x7f8ed0100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:54.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 --2- 172.21.15.161:0/2245376121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ed0102740 0x7f8ed0108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:54.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 --2- 172.21.15.161:0/2245376121 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed0101da0 0x7f8ed0102200 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:54.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245376121 >> 172.21.15.161:0/2245376121 conn(0x7f8ed00fc160 msgr2=0x7f8ed00fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:52:54.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245376121 shutdown_connections 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245376121 wait complete. 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 Processor -- start 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 -- start start 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed0100ba0 0x7f8ed0197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ed0101da0 0x7f8ed0198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed0102740 0x7f8ed019f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed019f830 con 0x7f8ed0102740 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed019f970 con 0x7f8ed0101da0 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed85df700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed019fb30 con 0x7f8ed0100ba0 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed637b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed0100ba0 0x7f8ed0197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.080+0000 7f8ed637b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed0100ba0 0x7f8ed0197d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:54552/0 (socket says 172.21.15.161:54552) 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.081+0000 7f8ed637b700 1 -- 172.21.15.161:0/2245366751 learned_addr learned my addr 172.21.15.161:0/2245366751 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.081+0000 7f8ed637b700 1 -- 172.21.15.161:0/2245366751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ed0101da0 msgr2=0x7f8ed0198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:52:54.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.081+0000 7f8ed5b7a700 1 --2- 172.21.15.161:0/2245366751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ed0101da0 0x7f8ed0198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:52:54.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.081+0000 7f8ed637b700 1 --2- 172.21.15.161:0/2245366751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ed0101da0 0x7f8ed0198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:54.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.081+0000 7f8ed637b700 1 -- 172.21.15.161:0/2245366751 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed0102740 msgr2=0x7f8ed019f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:52:54.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.081+0000 7f8ed637b700 1 --2- 172.21.15.161:0/2245366751 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed0102740 0x7f8ed019f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:52:54.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.081+0000 7f8ed637b700 1 -- 172.21.15.161:0/2245366751 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec00095f0 con 0x7f8ed0100ba0 2024-03-11T19:52:54.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.081+0000 7f8ed637b700 1 --2- 172.21.15.161:0/2245366751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed0100ba0 0x7f8ed0197d30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00b500 tx=0x7f8ecc00b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:52:54.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.081+0000 7f8ec77fe700 1 -- 172.21.15.161:0/2245366751 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc011830 con 0x7f8ed0100ba0 2024-03-11T19:52:54.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.081+0000 7f8ec77fe700 1 -- 172.21.15.161:0/2245366751 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ecc011e70 con 0x7f8ed0100ba0 2024-03-11T19:52:54.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.081+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245366751 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed019fe10 con 0x7f8ed0100ba0 2024-03-11T19:52:54.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.081+0000 7f8ec77fe700 1 -- 172.21.15.161:0/2245366751 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc00f2e0 con 0x7f8ed0100ba0 2024-03-11T19:52:54.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.081+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245366751 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed01a02a0 con 0x7f8ed0100ba0 2024-03-11T19:52:54.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.082+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245366751 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed010c580 con 0x7f8ed0100ba0 2024-03-11T19:52:54.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.083+0000 7f8ec77fe700 1 -- 172.21.15.161:0/2245366751 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8ecc0105f0 con 0x7f8ed0100ba0 2024-03-11T19:52:54.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.084+0000 7f8ec77fe700 1 --2- 172.21.15.161:0/2245366751 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8ebc0a1ac0 0x7f8ebc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:52:54.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.084+0000 7f8ec77fe700 1 -- 172.21.15.161:0/2245366751 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f8ecc0cc870 con 0x7f8ed0100ba0 2024-03-11T19:52:54.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.085+0000 7f8ed5b7a700 1 --2- 172.21.15.161:0/2245366751 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8ebc0a1ac0 0x7f8ebc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:52:54.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.085+0000 7f8ed5b7a700 1 --2- 172.21.15.161:0/2245366751 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8ebc0a1ac0 0x7f8ebc0a3f80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00095c0 tx=0x7f8ec0009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:52:54.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.087+0000 7f8ec77fe700 1 -- 172.21.15.161:0/2245366751 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8ecc0965c0 con 0x7f8ed0100ba0 2024-03-11T19:52:54.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:52:54.285+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245366751 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ed00fe340 con 0x7f8ebc0a1ac0 2024-03-11T19:52:54.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:54 smithi161 ceph-mon[36332]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-11T19:52:54.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:54 smithi161 ceph-mon[36332]: osdmap e53: 8 total, 8 up, 8 in 2024-03-11T19:52:54.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:54 smithi097 ceph-mon[31898]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-11T19:52:54.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:54 smithi097 ceph-mon[31898]: osdmap e53: 8 total, 8 up, 8 in 2024-03-11T19:52:54.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:54 smithi114 ceph-mon[36376]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-11T19:52:54.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:54 smithi114 ceph-mon[36376]: osdmap e53: 8 total, 8 up, 8 in 2024-03-11T19:52:55.232 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:55 smithi161 ceph-mon[36332]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:52:55.233 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:52:55.233 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:52:55.233 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:55.233 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:55 smithi161 ceph-mon[36332]: pgmap v198: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:52:55.233 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:55 smithi161 ceph-mon[36332]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T19:52:55.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:55 smithi097 ceph-mon[31898]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:52:55.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:52:55.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:52:55.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:55.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:55 smithi097 ceph-mon[31898]: pgmap v198: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:52:55.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:55 smithi097 ceph-mon[31898]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T19:52:55.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:55 smithi114 ceph-mon[36376]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:52:55.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:52:55.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:52:55.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:52:55.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:55 smithi114 ceph-mon[36376]: pgmap v198: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:52:55.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:55 smithi114 ceph-mon[36376]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T19:52:57.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:56 smithi097 ceph-mon[31898]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T19:52:57.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:56 smithi097 ceph-mon[31898]: pgmap v201: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:52:57.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:56 smithi114 ceph-mon[36376]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T19:52:57.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:56 smithi114 ceph-mon[36376]: pgmap v201: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:52:57.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:56 smithi161 ceph-mon[36332]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T19:52:57.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:56 smithi161 ceph-mon[36332]: pgmap v201: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:52:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:57 smithi097 ceph-mon[31898]: from='client.? 172.21.15.161:0/402538354' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c0abbf6-d848-4911-beff-7b4d3ce7b330"}]: dispatch 2024-03-11T19:52:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:57 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c0abbf6-d848-4911-beff-7b4d3ce7b330"}]: dispatch 2024-03-11T19:52:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:57 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c0abbf6-d848-4911-beff-7b4d3ce7b330"}]': finished 2024-03-11T19:52:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:57 smithi097 ceph-mon[31898]: osdmap e56: 9 total, 8 up, 9 in 2024-03-11T19:52:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:57 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:52:58.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:57 smithi097 ceph-mon[31898]: from='client.? 172.21.15.161:0/4221737206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:52:58.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:57 smithi114 ceph-mon[36376]: from='client.? 172.21.15.161:0/402538354' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c0abbf6-d848-4911-beff-7b4d3ce7b330"}]: dispatch 2024-03-11T19:52:58.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:57 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c0abbf6-d848-4911-beff-7b4d3ce7b330"}]: dispatch 2024-03-11T19:52:58.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:57 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c0abbf6-d848-4911-beff-7b4d3ce7b330"}]': finished 2024-03-11T19:52:58.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:57 smithi114 ceph-mon[36376]: osdmap e56: 9 total, 8 up, 9 in 2024-03-11T19:52:58.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:57 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:52:58.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:57 smithi114 ceph-mon[36376]: from='client.? 172.21.15.161:0/4221737206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:52:58.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:57 smithi161 ceph-mon[36332]: from='client.? 172.21.15.161:0/402538354' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c0abbf6-d848-4911-beff-7b4d3ce7b330"}]: dispatch 2024-03-11T19:52:58.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:57 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c0abbf6-d848-4911-beff-7b4d3ce7b330"}]: dispatch 2024-03-11T19:52:58.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:57 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c0abbf6-d848-4911-beff-7b4d3ce7b330"}]': finished 2024-03-11T19:52:58.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:57 smithi161 ceph-mon[36332]: osdmap e56: 9 total, 8 up, 9 in 2024-03-11T19:52:58.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:57 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:52:58.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:57 smithi161 ceph-mon[36332]: from='client.? 172.21.15.161:0/4221737206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:52:59.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:58 smithi097 ceph-mon[31898]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:52:59.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:58 smithi114 ceph-mon[36376]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:52:59.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:58 smithi161 ceph-mon[36332]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:00.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:59 smithi097 ceph-mon[31898]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-11T19:53:00.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:52:59 smithi097 ceph-mon[31898]: Cluster is now healthy 2024-03-11T19:53:00.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:59 smithi114 ceph-mon[36376]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-11T19:53:00.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:52:59 smithi114 ceph-mon[36376]: Cluster is now healthy 2024-03-11T19:53:00.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:59 smithi161 ceph-mon[36332]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-11T19:53:00.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:52:59 smithi161 ceph-mon[36332]: Cluster is now healthy 2024-03-11T19:53:01.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:00 smithi161 ceph-mon[36332]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:01.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:00 smithi097 ceph-mon[31898]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:01.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:00 smithi114 ceph-mon[36376]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:03.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:03 smithi161 ceph-mon[36332]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:03.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:03 smithi097 ceph-mon[31898]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:03.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:03 smithi114 ceph-mon[36376]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:04.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:04 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-11T19:53:04.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:04 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:04.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:04 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-11T19:53:04.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:04 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:04.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:04 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-11T19:53:04.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:04 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:05.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:05 smithi161 ceph-mon[36332]: Deploying daemon osd.8 on smithi161 2024-03-11T19:53:05.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:05 smithi161 ceph-mon[36332]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:05.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:05 smithi114 ceph-mon[36376]: Deploying daemon osd.8 on smithi161 2024-03-11T19:53:05.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:05 smithi114 ceph-mon[36376]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:05 smithi097 ceph-mon[31898]: Deploying daemon osd.8 on smithi161 2024-03-11T19:53:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:05 smithi097 ceph-mon[31898]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:07.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:07 smithi161 ceph-mon[36332]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:07.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:07 smithi097 ceph-mon[31898]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:07.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:07 smithi114 ceph-mon[36376]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:09.551 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:09 smithi161 ceph-mon[36332]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:09.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:09 smithi097 ceph-mon[31898]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:09 smithi114 ceph-mon[36376]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:11.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:11.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:53:11.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:11.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:10 smithi114 ceph-mon[36376]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:11.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:11.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:53:11.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:11.290 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:10 smithi161 ceph-mon[36332]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:11.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:11.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:53:11.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:11.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:10 smithi097 ceph-mon[31898]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:13.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:53:13.568+0000 7f8ec77fe700 1 -- 172.21.15.161:0/2245366751 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8ed00fe340 con 0x7f8ebc0a1ac0 2024-03-11T19:53:13.574 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 8 on host 'smithi161' 2024-03-11T19:53:13.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:53:13.571+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245366751 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8ebc0a1ac0 msgr2=0x7f8ebc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:53:13.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:53:13.571+0000 7f8ed85df700 1 --2- 172.21.15.161:0/2245366751 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8ebc0a1ac0 0x7f8ebc0a3f80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00095c0 tx=0x7f8ec0009510 comp rx=0 tx=0).stop 2024-03-11T19:53:13.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:53:13.571+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245366751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed0100ba0 msgr2=0x7f8ed0197d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:53:13.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:53:13.571+0000 7f8ed85df700 1 --2- 172.21.15.161:0/2245366751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed0100ba0 0x7f8ed0197d30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00b500 tx=0x7f8ecc00b8c0 comp rx=0 tx=0).stop 2024-03-11T19:53:13.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:53:13.571+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245366751 shutdown_connections 2024-03-11T19:53:13.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:53:13.571+0000 7f8ed85df700 1 --2- 172.21.15.161:0/2245366751 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8ebc0a1ac0 0x7f8ebc0a3f80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:13.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:53:13.571+0000 7f8ed85df700 1 --2- 172.21.15.161:0/2245366751 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8ed0102740 0x7f8ed019f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:13.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:53:13.571+0000 7f8ed85df700 1 --2- 172.21.15.161:0/2245366751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed0100ba0 0x7f8ed0197d30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:13.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:53:13.571+0000 7f8ed85df700 1 --2- 172.21.15.161:0/2245366751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ed0101da0 0x7f8ed0198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:13.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:53:13.571+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245366751 >> 172.21.15.161:0/2245366751 conn(0x7f8ed00fc160 msgr2=0x7f8ed00fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:53:13.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:53:13.571+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245366751 shutdown_connections 2024-03-11T19:53:13.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:53:13.571+0000 7f8ed85df700 1 -- 172.21.15.161:0/2245366751 wait complete. 2024-03-11T19:53:13.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:13 smithi161 ceph-mon[36332]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:13.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:13 smithi097 ceph-mon[31898]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:13.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:13 smithi114 ceph-mon[36376]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:14.283 DEBUG:teuthology.orchestra.run.smithi161:osd.8> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.8.service 2024-03-11T19:53:14.285 INFO:tasks.cephadm:Deploying osd.9 on smithi097 with /dev/vg_nvme/lv_1... 2024-03-11T19:53:14.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T19:53:14.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:14 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:14.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:14 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:14.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:14 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:14.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:14 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:14.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:14 smithi097 ceph-mon[31898]: from='osd.8 [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T19:53:14.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:14 smithi097 ceph-mon[31898]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T19:53:14.731 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:53:14.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:14 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:14.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:14 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:14.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:14 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:14.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:14 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:14.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:14 smithi161 ceph-mon[36332]: from='osd.8 [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T19:53:14.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:14 smithi161 ceph-mon[36332]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T19:53:14.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:14 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:14.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:14 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:14.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:14 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:14.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:14 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:14.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:14 smithi114 ceph-mon[36376]: from='osd.8 [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T19:53:14.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:14 smithi114 ceph-mon[36376]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T19:53:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:15 smithi161 ceph-mon[36332]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:15 smithi161 ceph-mon[36332]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-11T19:53:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:15 smithi161 ceph-mon[36332]: osdmap e57: 9 total, 8 up, 9 in 2024-03-11T19:53:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:15 smithi161 ceph-mon[36332]: from='osd.8 [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:53:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:15 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:15.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:15 smithi161 ceph-mon[36332]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:53:15.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:15 smithi097 ceph-mon[31898]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:15.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:15 smithi097 ceph-mon[31898]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-11T19:53:15.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:15 smithi097 ceph-mon[31898]: osdmap e57: 9 total, 8 up, 9 in 2024-03-11T19:53:15.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:15 smithi097 ceph-mon[31898]: from='osd.8 [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:53:15.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:15 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:15.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:15 smithi097 ceph-mon[31898]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:53:15.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:15 smithi114 ceph-mon[36376]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:15 smithi114 ceph-mon[36376]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-11T19:53:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:15 smithi114 ceph-mon[36376]: osdmap e57: 9 total, 8 up, 9 in 2024-03-11T19:53:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:15 smithi114 ceph-mon[36376]: from='osd.8 [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:53:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:15 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:15 smithi114 ceph-mon[36376]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:53:16.702 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 19:53:16 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T19:53:16.376+0000 7fc973a58700 -1 osd.8 0 waiting for initial osdmap 2024-03-11T19:53:16.702 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 19:53:16 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T19:53:16.387+0000 7fc96b843700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:53:16.897 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:16 smithi097 ceph-mon[31898]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-03-11T19:53:16.898 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:16 smithi097 ceph-mon[31898]: osdmap e58: 9 total, 8 up, 9 in 2024-03-11T19:53:16.898 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:16 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:16.898 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:16 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:16.898 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:16 smithi097 ceph-mon[31898]: from='osd.8 ' entity='osd.8' 2024-03-11T19:53:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:16 smithi114 ceph-mon[36376]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-03-11T19:53:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:16 smithi114 ceph-mon[36376]: osdmap e58: 9 total, 8 up, 9 in 2024-03-11T19:53:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:16 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:16 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:16 smithi114 ceph-mon[36376]: from='osd.8 ' entity='osd.8' 2024-03-11T19:53:17.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:16 smithi161 ceph-mon[36332]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-03-11T19:53:17.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:16 smithi161 ceph-mon[36332]: osdmap e58: 9 total, 8 up, 9 in 2024-03-11T19:53:17.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:16 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:17.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:16 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:17.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:16 smithi161 ceph-mon[36332]: from='osd.8 ' entity='osd.8' 2024-03-11T19:53:17.735 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T19:53:17.735 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T19:53:17.735 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-03-11T19:53:17.735 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-03-11T19:53:17.735 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0381493 s, 275 MB/s 2024-03-11T19:53:17.735 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-03-11T19:53:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: purged_snaps scrub starts 2024-03-11T19:53:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: purged_snaps scrub ok 2024-03-11T19:53:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:17.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:17.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: osd.8 [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818] boot 2024-03-11T19:53:17.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: osdmap e59: 9 total, 9 up, 9 in 2024-03-11T19:53:17.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:17.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:17.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:17.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:17.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:17.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:17.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:17.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:17.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:53:17.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:17.997 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:17 smithi114 ceph-mon[36376]: osdmap e60: 9 total, 9 up, 9 in 2024-03-11T19:53:18.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: purged_snaps scrub starts 2024-03-11T19:53:18.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: purged_snaps scrub ok 2024-03-11T19:53:18.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:18.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:18.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: osd.8 [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818] boot 2024-03-11T19:53:18.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: osdmap e59: 9 total, 9 up, 9 in 2024-03-11T19:53:18.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:18.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:18.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:18.042 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:18.042 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:18.042 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:18.042 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:18.042 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:18.042 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:53:18.043 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:18.043 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:17 smithi161 ceph-mon[36332]: osdmap e60: 9 total, 9 up, 9 in 2024-03-11T19:53:18.118 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: purged_snaps scrub starts 2024-03-11T19:53:18.118 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: purged_snaps scrub ok 2024-03-11T19:53:18.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T19:53:18.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:18.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: osd.8 [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818] boot 2024-03-11T19:53:18.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: osdmap e59: 9 total, 9 up, 9 in 2024-03-11T19:53:18.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:53:18.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:18.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:18.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:18.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:18.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:18.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:18.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:18.119 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:53:18.120 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:18.120 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:17 smithi097 ceph-mon[31898]: osdmap e60: 9 total, 9 up, 9 in 2024-03-11T19:53:18.751 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi097:vg_nvme/lv_1 2024-03-11T19:53:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:18 smithi114 ceph-mon[36376]: Detected new or changed devices on smithi161 2024-03-11T19:53:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:18 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi161 to 5693M 2024-03-11T19:53:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:18 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:53:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:18 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:18 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:53:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:18 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:19.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:18 smithi161 ceph-mon[36332]: Detected new or changed devices on smithi161 2024-03-11T19:53:19.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:18 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi161 to 5693M 2024-03-11T19:53:19.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:18 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:53:19.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:18 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:19.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:18 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:53:19.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:18 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:19.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:18 smithi097 ceph-mon[31898]: Detected new or changed devices on smithi161 2024-03-11T19:53:19.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:18 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi161 to 5693M 2024-03-11T19:53:19.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:18 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:53:19.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:18 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:19.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:18 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:53:19.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:18 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:19.186 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:53:19.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:19 smithi114 ceph-mon[36376]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:19.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:19 smithi114 ceph-mon[36376]: osdmap e61: 9 total, 9 up, 9 in 2024-03-11T19:53:20.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:19 smithi161 ceph-mon[36332]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:20.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:19 smithi161 ceph-mon[36332]: osdmap e61: 9 total, 9 up, 9 in 2024-03-11T19:53:20.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:19 smithi097 ceph-mon[31898]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:20.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:19 smithi097 ceph-mon[31898]: osdmap e61: 9 total, 9 up, 9 in 2024-03-11T19:53:20.906 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:20 smithi097 ceph-mon[31898]: osdmap e62: 9 total, 9 up, 9 in 2024-03-11T19:53:20.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:20 smithi097 ceph-mon[31898]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:21.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:20 smithi161 ceph-mon[36332]: osdmap e62: 9 total, 9 up, 9 in 2024-03-11T19:53:21.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:20 smithi161 ceph-mon[36332]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:21.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:20 smithi114 ceph-mon[36376]: osdmap e62: 9 total, 9 up, 9 in 2024-03-11T19:53:21.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:20 smithi114 ceph-mon[36376]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:21.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.441+0000 7f3c26e00700 1 -- 172.21.15.97:0/1658898314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c20105060 msgr2=0x7f3c20107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:53:21.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.441+0000 7f3c26e00700 1 --2- 172.21.15.97:0/1658898314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c20105060 0x7f3c20107450 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009990 tx=0x7f3c0c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:53:21.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.441+0000 7f3c26e00700 1 -- 172.21.15.97:0/1658898314 shutdown_connections 2024-03-11T19:53:21.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.441+0000 7f3c26e00700 1 --2- 172.21.15.97:0/1658898314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c20107990 0x7f3c20109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:21.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.441+0000 7f3c26e00700 1 --2- 172.21.15.97:0/1658898314 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c20102730 0x7f3c20104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:21.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.441+0000 7f3c26e00700 1 --2- 172.21.15.97:0/1658898314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c20105060 0x7f3c20107450 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:21.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.441+0000 7f3c26e00700 1 -- 172.21.15.97:0/1658898314 >> 172.21.15.97:0/1658898314 conn(0x7f3c200fc140 msgr2=0x7f3c200fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:53:21.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.441+0000 7f3c26e00700 1 -- 172.21.15.97:0/1658898314 shutdown_connections 2024-03-11T19:53:21.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.441+0000 7f3c26e00700 1 -- 172.21.15.97:0/1658898314 wait complete. 2024-03-11T19:53:21.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.442+0000 7f3c26e00700 1 Processor -- start 2024-03-11T19:53:21.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.443+0000 7f3c26e00700 1 -- start start 2024-03-11T19:53:21.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.443+0000 7f3c26e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c20102730 0x7f3c20199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:53:21.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.443+0000 7f3c26e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c20105060 0x7f3c2019a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:53:21.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.443+0000 7f3c26e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c20107990 0x7f3c201a14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:53:21.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.443+0000 7f3c26e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c201a1ac0 con 0x7f3c20107990 2024-03-11T19:53:21.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.443+0000 7f3c26e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c201a1c00 con 0x7f3c20105060 2024-03-11T19:53:21.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.443+0000 7f3c26e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c201a1d90 con 0x7f3c20102730 2024-03-11T19:53:21.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.443+0000 7f3c1fe00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c20102730 0x7f3c20199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:53:21.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.443+0000 7f3c1fe00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c20102730 0x7f3c20199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:57368/0 (socket says 172.21.15.97:57368) 2024-03-11T19:53:21.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.443+0000 7f3c1fe00700 1 -- 172.21.15.97:0/3232253812 learned_addr learned my addr 172.21.15.97:0/3232253812 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:53:21.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.443+0000 7f3c24e00700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c20107990 0x7f3c201a14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:53:21.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.443+0000 7f3c1f400700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c20105060 0x7f3c2019a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:53:21.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.444+0000 7f3c1fe00700 1 -- 172.21.15.97:0/3232253812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c20105060 msgr2=0x7f3c2019a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:53:21.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.444+0000 7f3c1fe00700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c20105060 0x7f3c2019a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:21.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.444+0000 7f3c1fe00700 1 -- 172.21.15.97:0/3232253812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c20107990 msgr2=0x7f3c201a14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:53:21.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.444+0000 7f3c1fe00700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c20107990 0x7f3c201a14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:21.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.444+0000 7f3c1fe00700 1 -- 172.21.15.97:0/3232253812 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c0095f0 con 0x7f3c20102730 2024-03-11T19:53:21.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.444+0000 7f3c1fe00700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c20102730 0x7f3c20199f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000e9d0 tx=0x7f3c1000ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:53:21.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.444+0000 7f3c1cc00700 1 -- 172.21.15.97:0/3232253812 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c10009790 con 0x7f3c20102730 2024-03-11T19:53:21.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.444+0000 7f3c1cc00700 1 -- 172.21.15.97:0/3232253812 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c1000ee70 con 0x7f3c20102730 2024-03-11T19:53:21.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.444+0000 7f3c26e00700 1 -- 172.21.15.97:0/3232253812 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c201a2070 con 0x7f3c20102730 2024-03-11T19:53:21.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.444+0000 7f3c1cc00700 1 -- 172.21.15.97:0/3232253812 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c10018680 con 0x7f3c20102730 2024-03-11T19:53:21.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.444+0000 7f3c26e00700 1 -- 172.21.15.97:0/3232253812 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c201a2570 con 0x7f3c20102730 2024-03-11T19:53:21.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.445+0000 7f3c26e00700 1 -- 172.21.15.97:0/3232253812 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c20109560 con 0x7f3c20102730 2024-03-11T19:53:21.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.447+0000 7f3c1cc00700 1 -- 172.21.15.97:0/3232253812 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3c1000cbd0 con 0x7f3c20102730 2024-03-11T19:53:21.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.449+0000 7f3c1cc00700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3c040a1ac0 0x7f3c040a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:53:21.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.449+0000 7f3c1cc00700 1 -- 172.21.15.97:0/3232253812 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5829+0+0 (secure 0 0 0) 0x7f3c10014070 con 0x7f3c20102730 2024-03-11T19:53:21.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.449+0000 7f3c1f400700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3c040a1ac0 0x7f3c040a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:53:21.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.450+0000 7f3c1f400700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3c040a1ac0 0x7f3c040a3f80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c00b6f0 tx=0x7f3c0c00b7c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:53:21.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.451+0000 7f3c1cc00700 1 -- 172.21.15.97:0/3232253812 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3c100967a0 con 0x7f3c20102730 2024-03-11T19:53:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:21.680+0000 7f3c26e00700 1 -- 172.21.15.97:0/3232253812 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c2002d8a0 con 0x7f3c040a1ac0 2024-03-11T19:53:22.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:21 smithi161 ceph-mon[36332]: osdmap e63: 9 total, 9 up, 9 in 2024-03-11T19:53:22.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:21 smithi161 ceph-mon[36332]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:53:22.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:21 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:53:22.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:21 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:53:22.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:21 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:22.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:21 smithi097 ceph-mon[31898]: osdmap e63: 9 total, 9 up, 9 in 2024-03-11T19:53:22.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:21 smithi097 ceph-mon[31898]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:53:22.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:21 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:53:22.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:21 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:53:22.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:21 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:21 smithi114 ceph-mon[36376]: osdmap e63: 9 total, 9 up, 9 in 2024-03-11T19:53:22.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:21 smithi114 ceph-mon[36376]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:53:22.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:21 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:53:22.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:21 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:53:22.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:21 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:22.896 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:22 smithi097 ceph-mon[31898]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:23.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:22 smithi161 ceph-mon[36332]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:22 smithi114 ceph-mon[36376]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:25.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:25 smithi161 ceph-mon[36332]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:53:25.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:25 smithi097 ceph-mon[31898]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:53:25.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:25 smithi114 ceph-mon[36376]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T19:53:26.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:26 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/380110874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d55f416c-9d0f-4938-8e34-b364abeaedcc"}]: dispatch 2024-03-11T19:53:26.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:26 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d55f416c-9d0f-4938-8e34-b364abeaedcc"}]: dispatch 2024-03-11T19:53:26.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:26 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d55f416c-9d0f-4938-8e34-b364abeaedcc"}]': finished 2024-03-11T19:53:26.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:26 smithi097 ceph-mon[31898]: osdmap e64: 10 total, 9 up, 10 in 2024-03-11T19:53:26.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:26 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:26.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:26 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/380110874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d55f416c-9d0f-4938-8e34-b364abeaedcc"}]: dispatch 2024-03-11T19:53:26.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:26 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d55f416c-9d0f-4938-8e34-b364abeaedcc"}]: dispatch 2024-03-11T19:53:26.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:26 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d55f416c-9d0f-4938-8e34-b364abeaedcc"}]': finished 2024-03-11T19:53:26.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:26 smithi161 ceph-mon[36332]: osdmap e64: 10 total, 9 up, 10 in 2024-03-11T19:53:26.791 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:26 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:26.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:26 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/380110874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d55f416c-9d0f-4938-8e34-b364abeaedcc"}]: dispatch 2024-03-11T19:53:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:26 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d55f416c-9d0f-4938-8e34-b364abeaedcc"}]: dispatch 2024-03-11T19:53:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:26 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d55f416c-9d0f-4938-8e34-b364abeaedcc"}]': finished 2024-03-11T19:53:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:26 smithi114 ceph-mon[36376]: osdmap e64: 10 total, 9 up, 10 in 2024-03-11T19:53:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:26 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:27.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:27 smithi161 ceph-mon[36332]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-11T19:53:27.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:27 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3885652160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:53:27.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:27 smithi097 ceph-mon[31898]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-11T19:53:27.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:27 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3885652160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:53:27.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:27 smithi114 ceph-mon[36376]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-11T19:53:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:27 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3885652160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:53:29.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:29 smithi161 ceph-mon[36332]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T19:53:29.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:29 smithi097 ceph-mon[31898]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T19:53:29.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:29 smithi114 ceph-mon[36376]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T19:53:31.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:31 smithi161 ceph-mon[36332]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-11T19:53:31.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:31 smithi097 ceph-mon[31898]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-11T19:53:31.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:31 smithi114 ceph-mon[36376]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-11T19:53:33.644 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:33 smithi097 ceph-mon[31898]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:53:33.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:33 smithi161 ceph-mon[36332]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:53:33.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:33 smithi114 ceph-mon[36376]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:53:35.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:35 smithi161 ceph-mon[36332]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:35.796 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:35 smithi114 ceph-mon[36376]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:35.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:35 smithi097 ceph-mon[31898]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:36.856 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:36 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-11T19:53:36.857 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:36 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:36.857 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:36 smithi097 ceph-mon[31898]: Deploying daemon osd.9 on smithi097 2024-03-11T19:53:36.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:36 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-11T19:53:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:36 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:36 smithi114 ceph-mon[36376]: Deploying daemon osd.9 on smithi097 2024-03-11T19:53:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:36 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-11T19:53:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:36 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:36 smithi161 ceph-mon[36332]: Deploying daemon osd.9 on smithi097 2024-03-11T19:53:37.767 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:37 smithi097 ceph-mon[31898]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:37.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:37 smithi114 ceph-mon[36376]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:38.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:37 smithi161 ceph-mon[36332]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:39.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:39 smithi097 ceph-mon[31898]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:39.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:39 smithi114 ceph-mon[36376]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:40.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:39 smithi161 ceph-mon[36332]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:41.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:41 smithi097 ceph-mon[31898]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:41.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:41 smithi114 ceph-mon[36376]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:42.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:41 smithi161 ceph-mon[36332]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:43.866 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:43 smithi097 ceph-mon[31898]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:43.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:43 smithi114 ceph-mon[36376]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:44.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:43 smithi161 ceph-mon[36332]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:45.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:45.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:53:45.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:45.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:45 smithi097 ceph-mon[31898]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:45.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:53:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:45 smithi114 ceph-mon[36376]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:53:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:45.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:45 smithi161 ceph-mon[36332]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:47.775 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:47 smithi097 ceph-mon[31898]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:47.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:47 smithi161 ceph-mon[36332]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:47 smithi114 ceph-mon[36376]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:48.674 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:48 smithi097 ceph-mon[31898]: from='osd.9 [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T19:53:48.674 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:48 smithi097 ceph-mon[31898]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T19:53:48.674 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:48.674 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:48.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:48 smithi161 ceph-mon[36332]: from='osd.9 [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T19:53:48.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:48 smithi161 ceph-mon[36332]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T19:53:48.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:48 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:48.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:48 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:48.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:48 smithi114 ceph-mon[36376]: from='osd.9 [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T19:53:48.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:48 smithi114 ceph-mon[36376]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T19:53:48.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:48.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:49.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:49.385+0000 7f3c1cc00700 1 -- 172.21.15.97:0/3232253812 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3c2002d8a0 con 0x7f3c040a1ac0 2024-03-11T19:53:49.391 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 9 on host 'smithi097' 2024-03-11T19:53:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:49.389+0000 7f3c26e00700 1 -- 172.21.15.97:0/3232253812 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3c040a1ac0 msgr2=0x7f3c040a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:53:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:49.389+0000 7f3c26e00700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3c040a1ac0 0x7f3c040a3f80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c00b6f0 tx=0x7f3c0c00b7c0 comp rx=0 tx=0).stop 2024-03-11T19:53:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:49.389+0000 7f3c26e00700 1 -- 172.21.15.97:0/3232253812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c20102730 msgr2=0x7f3c20199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:53:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:49.389+0000 7f3c26e00700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c20102730 0x7f3c20199f30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000e9d0 tx=0x7f3c1000ece0 comp rx=0 tx=0).stop 2024-03-11T19:53:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:49.390+0000 7f3c26e00700 1 -- 172.21.15.97:0/3232253812 shutdown_connections 2024-03-11T19:53:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:49.390+0000 7f3c26e00700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3c040a1ac0 0x7f3c040a3f80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:49.390+0000 7f3c26e00700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c20107990 0x7f3c201a14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:49.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:49.390+0000 7f3c26e00700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c20102730 0x7f3c20199f30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:49.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:49.390+0000 7f3c26e00700 1 --2- 172.21.15.97:0/3232253812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c20105060 0x7f3c2019a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:49.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:49.390+0000 7f3c26e00700 1 -- 172.21.15.97:0/3232253812 >> 172.21.15.97:0/3232253812 conn(0x7f3c200fc140 msgr2=0x7f3c20101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:53:49.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:49.390+0000 7f3c26e00700 1 -- 172.21.15.97:0/3232253812 shutdown_connections 2024-03-11T19:53:49.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:53:49.390+0000 7f3c26e00700 1 -- 172.21.15.97:0/3232253812 wait complete. 2024-03-11T19:53:49.809 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:49 smithi097 ceph-mon[31898]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:49.809 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:49 smithi097 ceph-mon[31898]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-11T19:53:49.809 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:49 smithi097 ceph-mon[31898]: osdmap e65: 10 total, 9 up, 10 in 2024-03-11T19:53:49.809 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:49 smithi097 ceph-mon[31898]: from='osd.9 [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:53:49.809 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:49 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:49.809 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:49 smithi097 ceph-mon[31898]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:53:49.809 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:49 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:49.809 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:49 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:49 smithi114 ceph-mon[36376]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:49 smithi114 ceph-mon[36376]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-11T19:53:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:49 smithi114 ceph-mon[36376]: osdmap e65: 10 total, 9 up, 10 in 2024-03-11T19:53:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:49 smithi114 ceph-mon[36376]: from='osd.9 [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:53:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:49 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:49 smithi114 ceph-mon[36376]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:53:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:49 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:49 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:50.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:49 smithi161 ceph-mon[36332]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:50.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:49 smithi161 ceph-mon[36332]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-11T19:53:50.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:49 smithi161 ceph-mon[36332]: osdmap e65: 10 total, 9 up, 10 in 2024-03-11T19:53:50.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:49 smithi161 ceph-mon[36332]: from='osd.9 [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:53:50.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:49 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:50.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:49 smithi161 ceph-mon[36332]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T19:53:50.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:49 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:50.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:49 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:50.202 DEBUG:teuthology.orchestra.run.smithi097:osd.9> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.9.service 2024-03-11T19:53:50.206 INFO:tasks.cephadm:Deploying osd.10 on smithi114 with /dev/vg_nvme/lv_1... 2024-03-11T19:53:50.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T19:53:50.452 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:53:50.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:50 smithi097 ceph-mon[31898]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T19:53:50.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:50 smithi097 ceph-mon[31898]: osdmap e66: 10 total, 9 up, 10 in 2024-03-11T19:53:50.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:50 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:50.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:50 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:50.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:50 smithi097 ceph-mon[31898]: from='osd.9 ' entity='osd.9' 2024-03-11T19:53:50.673 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 19:53:50 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T19:53:50.403+0000 7f05e6200700 -1 osd.9 0 waiting for initial osdmap 2024-03-11T19:53:50.673 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 19:53:50 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T19:53:50.422+0000 7f05dc000700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:53:50.862 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:50 smithi114 ceph-mon[36376]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T19:53:50.862 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:50 smithi114 ceph-mon[36376]: osdmap e66: 10 total, 9 up, 10 in 2024-03-11T19:53:50.862 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:50 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:50.862 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:50 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:50.862 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:50 smithi114 ceph-mon[36376]: from='osd.9 ' entity='osd.9' 2024-03-11T19:53:51.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:50 smithi161 ceph-mon[36332]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T19:53:51.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:50 smithi161 ceph-mon[36332]: osdmap e66: 10 total, 9 up, 10 in 2024-03-11T19:53:51.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:50 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:51.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:50 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:51.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:50 smithi161 ceph-mon[36332]: from='osd.9 ' entity='osd.9' 2024-03-11T19:53:51.863 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:51 smithi097 ceph-mon[31898]: purged_snaps scrub starts 2024-03-11T19:53:51.863 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:51 smithi097 ceph-mon[31898]: purged_snaps scrub ok 2024-03-11T19:53:51.863 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:51 smithi097 ceph-mon[31898]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:51.863 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:51 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:51.863 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:51 smithi097 ceph-mon[31898]: osd.9 [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428] boot 2024-03-11T19:53:51.863 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:51 smithi097 ceph-mon[31898]: osdmap e67: 10 total, 10 up, 10 in 2024-03-11T19:53:51.863 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:51 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:51.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:51 smithi114 ceph-mon[36376]: purged_snaps scrub starts 2024-03-11T19:53:51.897 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:51 smithi114 ceph-mon[36376]: purged_snaps scrub ok 2024-03-11T19:53:51.897 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:51 smithi114 ceph-mon[36376]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:51.897 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:51 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:51.897 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:51 smithi114 ceph-mon[36376]: osd.9 [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428] boot 2024-03-11T19:53:51.897 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:51 smithi114 ceph-mon[36376]: osdmap e67: 10 total, 10 up, 10 in 2024-03-11T19:53:51.897 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:51 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:52.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:51 smithi161 ceph-mon[36332]: purged_snaps scrub starts 2024-03-11T19:53:52.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:51 smithi161 ceph-mon[36332]: purged_snaps scrub ok 2024-03-11T19:53:52.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:51 smithi161 ceph-mon[36332]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-03-11T19:53:52.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:51 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:52.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:51 smithi161 ceph-mon[36332]: osd.9 [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428] boot 2024-03-11T19:53:52.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:51 smithi161 ceph-mon[36332]: osdmap e67: 10 total, 10 up, 10 in 2024-03-11T19:53:52.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:51 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:53:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T19:53:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T19:53:52.481 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-03-11T19:53:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-03-11T19:53:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122288 s, 857 MB/s 2024-03-11T19:53:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-03-11T19:53:53.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi114:vg_nvme/lv_1 2024-03-11T19:53:53.364 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T19:53:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:53 smithi114 ceph-mon[36376]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:53:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:53 smithi114 ceph-mon[36376]: osdmap e68: 10 total, 10 up, 10 in 2024-03-11T19:53:53.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:53 smithi097 ceph-mon[31898]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:53:53.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:53 smithi097 ceph-mon[31898]: osdmap e68: 10 total, 10 up, 10 in 2024-03-11T19:53:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:53 smithi161 ceph-mon[36332]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:53:54.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:53 smithi161 ceph-mon[36332]: osdmap e68: 10 total, 10 up, 10 in 2024-03-11T19:53:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.897+0000 7f227ffa3700 1 -- 172.21.15.114:0/1549663175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2278101ca0 msgr2=0x7f22781020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:53:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.897+0000 7f227ffa3700 1 --2- 172.21.15.114:0/1549663175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2278101ca0 0x7f22781020c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f22680099c0 tx=0x7f2268009cd0 comp rx=0 tx=0).stop 2024-03-11T19:53:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.897+0000 7f227ffa3700 1 -- 172.21.15.114:0/1549663175 shutdown_connections 2024-03-11T19:53:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.897+0000 7f227ffa3700 1 --2- 172.21.15.114:0/1549663175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2278103840 0x7f227810a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:54.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.897+0000 7f227ffa3700 1 --2- 172.21.15.114:0/1549663175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2278101ca0 0x7f22781020c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.897+0000 7f227ffa3700 1 --2- 172.21.15.114:0/1549663175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2278102ea0 0x7f2278103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.897+0000 7f227ffa3700 1 -- 172.21.15.114:0/1549663175 >> 172.21.15.114:0/1549663175 conn(0x7f22780fd280 msgr2=0x7f22780ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.898+0000 7f227ffa3700 1 -- 172.21.15.114:0/1549663175 shutdown_connections 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.898+0000 7f227ffa3700 1 -- 172.21.15.114:0/1549663175 wait complete. 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.898+0000 7f227ffa3700 1 Processor -- start 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.898+0000 7f227ffa3700 1 -- start start 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.898+0000 7f227ffa3700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2278101ca0 0x7f2278197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.898+0000 7f227ffa3700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2278102ea0 0x7f2278198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.898+0000 7f227ffa3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2278103840 0x7f227819f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.898+0000 7f227ffa3700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227819f880 con 0x7f2278102ea0 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.898+0000 7f227ffa3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227819f9c0 con 0x7f2278103840 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.898+0000 7f227ffa3700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227819fb80 con 0x7f2278101ca0 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.898+0000 7f227e540700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2278103840 0x7f227819f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:53:54.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.898+0000 7f227e540700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2278103840 0x7f227819f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47488/0 (socket says 172.21.15.114:47488) 2024-03-11T19:53:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.898+0000 7f227e540700 1 -- 172.21.15.114:0/2948145438 learned_addr learned my addr 172.21.15.114:0/2948145438 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-03-11T19:53:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.899+0000 7f227dd3f700 1 --2- 172.21.15.114:0/2948145438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2278101ca0 0x7f2278197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:53:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.899+0000 7f227e540700 1 -- 172.21.15.114:0/2948145438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2278101ca0 msgr2=0x7f2278197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:53:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.899+0000 7f227e540700 1 --2- 172.21.15.114:0/2948145438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2278101ca0 0x7f2278197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.899+0000 7f227e540700 1 -- 172.21.15.114:0/2948145438 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2278102ea0 msgr2=0x7f2278198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:53:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.899+0000 7f227e540700 1 --2- 172.21.15.114:0/2948145438 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2278102ea0 0x7f2278198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:53:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.899+0000 7f227e540700 1 -- 172.21.15.114:0/2948145438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22680095f0 con 0x7f2278103840 2024-03-11T19:53:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.899+0000 7f227e540700 1 --2- 172.21.15.114:0/2948145438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2278103840 0x7f227819f2b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f227000b970 tx=0x7f227000bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:53:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.899+0000 7f226effd700 1 -- 172.21.15.114:0/2948145438 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227000c930 con 0x7f2278103840 2024-03-11T19:53:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.899+0000 7f226effd700 1 -- 172.21.15.114:0/2948145438 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2270014440 con 0x7f2278103840 2024-03-11T19:53:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.899+0000 7f227ffa3700 1 -- 172.21.15.114:0/2948145438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227819fe60 con 0x7f2278103840 2024-03-11T19:53:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.899+0000 7f226effd700 1 -- 172.21.15.114:0/2948145438 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22700075d0 con 0x7f2278103840 2024-03-11T19:53:54.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.899+0000 7f227ffa3700 1 -- 172.21.15.114:0/2948145438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22781a02a0 con 0x7f2278103840 2024-03-11T19:53:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.900+0000 7f227ffa3700 1 -- 172.21.15.114:0/2948145438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2278109080 con 0x7f2278103840 2024-03-11T19:53:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.901+0000 7f226effd700 1 -- 172.21.15.114:0/2948145438 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f22700145b0 con 0x7f2278103840 2024-03-11T19:53:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.902+0000 7f226effd700 1 --2- 172.21.15.114:0/2948145438 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22640a1ac0 0x7f22640a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:53:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.902+0000 7f226effd700 1 -- 172.21.15.114:0/2948145438 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f2270006e40 con 0x7f2278103840 2024-03-11T19:53:54.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.902+0000 7f227dd3f700 1 --2- 172.21.15.114:0/2948145438 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22640a1ac0 0x7f22640a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:53:54.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.903+0000 7f227dd3f700 1 --2- 172.21.15.114:0/2948145438 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22640a1ac0 0x7f22640a3f80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f22680038c0 tx=0x7f226800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:53:54.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:54.907+0000 7f226effd700 1 -- 172.21.15.114:0/2948145438 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2270095cc0 con 0x7f2278103840 2024-03-11T19:53:55.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:53:55.110+0000 7f227ffa3700 1 -- 172.21.15.114:0/2948145438 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f227802da20 con 0x7f22640a1ac0 2024-03-11T19:53:55.857 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:53:55.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:53:55.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:53:55.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:55.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:55.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:55.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:55.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:55.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:55.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:55.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:55.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:55.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:53:55.859 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:55.859 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:53:55.859 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:55.859 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:55 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:53:55.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:53:55.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:53:55.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:55.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:55 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:53:56.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:53:56.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:53:56.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:53:56.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:56.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:56.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:56.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:56.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:56.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:56.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:53:56.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:56.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:56.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:53:56.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:56.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:53:56.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:53:56.041 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:55 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:53:56.883 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:56 smithi114 ceph-mon[36376]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:53:56.883 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:56 smithi114 ceph-mon[36376]: Detected new or changed devices on smithi097 2024-03-11T19:53:56.883 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:56 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi097 to 4272M 2024-03-11T19:53:56.883 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:56 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:56.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:56 smithi097 ceph-mon[31898]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:53:56.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:56 smithi097 ceph-mon[31898]: Detected new or changed devices on smithi097 2024-03-11T19:53:56.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:56 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi097 to 4272M 2024-03-11T19:53:56.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:56 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:56 smithi161 ceph-mon[36332]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:53:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:56 smithi161 ceph-mon[36332]: Detected new or changed devices on smithi097 2024-03-11T19:53:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:56 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi097 to 4272M 2024-03-11T19:53:57.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:56 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:53:57.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:57 smithi097 ceph-mon[31898]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:53:57.941 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:57 smithi161 ceph-mon[36332]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:53:57.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:57 smithi114 ceph-mon[36376]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:53:58.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:58 smithi097 ceph-mon[31898]: from='client.? 172.21.15.114:0/1217176185' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27d60b08-8d2b-453e-841d-abb1339ce6e3"}]: dispatch 2024-03-11T19:53:58.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:58 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27d60b08-8d2b-453e-841d-abb1339ce6e3"}]: dispatch 2024-03-11T19:53:58.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:58 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27d60b08-8d2b-453e-841d-abb1339ce6e3"}]': finished 2024-03-11T19:53:58.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:58 smithi097 ceph-mon[31898]: osdmap e69: 11 total, 10 up, 11 in 2024-03-11T19:53:58.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:58 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:53:58.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:58 smithi097 ceph-mon[31898]: from='client.? 172.21.15.114:0/1479351758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:53:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:58 smithi114 ceph-mon[36376]: from='client.? 172.21.15.114:0/1217176185' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27d60b08-8d2b-453e-841d-abb1339ce6e3"}]: dispatch 2024-03-11T19:53:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:58 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27d60b08-8d2b-453e-841d-abb1339ce6e3"}]: dispatch 2024-03-11T19:53:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:58 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27d60b08-8d2b-453e-841d-abb1339ce6e3"}]': finished 2024-03-11T19:53:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:58 smithi114 ceph-mon[36376]: osdmap e69: 11 total, 10 up, 11 in 2024-03-11T19:53:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:58 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:53:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:58 smithi114 ceph-mon[36376]: from='client.? 172.21.15.114:0/1479351758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:53:59.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:58 smithi161 ceph-mon[36332]: from='client.? 172.21.15.114:0/1217176185' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27d60b08-8d2b-453e-841d-abb1339ce6e3"}]: dispatch 2024-03-11T19:53:59.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:58 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27d60b08-8d2b-453e-841d-abb1339ce6e3"}]: dispatch 2024-03-11T19:53:59.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:58 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27d60b08-8d2b-453e-841d-abb1339ce6e3"}]': finished 2024-03-11T19:53:59.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:58 smithi161 ceph-mon[36332]: osdmap e69: 11 total, 10 up, 11 in 2024-03-11T19:53:59.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:58 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:53:59.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:58 smithi161 ceph-mon[36332]: from='client.? 172.21.15.114:0/1479351758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:53:59.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:53:59 smithi097 ceph-mon[31898]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:53:59.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:53:59 smithi114 ceph-mon[36376]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:00.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:53:59 smithi161 ceph-mon[36332]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:01 smithi114 ceph-mon[36376]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:01.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:01 smithi097 ceph-mon[31898]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:02.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:01 smithi161 ceph-mon[36332]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:03.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:03 smithi097 ceph-mon[31898]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:03.926 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:03 smithi114 ceph-mon[36376]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:04.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:03 smithi161 ceph-mon[36332]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:05 smithi097 ceph-mon[31898]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-11T19:54:05.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:05.935 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:05 smithi114 ceph-mon[36376]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:05.935 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-11T19:54:05.935 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:06.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:05 smithi161 ceph-mon[36332]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:06.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-11T19:54:06.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:06.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:06 smithi097 ceph-mon[31898]: Deploying daemon osd.10 on smithi114 2024-03-11T19:54:06.939 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:06 smithi114 ceph-mon[36376]: Deploying daemon osd.10 on smithi114 2024-03-11T19:54:07.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:06 smithi161 ceph-mon[36332]: Deploying daemon osd.10 on smithi114 2024-03-11T19:54:07.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:07 smithi097 ceph-mon[31898]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:07.926 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:07 smithi114 ceph-mon[36376]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:08.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:07 smithi161 ceph-mon[36332]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:09 smithi114 ceph-mon[36376]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:09.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:09 smithi097 ceph-mon[31898]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:10.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:09 smithi161 ceph-mon[36332]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:11.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:11 smithi097 ceph-mon[31898]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:11.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:11 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:11.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:11 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:11.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:11 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:11.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:11 smithi114 ceph-mon[36376]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:11.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:11 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:11.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:11 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:11.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:11 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:12.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:11 smithi161 ceph-mon[36332]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:12.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:11 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:12.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:11 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:12.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:11 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:13.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:13 smithi097 ceph-mon[31898]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:13.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:13 smithi114 ceph-mon[36376]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:14.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:13 smithi161 ceph-mon[36332]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:14.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:54:14.094+0000 7f226effd700 1 -- 172.21.15.114:0/2948145438 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f227802da20 con 0x7f22640a1ac0 2024-03-11T19:54:14.101 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 10 on host 'smithi114' 2024-03-11T19:54:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:54:14.096+0000 7f227ffa3700 1 -- 172.21.15.114:0/2948145438 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22640a1ac0 msgr2=0x7f22640a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:54:14.096+0000 7f227ffa3700 1 --2- 172.21.15.114:0/2948145438 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22640a1ac0 0x7f22640a3f80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f22680038c0 tx=0x7f226800b540 comp rx=0 tx=0).stop 2024-03-11T19:54:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:54:14.096+0000 7f227ffa3700 1 -- 172.21.15.114:0/2948145438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2278103840 msgr2=0x7f227819f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:54:14.096+0000 7f227ffa3700 1 --2- 172.21.15.114:0/2948145438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2278103840 0x7f227819f2b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f227000b970 tx=0x7f227000bc80 comp rx=0 tx=0).stop 2024-03-11T19:54:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:54:14.096+0000 7f227ffa3700 1 -- 172.21.15.114:0/2948145438 shutdown_connections 2024-03-11T19:54:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:54:14.096+0000 7f227ffa3700 1 --2- 172.21.15.114:0/2948145438 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22640a1ac0 0x7f22640a3f80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:54:14.096+0000 7f227ffa3700 1 --2- 172.21.15.114:0/2948145438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2278103840 0x7f227819f2b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:54:14.096+0000 7f227ffa3700 1 --2- 172.21.15.114:0/2948145438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2278101ca0 0x7f2278197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:54:14.096+0000 7f227ffa3700 1 --2- 172.21.15.114:0/2948145438 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2278102ea0 0x7f2278198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:54:14.096+0000 7f227ffa3700 1 -- 172.21.15.114:0/2948145438 >> 172.21.15.114:0/2948145438 conn(0x7f22780fd280 msgr2=0x7f22781061c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:54:14.097+0000 7f227ffa3700 1 -- 172.21.15.114:0/2948145438 shutdown_connections 2024-03-11T19:54:14.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-11T19:54:14.097+0000 7f227ffa3700 1 -- 172.21.15.114:0/2948145438 wait complete. 2024-03-11T19:54:14.700 DEBUG:teuthology.orchestra.run.smithi114:osd.10> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.10.service 2024-03-11T19:54:14.703 INFO:tasks.cephadm:Deploying osd.11 on smithi161 with /dev/vg_nvme/lv_1... 2024-03-11T19:54:14.703 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T19:54:14.955 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:54:15.303 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:15 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:15.303 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:15 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:15.303 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:15 smithi161 ceph-mon[36332]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:15.303 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:15 smithi161 ceph-mon[36332]: from='osd.10 [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T19:54:15.304 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:15 smithi161 ceph-mon[36332]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T19:54:15.304 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:15 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:15.304 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:15 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:15.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:15 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:15.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:15 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:15.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:15 smithi097 ceph-mon[31898]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:15.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:15 smithi097 ceph-mon[31898]: from='osd.10 [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T19:54:15.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:15 smithi097 ceph-mon[31898]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T19:54:15.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:15 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:15.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:15 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:15.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:15 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:15.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:15 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:15.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:15 smithi114 ceph-mon[36376]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:15.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:15 smithi114 ceph-mon[36376]: from='osd.10 [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T19:54:15.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:15 smithi114 ceph-mon[36376]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T19:54:15.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:15 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:15.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:15 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:16.347 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:16 smithi161 ceph-mon[36332]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-11T19:54:16.347 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:16 smithi161 ceph-mon[36332]: from='osd.10 [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:54:16.347 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:16 smithi161 ceph-mon[36332]: osdmap e70: 11 total, 10 up, 11 in 2024-03-11T19:54:16.347 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:16 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:54:16.348 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:16 smithi161 ceph-mon[36332]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:54:16.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:16 smithi097 ceph-mon[31898]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-11T19:54:16.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:16 smithi097 ceph-mon[31898]: from='osd.10 [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:54:16.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:16 smithi097 ceph-mon[31898]: osdmap e70: 11 total, 10 up, 11 in 2024-03-11T19:54:16.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:16 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:54:16.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:16 smithi097 ceph-mon[31898]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:54:16.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:16 smithi114 ceph-mon[36376]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-11T19:54:16.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:16 smithi114 ceph-mon[36376]: from='osd.10 [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:54:16.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:16 smithi114 ceph-mon[36376]: osdmap e70: 11 total, 10 up, 11 in 2024-03-11T19:54:16.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:16 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:54:16.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:16 smithi114 ceph-mon[36376]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-03-11T19:54:16.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 19:54:16 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T19:54:16.728+0000 7f6aaa257700 -1 osd.10 0 waiting for initial osdmap 2024-03-11T19:54:16.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 19:54:16 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T19:54:16.741+0000 7f6aa2042700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:54:17.054 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T19:54:17.054 INFO:teuthology.orchestra.run.smithi161.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T19:54:17.054 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10+0 records in 2024-03-11T19:54:17.054 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2024-03-11T19:54:17.054 INFO:teuthology.orchestra.run.smithi161.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114024 s, 920 MB/s 2024-03-11T19:54:17.054 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 2024-03-11T19:54:17.054 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping successful for: 2024-03-11T19:54:17.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:17 smithi114 ceph-mon[36376]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-11T19:54:17.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:17 smithi114 ceph-mon[36376]: osdmap e71: 11 total, 10 up, 11 in 2024-03-11T19:54:17.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:54:17.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:17 smithi114 ceph-mon[36376]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:17.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:17 smithi114 ceph-mon[36376]: from='osd.10 ' entity='osd.10' 2024-03-11T19:54:17.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:17 smithi114 ceph-mon[36376]: osd.10 [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537] boot 2024-03-11T19:54:17.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:17 smithi114 ceph-mon[36376]: osdmap e72: 11 total, 11 up, 11 in 2024-03-11T19:54:17.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:54:17.356 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:17 smithi161 ceph-mon[36332]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-11T19:54:17.357 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:17 smithi161 ceph-mon[36332]: osdmap e71: 11 total, 10 up, 11 in 2024-03-11T19:54:17.357 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:54:17.357 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:17 smithi161 ceph-mon[36332]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:17.357 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:17 smithi161 ceph-mon[36332]: from='osd.10 ' entity='osd.10' 2024-03-11T19:54:17.357 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:17 smithi161 ceph-mon[36332]: osd.10 [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537] boot 2024-03-11T19:54:17.357 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:17 smithi161 ceph-mon[36332]: osdmap e72: 11 total, 11 up, 11 in 2024-03-11T19:54:17.357 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:54:17.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:17 smithi097 ceph-mon[31898]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-03-11T19:54:17.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:17 smithi097 ceph-mon[31898]: osdmap e71: 11 total, 10 up, 11 in 2024-03-11T19:54:17.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:54:17.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:17 smithi097 ceph-mon[31898]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T19:54:17.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:17 smithi097 ceph-mon[31898]: from='osd.10 ' entity='osd.10' 2024-03-11T19:54:17.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:17 smithi097 ceph-mon[31898]: osd.10 [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537] boot 2024-03-11T19:54:17.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:17 smithi097 ceph-mon[31898]: osdmap e72: 11 total, 11 up, 11 in 2024-03-11T19:54:17.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:54:17.732 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi161:vg_nvme/lv_1 2024-03-11T19:54:17.976 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:54:18.367 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:18 smithi161 ceph-mon[36332]: purged_snaps scrub starts 2024-03-11T19:54:18.367 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:18 smithi161 ceph-mon[36332]: purged_snaps scrub ok 2024-03-11T19:54:18.367 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:18 smithi161 ceph-mon[36332]: osdmap e73: 11 total, 11 up, 11 in 2024-03-11T19:54:18.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:18 smithi097 ceph-mon[31898]: purged_snaps scrub starts 2024-03-11T19:54:18.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:18 smithi097 ceph-mon[31898]: purged_snaps scrub ok 2024-03-11T19:54:18.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:18 smithi097 ceph-mon[31898]: osdmap e73: 11 total, 11 up, 11 in 2024-03-11T19:54:18.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:18 smithi114 ceph-mon[36376]: purged_snaps scrub starts 2024-03-11T19:54:18.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:18 smithi114 ceph-mon[36376]: purged_snaps scrub ok 2024-03-11T19:54:18.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:18 smithi114 ceph-mon[36376]: osdmap e73: 11 total, 11 up, 11 in 2024-03-11T19:54:19.426 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: Detected new or changed devices on smithi114 2024-03-11T19:54:19.426 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.426 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.426 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:19.426 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:19.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:19.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:19.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi114 to 4270M 2024-03-11T19:54:19.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:19.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:54:19.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 723 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:19.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:19.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:19.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:54:19.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.427 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:19 smithi161 ceph-mon[36332]: osdmap e74: 11 total, 11 up, 11 in 2024-03-11T19:54:19.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.648+0000 7fe9a90cb700 1 -- 172.21.15.161:0/4246873629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4101da0 msgr2=0x7fe9a4102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.648+0000 7fe9a90cb700 1 --2- 172.21.15.161:0/4246873629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4101da0 0x7fe9a4102200 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009990 tx=0x7fe98c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:54:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.648+0000 7fe9a90cb700 1 -- 172.21.15.161:0/4246873629 shutdown_connections 2024-03-11T19:54:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.648+0000 7fe9a90cb700 1 --2- 172.21.15.161:0/4246873629 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9a4102740 0x7fe9a4108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.648+0000 7fe9a90cb700 1 --2- 172.21.15.161:0/4246873629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9a4100ba0 0x7fe9a4100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.648+0000 7fe9a90cb700 1 --2- 172.21.15.161:0/4246873629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4101da0 0x7fe9a4102200 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.648+0000 7fe9a90cb700 1 -- 172.21.15.161:0/4246873629 >> 172.21.15.161:0/4246873629 conn(0x7fe9a40fc160 msgr2=0x7fe9a40fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.648+0000 7fe9a90cb700 1 -- 172.21.15.161:0/4246873629 shutdown_connections 2024-03-11T19:54:19.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.648+0000 7fe9a90cb700 1 -- 172.21.15.161:0/4246873629 wait complete. 2024-03-11T19:54:19.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe9a90cb700 1 Processor -- start 2024-03-11T19:54:19.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe9a90cb700 1 -- start start 2024-03-11T19:54:19.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe9a90cb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9a4100ba0 0x7fe9a406aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:19.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe9a90cb700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4101da0 0x7fe9a419cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:19.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe9a90cb700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9a4102740 0x7fe9a419f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:19.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe9a90cb700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a419f8e0 con 0x7fe9a4102740 2024-03-11T19:54:19.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe9a90cb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a419fa50 con 0x7fe9a4100ba0 2024-03-11T19:54:19.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe9a90cb700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a419fc10 con 0x7fe9a4101da0 2024-03-11T19:54:19.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe99bfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4101da0 0x7fe9a419cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:19.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe99bfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4101da0 0x7fe9a419cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60652/0 (socket says 172.21.15.161:60652) 2024-03-11T19:54:19.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe99bfff700 1 -- 172.21.15.161:0/3639708062 learned_addr learned my addr 172.21.15.161:0/3639708062 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-11T19:54:19.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe9a2d9d700 1 --2- 172.21.15.161:0/3639708062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9a4100ba0 0x7fe9a406aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:19.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe99bfff700 1 -- 172.21.15.161:0/3639708062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9a4100ba0 msgr2=0x7fe9a406aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:19.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe99bfff700 1 --2- 172.21.15.161:0/3639708062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9a4100ba0 0x7fe9a406aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:19.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe99bfff700 1 -- 172.21.15.161:0/3639708062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9a4102740 msgr2=0x7fe9a419f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:19.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.649+0000 7fe99bfff700 1 --2- 172.21.15.161:0/3639708062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9a4102740 0x7fe9a419f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:19.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.650+0000 7fe99bfff700 1 -- 172.21.15.161:0/3639708062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98c0095f0 con 0x7fe9a4101da0 2024-03-11T19:54:19.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.650+0000 7fe99bfff700 1 --2- 172.21.15.161:0/3639708062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4101da0 0x7fe9a419cb00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe98c005910 tx=0x7fe98c00f700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:19.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.650+0000 7fe9a0d99700 1 -- 172.21.15.161:0/3639708062 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c01d070 con 0x7fe9a4101da0 2024-03-11T19:54:19.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.650+0000 7fe9a0d99700 1 -- 172.21.15.161:0/3639708062 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe98c00fbe0 con 0x7fe9a4101da0 2024-03-11T19:54:19.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.650+0000 7fe9a90cb700 1 -- 172.21.15.161:0/3639708062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a419fe90 con 0x7fe9a4101da0 2024-03-11T19:54:19.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.650+0000 7fe9a0d99700 1 -- 172.21.15.161:0/3639708062 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c017720 con 0x7fe9a4101da0 2024-03-11T19:54:19.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.650+0000 7fe9a90cb700 1 -- 172.21.15.161:0/3639708062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a41a0320 con 0x7fe9a4101da0 2024-03-11T19:54:19.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.651+0000 7fe9a90cb700 1 -- 172.21.15.161:0/3639708062 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a4108080 con 0x7fe9a4101da0 2024-03-11T19:54:19.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.652+0000 7fe9a0d99700 1 -- 172.21.15.161:0/3639708062 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe98c00fd50 con 0x7fe9a4101da0 2024-03-11T19:54:19.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.653+0000 7fe9a0d99700 1 --2- 172.21.15.161:0/3639708062 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9900a1b40 0x7fe9900a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:19.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.653+0000 7fe9a0d99700 1 -- 172.21.15.161:0/3639708062 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6723+0+0 (secure 0 0 0) 0x7fe98c0ce160 con 0x7fe9a4101da0 2024-03-11T19:54:19.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.653+0000 7fe9a2d9d700 1 --2- 172.21.15.161:0/3639708062 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9900a1b40 0x7fe9900a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:19.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.654+0000 7fe9a2d9d700 1 --2- 172.21.15.161:0/3639708062 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9900a1b40 0x7fe9900a4000 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4101c00 tx=0x7fe994009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:19.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.656+0000 7fe9a0d99700 1 -- 172.21.15.161:0/3639708062 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe98c05dd20 con 0x7fe9a4101da0 2024-03-11T19:54:19.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: Detected new or changed devices on smithi114 2024-03-11T19:54:19.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:19.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:19.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:19.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:19.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi114 to 4270M 2024-03-11T19:54:19.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:19.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:54:19.674 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.674 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 723 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:19.674 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:19.674 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:19.674 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:54:19.674 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.674 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:19 smithi097 ceph-mon[31898]: osdmap e74: 11 total, 11 up, 11 in 2024-03-11T19:54:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: Detected new or changed devices on smithi114 2024-03-11T19:54:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi114 to 4270M 2024-03-11T19:54:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:54:19.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 723 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:19.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:19.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:19.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:54:19.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:19.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:19 smithi114 ceph-mon[36376]: osdmap e74: 11 total, 11 up, 11 in 2024-03-11T19:54:19.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:19.859+0000 7fe9a90cb700 1 -- 172.21.15.161:0/3639708062 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9a402da20 con 0x7fe9900a1b40 2024-03-11T19:54:20.737 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:20 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:54:20.737 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:20 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:54:20.737 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:20 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:20.737 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:20 smithi161 ceph-mon[36332]: osdmap e75: 11 total, 11 up, 11 in 2024-03-11T19:54:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:20 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:54:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:20 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:54:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:20 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:20 smithi114 ceph-mon[36376]: osdmap e75: 11 total, 11 up, 11 in 2024-03-11T19:54:20.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:20 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T19:54:20.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:20 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T19:54:20.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:20 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:20.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:20 smithi097 ceph-mon[31898]: osdmap e75: 11 total, 11 up, 11 in 2024-03-11T19:54:21.727 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:21 smithi161 ceph-mon[36332]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:54:21.727 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:21 smithi161 ceph-mon[36332]: pgmap v263: 1 pgs: 1 remapped+peering; 577 KiB data, 724 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:21.727 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:21 smithi161 ceph-mon[36332]: osdmap e76: 11 total, 11 up, 11 in 2024-03-11T19:54:21.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:21 smithi114 ceph-mon[36376]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:54:21.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:21 smithi114 ceph-mon[36376]: pgmap v263: 1 pgs: 1 remapped+peering; 577 KiB data, 724 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:21.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:21 smithi114 ceph-mon[36376]: osdmap e76: 11 total, 11 up, 11 in 2024-03-11T19:54:21.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:21 smithi097 ceph-mon[31898]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:54:21.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:21 smithi097 ceph-mon[31898]: pgmap v263: 1 pgs: 1 remapped+peering; 577 KiB data, 724 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:21.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:21 smithi097 ceph-mon[31898]: osdmap e76: 11 total, 11 up, 11 in 2024-03-11T19:54:23.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:23 smithi161 ceph-mon[36332]: pgmap v265: 1 pgs: 1 remapped+peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:23.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:23 smithi161 ceph-mon[36332]: from='client.? 172.21.15.161:0/3517410036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f16c85e3-0e4a-4ae3-894f-6dfe489e7325"}]: dispatch 2024-03-11T19:54:23.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:23 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f16c85e3-0e4a-4ae3-894f-6dfe489e7325"}]: dispatch 2024-03-11T19:54:23.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:23 smithi161 ceph-mon[36332]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f16c85e3-0e4a-4ae3-894f-6dfe489e7325"}]': finished 2024-03-11T19:54:23.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:23 smithi161 ceph-mon[36332]: osdmap e77: 12 total, 11 up, 12 in 2024-03-11T19:54:23.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:23.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:23 smithi161 ceph-mon[36332]: from='client.? 172.21.15.161:0/2914849774' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:54:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:23 smithi097 ceph-mon[31898]: pgmap v265: 1 pgs: 1 remapped+peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:23 smithi097 ceph-mon[31898]: from='client.? 172.21.15.161:0/3517410036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f16c85e3-0e4a-4ae3-894f-6dfe489e7325"}]: dispatch 2024-03-11T19:54:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:23 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f16c85e3-0e4a-4ae3-894f-6dfe489e7325"}]: dispatch 2024-03-11T19:54:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:23 smithi097 ceph-mon[31898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f16c85e3-0e4a-4ae3-894f-6dfe489e7325"}]': finished 2024-03-11T19:54:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:23 smithi097 ceph-mon[31898]: osdmap e77: 12 total, 11 up, 12 in 2024-03-11T19:54:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:23.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:23 smithi097 ceph-mon[31898]: from='client.? 172.21.15.161:0/2914849774' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:54:23.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:23 smithi114 ceph-mon[36376]: pgmap v265: 1 pgs: 1 remapped+peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:23.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:23 smithi114 ceph-mon[36376]: from='client.? 172.21.15.161:0/3517410036' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f16c85e3-0e4a-4ae3-894f-6dfe489e7325"}]: dispatch 2024-03-11T19:54:23.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:23 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f16c85e3-0e4a-4ae3-894f-6dfe489e7325"}]: dispatch 2024-03-11T19:54:23.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:23 smithi114 ceph-mon[36376]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f16c85e3-0e4a-4ae3-894f-6dfe489e7325"}]': finished 2024-03-11T19:54:23.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:23 smithi114 ceph-mon[36376]: osdmap e77: 12 total, 11 up, 12 in 2024-03-11T19:54:23.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:23.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:23 smithi114 ceph-mon[36376]: from='client.? 172.21.15.161:0/2914849774' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T19:54:25.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:25 smithi161 ceph-mon[36332]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-11T19:54:25.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:25 smithi097 ceph-mon[31898]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-11T19:54:25.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:25 smithi114 ceph-mon[36376]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-11T19:54:27.604 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:27 smithi161 ceph-mon[36332]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-11T19:54:27.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:27 smithi097 ceph-mon[31898]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-11T19:54:27.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:27 smithi114 ceph-mon[36376]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-11T19:54:29.775 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:29 smithi161 ceph-mon[36332]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T19:54:29.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:29 smithi097 ceph-mon[31898]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T19:54:29.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:29 smithi114 ceph-mon[36376]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T19:54:30.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:30 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-11T19:54:30.905 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:30 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:30.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:30 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-11T19:54:30.923 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:30 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:30.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:30 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-11T19:54:30.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:30 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:31.817 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:31 smithi161 ceph-mon[36332]: Deploying daemon osd.11 on smithi161 2024-03-11T19:54:31.817 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:31 smithi161 ceph-mon[36332]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-11T19:54:31.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:31 smithi097 ceph-mon[31898]: Deploying daemon osd.11 on smithi161 2024-03-11T19:54:31.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:31 smithi097 ceph-mon[31898]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-11T19:54:31.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:31 smithi114 ceph-mon[36376]: Deploying daemon osd.11 on smithi161 2024-03-11T19:54:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:31 smithi114 ceph-mon[36376]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-11T19:54:33.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:33 smithi161 ceph-mon[36332]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:54:33.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:33 smithi097 ceph-mon[31898]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:54:33.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:33 smithi114 ceph-mon[36376]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T19:54:35.815 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:35 smithi161 ceph-mon[36332]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-11T19:54:35.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:35 smithi097 ceph-mon[31898]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-11T19:54:35.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:35 smithi114 ceph-mon[36376]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-11T19:54:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:36 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:36 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:36 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:37.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:36 smithi161 ceph-mon[36332]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:37.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:36 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:37.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:36 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:37.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:36 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:37.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:36 smithi097 ceph-mon[31898]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:36 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:36 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:36 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:36 smithi114 ceph-mon[36376]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:39.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:38 smithi097 ceph-mon[31898]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:39.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:38 smithi114 ceph-mon[36376]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:39.289 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:38 smithi161 ceph-mon[36332]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:39.349 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 11 on host 'smithi161' 2024-03-11T19:54:39.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:39.344+0000 7fe9a0d99700 1 -- 172.21.15.161:0/3639708062 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fe9a402da20 con 0x7fe9900a1b40 2024-03-11T19:54:39.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:39.346+0000 7fe9a90cb700 1 -- 172.21.15.161:0/3639708062 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9900a1b40 msgr2=0x7fe9900a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:39.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:39.346+0000 7fe9a90cb700 1 --2- 172.21.15.161:0/3639708062 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9900a1b40 0x7fe9900a4000 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4101c00 tx=0x7fe994009190 comp rx=0 tx=0).stop 2024-03-11T19:54:39.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:39.347+0000 7fe9a90cb700 1 -- 172.21.15.161:0/3639708062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4101da0 msgr2=0x7fe9a419cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:39.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:39.347+0000 7fe9a90cb700 1 --2- 172.21.15.161:0/3639708062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4101da0 0x7fe9a419cb00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe98c005910 tx=0x7fe98c00f700 comp rx=0 tx=0).stop 2024-03-11T19:54:39.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:39.347+0000 7fe9a90cb700 1 -- 172.21.15.161:0/3639708062 shutdown_connections 2024-03-11T19:54:39.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:39.347+0000 7fe9a90cb700 1 --2- 172.21.15.161:0/3639708062 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9900a1b40 0x7fe9900a4000 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:39.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:39.347+0000 7fe9a90cb700 1 --2- 172.21.15.161:0/3639708062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9a4102740 0x7fe9a419f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:39.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:39.347+0000 7fe9a90cb700 1 --2- 172.21.15.161:0/3639708062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9a4100ba0 0x7fe9a406aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:39.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:39.347+0000 7fe9a90cb700 1 --2- 172.21.15.161:0/3639708062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9a4101da0 0x7fe9a419cb00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:39.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:39.347+0000 7fe9a90cb700 1 -- 172.21.15.161:0/3639708062 >> 172.21.15.161:0/3639708062 conn(0x7fe9a40fc160 msgr2=0x7fe9a41051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:39.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:39.347+0000 7fe9a90cb700 1 -- 172.21.15.161:0/3639708062 shutdown_connections 2024-03-11T19:54:39.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:39.347+0000 7fe9a90cb700 1 -- 172.21.15.161:0/3639708062 wait complete. 2024-03-11T19:54:40.031 DEBUG:teuthology.orchestra.run.smithi161:osd.11> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.11.service 2024-03-11T19:54:40.034 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-11T19:54:40.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T19:54:40.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:40 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:40.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:40 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:40.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:40 smithi097 ceph-mon[31898]: from='osd.11 [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T19:54:40.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:40 smithi097 ceph-mon[31898]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T19:54:40.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:40 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:40.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:40 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:40.488 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:54:40.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:40 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:40.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:40 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:40.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:40 smithi114 ceph-mon[36376]: from='osd.11 [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T19:54:40.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:40 smithi114 ceph-mon[36376]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T19:54:40.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:40 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:40.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:40 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:40.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:40 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:40.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:40 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:40.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:40 smithi161 ceph-mon[36332]: from='osd.11 [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T19:54:40.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:40 smithi161 ceph-mon[36332]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T19:54:40.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:40 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:40.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:40 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:41.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:41 smithi161 ceph-mon[36332]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-11T19:54:41.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:41 smithi161 ceph-mon[36332]: from='osd.11 [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:54:41.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:41 smithi161 ceph-mon[36332]: osdmap e78: 12 total, 11 up, 12 in 2024-03-11T19:54:41.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:41 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:41.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:41 smithi161 ceph-mon[36332]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:54:41.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:41 smithi161 ceph-mon[36332]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:41.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:41 smithi097 ceph-mon[31898]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-11T19:54:41.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:41 smithi097 ceph-mon[31898]: from='osd.11 [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:54:41.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:41 smithi097 ceph-mon[31898]: osdmap e78: 12 total, 11 up, 12 in 2024-03-11T19:54:41.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:41 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:41.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:41 smithi097 ceph-mon[31898]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:54:41.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:41 smithi097 ceph-mon[31898]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:41.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:41 smithi114 ceph-mon[36376]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-11T19:54:41.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:41 smithi114 ceph-mon[36376]: from='osd.11 [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:54:41.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:41 smithi114 ceph-mon[36376]: osdmap e78: 12 total, 11 up, 12 in 2024-03-11T19:54:41.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:41 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:41.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:41 smithi114 ceph-mon[36376]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2024-03-11T19:54:41.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:41 smithi114 ceph-mon[36376]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:42.283 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 19:54:42 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T19:54:42.078+0000 7fdd26a10700 -1 osd.11 0 waiting for initial osdmap 2024-03-11T19:54:42.283 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 19:54:42 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T19:54:42.092+0000 7fdd1e7fb700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T19:54:42.532 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:42 smithi097 ceph-mon[31898]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-03-11T19:54:42.532 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:42 smithi097 ceph-mon[31898]: osdmap e79: 12 total, 11 up, 12 in 2024-03-11T19:54:42.532 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:42.532 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:42 smithi097 ceph-mon[31898]: from='osd.11 ' entity='osd.11' 2024-03-11T19:54:42.532 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:42.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:42 smithi161 ceph-mon[36332]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-03-11T19:54:42.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:42 smithi161 ceph-mon[36332]: osdmap e79: 12 total, 11 up, 12 in 2024-03-11T19:54:42.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:42.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:42 smithi161 ceph-mon[36332]: from='osd.11 ' entity='osd.11' 2024-03-11T19:54:42.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:42.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:42 smithi114 ceph-mon[36376]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2024-03-11T19:54:42.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:42 smithi114 ceph-mon[36376]: osdmap e79: 12 total, 11 up, 12 in 2024-03-11T19:54:42.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:42.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:42 smithi114 ceph-mon[36376]: from='osd.11 ' entity='osd.11' 2024-03-11T19:54:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:42.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.816+0000 7f6fd6400700 1 -- 172.21.15.97:0/2793002070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fd0102ea0 msgr2=0x7f6fd0103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:42.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.816+0000 7f6fd6400700 1 --2- 172.21.15.97:0/2793002070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fd0102ea0 0x7f6fd0103300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009990 tx=0x7f6fbc009ca0 comp rx=0 tx=0).stop 2024-03-11T19:54:42.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.816+0000 7f6fd6400700 1 -- 172.21.15.97:0/2793002070 shutdown_connections 2024-03-11T19:54:42.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.816+0000 7f6fd6400700 1 --2- 172.21.15.97:0/2793002070 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fd0103840 0x7f6fd010a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:42.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.816+0000 7f6fd6400700 1 --2- 172.21.15.97:0/2793002070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd0101ca0 0x7f6fd01020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.816+0000 7f6fd6400700 1 --2- 172.21.15.97:0/2793002070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fd0102ea0 0x7f6fd0103300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.816+0000 7f6fd6400700 1 -- 172.21.15.97:0/2793002070 >> 172.21.15.97:0/2793002070 conn(0x7f6fd00fd280 msgr2=0x7f6fd00ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.816+0000 7f6fd6400700 1 -- 172.21.15.97:0/2793002070 shutdown_connections 2024-03-11T19:54:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.816+0000 7f6fd6400700 1 -- 172.21.15.97:0/2793002070 wait complete. 2024-03-11T19:54:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.817+0000 7f6fd6400700 1 Processor -- start 2024-03-11T19:54:42.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.818+0000 7f6fd6400700 1 -- start start 2024-03-11T19:54:42.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.818+0000 7f6fd6400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fd0101ca0 0x7f6fd0075a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:42.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.818+0000 7f6fd6400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fd0102ea0 0x7f6fd0074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:42.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.818+0000 7f6fd6400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd0103840 0x7f6fd00745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:42.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.818+0000 7f6fd6400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0074d20 con 0x7f6fd0102ea0 2024-03-11T19:54:42.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.818+0000 7f6fd6400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0074e90 con 0x7f6fd0103840 2024-03-11T19:54:42.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.818+0000 7f6fd6400700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd01a1ce0 con 0x7f6fd0101ca0 2024-03-11T19:54:42.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.818+0000 7f6fcf400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fd0101ca0 0x7f6fd0075a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:42.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.818+0000 7f6fcf400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fd0101ca0 0x7f6fd0075a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:47952/0 (socket says 172.21.15.97:47952) 2024-03-11T19:54:42.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.818+0000 7f6fcf400700 1 -- 172.21.15.97:0/1416619626 learned_addr learned my addr 172.21.15.97:0/1416619626 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:54:42.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.818+0000 7f6fcea00700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fd0102ea0 0x7f6fd0074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:42.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.818+0000 7f6fcfe00700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd0103840 0x7f6fd00745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.819+0000 7f6fcf400700 1 -- 172.21.15.97:0/1416619626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd0103840 msgr2=0x7f6fd00745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.819+0000 7f6fcf400700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd0103840 0x7f6fd00745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.819+0000 7f6fcf400700 1 -- 172.21.15.97:0/1416619626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fd0102ea0 msgr2=0x7f6fd0074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.819+0000 7f6fcf400700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fd0102ea0 0x7f6fd0074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.819+0000 7f6fcf400700 1 -- 172.21.15.97:0/1416619626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc0095f0 con 0x7f6fd0101ca0 2024-03-11T19:54:42.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.819+0000 7f6fcea00700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fd0102ea0 0x7f6fd0074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:54:42.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.819+0000 7f6fcf400700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fd0101ca0 0x7f6fd0075a10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000e970 tx=0x7f6fc000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:42.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.819+0000 7f6fc7e00700 1 -- 172.21.15.97:0/1416619626 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0009790 con 0x7f6fd0101ca0 2024-03-11T19:54:42.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.819+0000 7f6fc7e00700 1 -- 172.21.15.97:0/1416619626 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6fc000ccc0 con 0x7f6fd0101ca0 2024-03-11T19:54:42.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.820+0000 7f6fd6400700 1 -- 172.21.15.97:0/1416619626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd01a1fc0 con 0x7f6fd0101ca0 2024-03-11T19:54:42.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.820+0000 7f6fc7e00700 1 -- 172.21.15.97:0/1416619626 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0003d80 con 0x7f6fd0101ca0 2024-03-11T19:54:42.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.820+0000 7f6fd6400700 1 -- 172.21.15.97:0/1416619626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd01a23e0 con 0x7f6fd0101ca0 2024-03-11T19:54:42.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.821+0000 7f6fd6400700 1 -- 172.21.15.97:0/1416619626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd0068930 con 0x7f6fd0101ca0 2024-03-11T19:54:42.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.824+0000 7f6fc7e00700 1 -- 172.21.15.97:0/1416619626 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6fc000c7e0 con 0x7f6fd0101ca0 2024-03-11T19:54:42.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.825+0000 7f6fc7e00700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6fb40a1b40 0x7f6fb40a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:42.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.825+0000 7f6fc7e00700 1 -- 172.21.15.97:0/1416619626 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f6fc0014070 con 0x7f6fd0101ca0 2024-03-11T19:54:42.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.826+0000 7f6fcea00700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6fb40a1b40 0x7f6fb40a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:42.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.826+0000 7f6fcea00700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6fb40a1b40 0x7f6fb40a4000 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc000c00 tx=0x7f6fbc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:42.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:42.829+0000 7f6fc7e00700 1 -- 172.21.15.97:0/1416619626 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6fc0096720 con 0x7f6fd0101ca0 2024-03-11T19:54:43.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.047+0000 7f6fd6400700 1 -- 172.21.15.97:0/1416619626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f6fd002d8e0 con 0x7f6fd0101ca0 2024-03-11T19:54:43.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.048+0000 7f6fc7e00700 1 -- 172.21.15.97:0/1416619626 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7f6fc0099de0 con 0x7f6fd0101ca0 2024-03-11T19:54:43.053 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:54:43.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.051+0000 7f6fd6400700 1 -- 172.21.15.97:0/1416619626 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6fb40a1b40 msgr2=0x7f6fb40a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:43.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.051+0000 7f6fd6400700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6fb40a1b40 0x7f6fb40a4000 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc000c00 tx=0x7f6fbc005fb0 comp rx=0 tx=0).stop 2024-03-11T19:54:43.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.051+0000 7f6fd6400700 1 -- 172.21.15.97:0/1416619626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fd0101ca0 msgr2=0x7f6fd0075a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:43.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.051+0000 7f6fd6400700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fd0101ca0 0x7f6fd0075a10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000e970 tx=0x7f6fc000ec80 comp rx=0 tx=0).stop 2024-03-11T19:54:43.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.051+0000 7f6fd6400700 1 -- 172.21.15.97:0/1416619626 shutdown_connections 2024-03-11T19:54:43.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.051+0000 7f6fd6400700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6fb40a1b40 0x7f6fb40a4000 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:43.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.051+0000 7f6fd6400700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd0103840 0x7f6fd00745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:43.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.051+0000 7f6fd6400700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6fd0101ca0 0x7f6fd0075a10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:43.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.051+0000 7f6fd6400700 1 --2- 172.21.15.97:0/1416619626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6fd0102ea0 0x7f6fd0074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:43.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.051+0000 7f6fd6400700 1 -- 172.21.15.97:0/1416619626 >> 172.21.15.97:0/1416619626 conn(0x7f6fd00fd280 msgr2=0x7f6fd0106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:43.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.051+0000 7f6fd6400700 1 -- 172.21.15.97:0/1416619626 shutdown_connections 2024-03-11T19:54:43.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:43.051+0000 7f6fd6400700 1 -- 172.21.15.97:0/1416619626 wait complete. 2024-03-11T19:54:43.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:43 smithi097 ceph-mon[31898]: purged_snaps scrub starts 2024-03-11T19:54:43.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:43 smithi097 ceph-mon[31898]: purged_snaps scrub ok 2024-03-11T19:54:43.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:43 smithi097 ceph-mon[31898]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:43.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:43 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1416619626' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T19:54:43.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:43 smithi097 ceph-mon[31898]: osd.11 [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703] boot 2024-03-11T19:54:43.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:43 smithi097 ceph-mon[31898]: osdmap e80: 12 total, 12 up, 12 in 2024-03-11T19:54:43.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:43 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:43.696 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:43 smithi161 ceph-mon[36332]: purged_snaps scrub starts 2024-03-11T19:54:43.696 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:43 smithi161 ceph-mon[36332]: purged_snaps scrub ok 2024-03-11T19:54:43.696 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:43 smithi161 ceph-mon[36332]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:43.696 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:43 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1416619626' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T19:54:43.696 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:43 smithi161 ceph-mon[36332]: osd.11 [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703] boot 2024-03-11T19:54:43.696 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:43 smithi161 ceph-mon[36332]: osdmap e80: 12 total, 12 up, 12 in 2024-03-11T19:54:43.696 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:43 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:43.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:43 smithi114 ceph-mon[36376]: purged_snaps scrub starts 2024-03-11T19:54:43.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:43 smithi114 ceph-mon[36376]: purged_snaps scrub ok 2024-03-11T19:54:43.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:43 smithi114 ceph-mon[36376]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-03-11T19:54:43.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:43 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1416619626' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T19:54:43.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:43 smithi114 ceph-mon[36376]: osd.11 [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703] boot 2024-03-11T19:54:43.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:43 smithi114 ceph-mon[36376]: osdmap e80: 12 total, 12 up, 12 in 2024-03-11T19:54:43.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:43 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:54:43.920 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1710186857,"num_in_osds":12,"osd_in_since":1710186862,"num_remapped_pgs":0} 2024-03-11T19:54:44.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T19:54:45.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: Detected new or changed devices on smithi161 2024-03-11T19:54:45.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:45.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:45.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:45.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:45.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi161 to 4270M 2024-03-11T19:54:45.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.323 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:45.324 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:54:45.324 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.324 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:45.324 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:45.324 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:54:45.324 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.324 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: osdmap e81: 12 total, 12 up, 12 in 2024-03-11T19:54:45.324 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:45 smithi097 ceph-mon[31898]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:45.364 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:54:45.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: Detected new or changed devices on smithi161 2024-03-11T19:54:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi161 to 4270M 2024-03-11T19:54:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:54:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:45.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:45.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:54:45.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: osdmap e81: 12 total, 12 up, 12 in 2024-03-11T19:54:45.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:45 smithi114 ceph-mon[36376]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: Detected new or changed devices on smithi161 2024-03-11T19:54:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:54:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi161 to 4270M 2024-03-11T19:54:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:54:45.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:45.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:54:45.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:54:45.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:45.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: osdmap e81: 12 total, 12 up, 12 in 2024-03-11T19:54:45.541 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:45 smithi161 ceph-mon[36332]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:47.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.639+0000 7f1d4d200700 1 -- 172.21.15.97:0/926622059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d48103890 msgr2=0x7f1d4810a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:47.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.639+0000 7f1d4d200700 1 --2- 172.21.15.97:0/926622059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d48103890 0x7f1d4810a140 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009fc0 tx=0x7f1d4000b330 comp rx=0 tx=0).stop 2024-03-11T19:54:47.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.639+0000 7f1d4d200700 1 -- 172.21.15.97:0/926622059 shutdown_connections 2024-03-11T19:54:47.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.639+0000 7f1d4d200700 1 --2- 172.21.15.97:0/926622059 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d48101cf0 0x7f1d48102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:47.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.639+0000 7f1d4d200700 1 --2- 172.21.15.97:0/926622059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d48103890 0x7f1d4810a140 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:47.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.639+0000 7f1d4d200700 1 --2- 172.21.15.97:0/926622059 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d48102ef0 0x7f1d48103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:47.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.640+0000 7f1d4d200700 1 -- 172.21.15.97:0/926622059 >> 172.21.15.97:0/926622059 conn(0x7f1d480fd290 msgr2=0x7f1d480ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:47.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.640+0000 7f1d4d200700 1 -- 172.21.15.97:0/926622059 shutdown_connections 2024-03-11T19:54:47.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.640+0000 7f1d4d200700 1 -- 172.21.15.97:0/926622059 wait complete. 2024-03-11T19:54:47.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.641+0000 7f1d4d200700 1 Processor -- start 2024-03-11T19:54:47.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.641+0000 7f1d4d200700 1 -- start start 2024-03-11T19:54:47.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.641+0000 7f1d4d200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d48101cf0 0x7f1d48075a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:47.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.641+0000 7f1d4d200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d48102ef0 0x7f1d48074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:47.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.641+0000 7f1d4d200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d48103890 0x7f1d480745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:47.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d4d200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d48074d20 con 0x7f1d48103890 2024-03-11T19:54:47.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d4d200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d48074e90 con 0x7f1d48102ef0 2024-03-11T19:54:47.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d4d200700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d481a1dd0 con 0x7f1d48101cf0 2024-03-11T19:54:47.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d46800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d48103890 0x7f1d480745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:47.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d46800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d48103890 0x7f1d480745a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36546/0 (socket says 172.21.15.97:36546) 2024-03-11T19:54:47.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d46800700 1 -- 172.21.15.97:0/2527382372 learned_addr learned my addr 172.21.15.97:0/2527382372 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:54:47.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d45400700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d48102ef0 0x7f1d48074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:47.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d45e00700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d48101cf0 0x7f1d48075a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d45e00700 1 -- 172.21.15.97:0/2527382372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d48102ef0 msgr2=0x7f1d48074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d45e00700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d48102ef0 0x7f1d48074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d45e00700 1 -- 172.21.15.97:0/2527382372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d48103890 msgr2=0x7f1d480745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d45e00700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d48103890 0x7f1d480745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d45e00700 1 -- 172.21.15.97:0/2527382372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d40009c40 con 0x7f1d48101cf0 2024-03-11T19:54:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.642+0000 7f1d46800700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d48103890 0x7f1d480745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:54:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.643+0000 7f1d45e00700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d48101cf0 0x7f1d48075a10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1d38005fd0 tx=0x7f1d3800eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.643+0000 7f1d3ea00700 1 -- 172.21.15.97:0/2527382372 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d38004220 con 0x7f1d48101cf0 2024-03-11T19:54:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.643+0000 7f1d3ea00700 1 -- 172.21.15.97:0/2527382372 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d38004380 con 0x7f1d48101cf0 2024-03-11T19:54:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.643+0000 7f1d4d200700 1 -- 172.21.15.97:0/2527382372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d481a20b0 con 0x7f1d48101cf0 2024-03-11T19:54:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.643+0000 7f1d3ea00700 1 -- 172.21.15.97:0/2527382372 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d38003c70 con 0x7f1d48101cf0 2024-03-11T19:54:47.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.643+0000 7f1d4d200700 1 -- 172.21.15.97:0/2527382372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d481a25b0 con 0x7f1d48101cf0 2024-03-11T19:54:47.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.644+0000 7f1d4d200700 1 -- 172.21.15.97:0/2527382372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d48108fe0 con 0x7f1d48101cf0 2024-03-11T19:54:47.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.646+0000 7f1d3ea00700 1 -- 172.21.15.97:0/2527382372 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1d38010700 con 0x7f1d48101cf0 2024-03-11T19:54:47.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.647+0000 7f1d3ea00700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1d2c0a1ac0 0x7f1d2c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:47.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.648+0000 7f1d3ea00700 1 -- 172.21.15.97:0/2527382372 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1d38014070 con 0x7f1d48101cf0 2024-03-11T19:54:47.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.648+0000 7f1d45400700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1d2c0a1ac0 0x7f1d2c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:47.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.648+0000 7f1d45400700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1d2c0a1ac0 0x7f1d2c0a3f80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009c60 tx=0x7f1d34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:47.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.652+0000 7f1d3ea00700 1 -- 172.21.15.97:0/2527382372 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1d38096650 con 0x7f1d48101cf0 2024-03-11T19:54:47.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:47 smithi161 ceph-mon[36332]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:47.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.872+0000 7f1d4d200700 1 -- 172.21.15.97:0/2527382372 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f1d48068930 con 0x7f1d48101cf0 2024-03-11T19:54:47.878 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:47 smithi097 ceph-mon[31898]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:47.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.873+0000 7f1d3ea00700 1 -- 172.21.15.97:0/2527382372 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7f1d38099d10 con 0x7f1d48101cf0 2024-03-11T19:54:47.878 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:54:47.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.876+0000 7f1d4d200700 1 -- 172.21.15.97:0/2527382372 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1d2c0a1ac0 msgr2=0x7f1d2c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:47.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.876+0000 7f1d4d200700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1d2c0a1ac0 0x7f1d2c0a3f80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009c60 tx=0x7f1d34009260 comp rx=0 tx=0).stop 2024-03-11T19:54:47.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.876+0000 7f1d4d200700 1 -- 172.21.15.97:0/2527382372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d48101cf0 msgr2=0x7f1d48075a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:47.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.876+0000 7f1d4d200700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d48101cf0 0x7f1d48075a10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1d38005fd0 tx=0x7f1d3800eca0 comp rx=0 tx=0).stop 2024-03-11T19:54:47.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.876+0000 7f1d4d200700 1 -- 172.21.15.97:0/2527382372 shutdown_connections 2024-03-11T19:54:47.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.876+0000 7f1d4d200700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1d2c0a1ac0 0x7f1d2c0a3f80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:47.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.876+0000 7f1d4d200700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1d48103890 0x7f1d480745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:47.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.876+0000 7f1d4d200700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d48101cf0 0x7f1d48075a10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:47.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.876+0000 7f1d4d200700 1 --2- 172.21.15.97:0/2527382372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d48102ef0 0x7f1d48074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:47.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.876+0000 7f1d4d200700 1 -- 172.21.15.97:0/2527382372 >> 172.21.15.97:0/2527382372 conn(0x7f1d480fd290 msgr2=0x7f1d48106120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:47.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.877+0000 7f1d4d200700 1 -- 172.21.15.97:0/2527382372 shutdown_connections 2024-03-11T19:54:47.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:47.877+0000 7f1d4d200700 1 -- 172.21.15.97:0/2527382372 wait complete. 2024-03-11T19:54:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:47 smithi114 ceph-mon[36376]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:48.662 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1710186883,"num_in_osds":12,"osd_in_since":1710186862,"num_remapped_pgs":0} 2024-03-11T19:54:48.663 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T19:54:48.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:48 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2527382372' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T19:54:48.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:48 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2527382372' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T19:54:48.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:48 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2527382372' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T19:54:49.112 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:54:49.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:49 smithi097 ceph-mon[31898]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:49.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:49 smithi114 ceph-mon[36376]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:50.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:49 smithi161 ceph-mon[36332]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.340+0000 7f12bbe00700 1 -- 172.21.15.97:0/3139020823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc102d70 msgr2=0x7f12bc103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.340+0000 7f12bbe00700 1 --2- 172.21.15.97:0/3139020823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc102d70 0x7f12bc103150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009990 tx=0x7f12ac009ca0 comp rx=0 tx=0).stop 2024-03-11T19:54:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.340+0000 7f12bbe00700 1 -- 172.21.15.97:0/3139020823 shutdown_connections 2024-03-11T19:54:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.340+0000 7f12bbe00700 1 --2- 172.21.15.97:0/3139020823 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12bc10c150 0x7f12bc10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.340+0000 7f12bbe00700 1 --2- 172.21.15.97:0/3139020823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc102d70 0x7f12bc103150 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.340+0000 7f12bbe00700 1 --2- 172.21.15.97:0/3139020823 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12bc103690 0x7f12bc10bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.340+0000 7f12bbe00700 1 -- 172.21.15.97:0/3139020823 >> 172.21.15.97:0/3139020823 conn(0x7f12bc0fc540 msgr2=0x7f12bc0fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.340+0000 7f12bbe00700 1 -- 172.21.15.97:0/3139020823 shutdown_connections 2024-03-11T19:54:51.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.340+0000 7f12bbe00700 1 -- 172.21.15.97:0/3139020823 wait complete. 2024-03-11T19:54:51.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.341+0000 7f12bbe00700 1 Processor -- start 2024-03-11T19:54:51.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.341+0000 7f12bbe00700 1 -- start start 2024-03-11T19:54:51.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.342+0000 7f12bbe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc102d70 0x7f12bc19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:51.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.342+0000 7f12bbe00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12bc103690 0x7f12bc19a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:51.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.342+0000 7f12bbe00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12bc10c150 0x7f12bc19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:51.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.342+0000 7f12bbe00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc19f190 con 0x7f12bc10c150 2024-03-11T19:54:51.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.342+0000 7f12bbe00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc19f300 con 0x7f12bc102d70 2024-03-11T19:54:51.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.342+0000 7f12bbe00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc19f4c0 con 0x7f12bc103690 2024-03-11T19:54:51.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.342+0000 7f12bb400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12bc10c150 0x7f12bc19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:51.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.342+0000 7f12baa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc102d70 0x7f12bc19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:51.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.342+0000 7f12bb400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12bc10c150 0x7f12bc19eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:36568/0 (socket says 172.21.15.97:36568) 2024-03-11T19:54:51.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.342+0000 7f12bb400700 1 -- 172.21.15.97:0/156803677 learned_addr learned my addr 172.21.15.97:0/156803677 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:54:51.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.342+0000 7f12ba000700 1 --2- 172.21.15.97:0/156803677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12bc103690 0x7f12bc19a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:51.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.343+0000 7f12baa00700 1 -- 172.21.15.97:0/156803677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12bc103690 msgr2=0x7f12bc19a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:51.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.343+0000 7f12baa00700 1 --2- 172.21.15.97:0/156803677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12bc103690 0x7f12bc19a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:51.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.343+0000 7f12baa00700 1 -- 172.21.15.97:0/156803677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12bc10c150 msgr2=0x7f12bc19eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:51.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.343+0000 7f12baa00700 1 --2- 172.21.15.97:0/156803677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12bc10c150 0x7f12bc19eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:51.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.343+0000 7f12baa00700 1 -- 172.21.15.97:0/156803677 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4009c40 con 0x7f12bc102d70 2024-03-11T19:54:51.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.343+0000 7f12baa00700 1 --2- 172.21.15.97:0/156803677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc102d70 0x7f12bc19a2b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0059d0 tx=0x7f12ac004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:51.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.343+0000 7f12b3400700 1 -- 172.21.15.97:0/156803677 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac01d070 con 0x7f12bc102d70 2024-03-11T19:54:51.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.343+0000 7f12b3400700 1 -- 172.21.15.97:0/156803677 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12ac00b610 con 0x7f12bc102d70 2024-03-11T19:54:51.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.343+0000 7f12bbe00700 1 -- 172.21.15.97:0/156803677 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ac0095f0 con 0x7f12bc102d70 2024-03-11T19:54:51.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.343+0000 7f12b3400700 1 -- 172.21.15.97:0/156803677 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac0173f0 con 0x7f12bc102d70 2024-03-11T19:54:51.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.344+0000 7f12bbe00700 1 -- 172.21.15.97:0/156803677 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12bc1a6e30 con 0x7f12bc102d70 2024-03-11T19:54:51.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.344+0000 7f12bbe00700 1 -- 172.21.15.97:0/156803677 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12bc068930 con 0x7f12bc102d70 2024-03-11T19:54:51.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.347+0000 7f12b3400700 1 -- 172.21.15.97:0/156803677 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f12ac022590 con 0x7f12bc102d70 2024-03-11T19:54:51.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.348+0000 7f12b3400700 1 --2- 172.21.15.97:0/156803677 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12a00a1b40 0x7f12a00a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:51.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.348+0000 7f12b3400700 1 -- 172.21.15.97:0/156803677 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f12ac0ce8e0 con 0x7f12bc102d70 2024-03-11T19:54:51.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.348+0000 7f12ba000700 1 --2- 172.21.15.97:0/156803677 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12a00a1b40 0x7f12a00a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:51.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.349+0000 7f12ba000700 1 --2- 172.21.15.97:0/156803677 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12a00a1b40 0x7f12a00a4000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f12bc10da60 tx=0x7f12a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:51.355 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.351+0000 7f12b3400700 1 -- 172.21.15.97:0/156803677 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f12ac097f80 con 0x7f12bc102d70 2024-03-11T19:54:51.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.572+0000 7f12bbe00700 1 -- 172.21.15.97:0/156803677 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12bc111bf0 con 0x7f12bc102d70 2024-03-11T19:54:51.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.574+0000 7f12b3400700 1 -- 172.21.15.97:0/156803677 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18480 (secure 0 0 0) 0x7f12ac09b640 con 0x7f12bc102d70 2024-03-11T19:54:51.578 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:54:51.578 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":81,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:54:44.381499+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:54:51.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.577+0000 7f12bbe00700 1 -- 172.21.15.97:0/156803677 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12a00a1b40 msgr2=0x7f12a00a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:51.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.577+0000 7f12bbe00700 1 --2- 172.21.15.97:0/156803677 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12a00a1b40 0x7f12a00a4000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f12bc10da60 tx=0x7f12a8009260 comp rx=0 tx=0).stop 2024-03-11T19:54:51.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.577+0000 7f12bbe00700 1 -- 172.21.15.97:0/156803677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc102d70 msgr2=0x7f12bc19a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:51.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.577+0000 7f12bbe00700 1 --2- 172.21.15.97:0/156803677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc102d70 0x7f12bc19a2b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0059d0 tx=0x7f12ac004730 comp rx=0 tx=0).stop 2024-03-11T19:54:51.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.578+0000 7f12bbe00700 1 -- 172.21.15.97:0/156803677 shutdown_connections 2024-03-11T19:54:51.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.578+0000 7f12bbe00700 1 --2- 172.21.15.97:0/156803677 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12a00a1b40 0x7f12a00a4000 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:51.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.578+0000 7f12bbe00700 1 --2- 172.21.15.97:0/156803677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12bc10c150 0x7f12bc19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:51.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.578+0000 7f12bbe00700 1 --2- 172.21.15.97:0/156803677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12bc102d70 0x7f12bc19a2b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:51.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.578+0000 7f12bbe00700 1 --2- 172.21.15.97:0/156803677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12bc103690 0x7f12bc19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:51.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.578+0000 7f12bbe00700 1 -- 172.21.15.97:0/156803677 >> 172.21.15.97:0/156803677 conn(0x7f12bc0fc540 msgr2=0x7f12bc0fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:51.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.578+0000 7f12bbe00700 1 -- 172.21.15.97:0/156803677 shutdown_connections 2024-03-11T19:54:51.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:51.578+0000 7f12bbe00700 1 -- 172.21.15.97:0/156803677 wait complete. 2024-03-11T19:54:51.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:51 smithi097 ceph-mon[31898]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:51.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:51 smithi114 ceph-mon[36376]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:52.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:51 smithi161 ceph-mon[36332]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:52.448 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:50:29.051985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:54:52.449 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T19:54:52.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:52 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/156803677' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:54:52.899 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:54:52.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:52 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/156803677' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:54:53.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:52 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/156803677' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:54:53.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:53 smithi097 ceph-mon[31898]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:53.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:53 smithi114 ceph-mon[36376]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:54.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:53 smithi161 ceph-mon[36332]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:55.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.056+0000 7f8d46e00700 1 -- 172.21.15.97:0/3447070632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d40112ae0 msgr2=0x7f8d40114ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:55.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.056+0000 7f8d46e00700 1 --2- 172.21.15.97:0/3447070632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d40112ae0 0x7f8d40114ed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8d38009f90 tx=0x7f8d3800b330 comp rx=0 tx=0).stop 2024-03-11T19:54:55.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.056+0000 7f8d46e00700 1 -- 172.21.15.97:0/3447070632 shutdown_connections 2024-03-11T19:54:55.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.056+0000 7f8d46e00700 1 --2- 172.21.15.97:0/3447070632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d40112ae0 0x7f8d40114ed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:55.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.056+0000 7f8d46e00700 1 --2- 172.21.15.97:0/3447070632 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d40077dd0 0x7f8d400781b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:55.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.056+0000 7f8d46e00700 1 --2- 172.21.15.97:0/3447070632 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d40078780 0x7f8d401125a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:55.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.056+0000 7f8d46e00700 1 -- 172.21.15.97:0/3447070632 >> 172.21.15.97:0/3447070632 conn(0x7f8d400ff910 msgr2=0x7f8d40101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:55.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.056+0000 7f8d46e00700 1 -- 172.21.15.97:0/3447070632 shutdown_connections 2024-03-11T19:54:55.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.056+0000 7f8d46e00700 1 -- 172.21.15.97:0/3447070632 wait complete. 2024-03-11T19:54:55.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.057+0000 7f8d46e00700 1 Processor -- start 2024-03-11T19:54:55.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.057+0000 7f8d46e00700 1 -- start start 2024-03-11T19:54:55.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.058+0000 7f8d46e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d40077dd0 0x7f8d401a4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.058+0000 7f8d46e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d40078780 0x7f8d401a4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.058+0000 7f8d46e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d40112ae0 0x7f8d4019e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.058+0000 7f8d46e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4019ee80 con 0x7f8d40078780 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.058+0000 7f8d46e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4019eff0 con 0x7f8d40112ae0 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.058+0000 7f8d46e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4019f1b0 con 0x7f8d40077dd0 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.058+0000 7f8d3fe00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d40077dd0 0x7f8d401a4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.058+0000 7f8d3fe00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d40077dd0 0x7f8d401a4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:59596/0 (socket says 172.21.15.97:59596) 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.058+0000 7f8d3fe00700 1 -- 172.21.15.97:0/2024910237 learned_addr learned my addr 172.21.15.97:0/2024910237 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.058+0000 7f8d3f400700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d40078780 0x7f8d401a4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.058+0000 7f8d44e00700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d40112ae0 0x7f8d4019e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.059+0000 7f8d3fe00700 1 -- 172.21.15.97:0/2024910237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d40112ae0 msgr2=0x7f8d4019e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.059+0000 7f8d3fe00700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d40112ae0 0x7f8d4019e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.059+0000 7f8d3fe00700 1 -- 172.21.15.97:0/2024910237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d40078780 msgr2=0x7f8d401a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.059+0000 7f8d3fe00700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d40078780 0x7f8d401a4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:55.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.059+0000 7f8d3fe00700 1 -- 172.21.15.97:0/2024910237 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c0095f0 con 0x7f8d40077dd0 2024-03-11T19:54:55.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.059+0000 7f8d44e00700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d40112ae0 0x7f8d4019e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:54:55.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.059+0000 7f8d3fe00700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d40077dd0 0x7f8d401a4650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000e9a0 tx=0x7f8d3000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:55.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.059+0000 7f8d3cc00700 1 -- 172.21.15.97:0/2024910237 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30009790 con 0x7f8d40077dd0 2024-03-11T19:54:55.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.059+0000 7f8d3cc00700 1 -- 172.21.15.97:0/2024910237 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d30010430 con 0x7f8d40077dd0 2024-03-11T19:54:55.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.059+0000 7f8d46e00700 1 -- 172.21.15.97:0/2024910237 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d38009c40 con 0x7f8d40077dd0 2024-03-11T19:54:55.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.059+0000 7f8d3cc00700 1 -- 172.21.15.97:0/2024910237 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30018680 con 0x7f8d40077dd0 2024-03-11T19:54:55.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.059+0000 7f8d46e00700 1 -- 172.21.15.97:0/2024910237 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d401ab4b0 con 0x7f8d40077dd0 2024-03-11T19:54:55.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.060+0000 7f8d46e00700 1 -- 172.21.15.97:0/2024910237 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4010dd00 con 0x7f8d40077dd0 2024-03-11T19:54:55.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.062+0000 7f8d3cc00700 1 -- 172.21.15.97:0/2024910237 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8d30010910 con 0x7f8d40077dd0 2024-03-11T19:54:55.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.064+0000 7f8d3cc00700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8d240a1ac0 0x7f8d240a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:55.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.064+0000 7f8d3cc00700 1 -- 172.21.15.97:0/2024910237 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8d30014070 con 0x7f8d40077dd0 2024-03-11T19:54:55.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.064+0000 7f8d3f400700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8d240a1ac0 0x7f8d240a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:55.068 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.064+0000 7f8d3f400700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8d240a1ac0 0x7f8d240a3f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8d40077050 tx=0x7f8d2c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:55.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.068+0000 7f8d3cc00700 1 -- 172.21.15.97:0/2024910237 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8d30052d20 con 0x7f8d40077dd0 2024-03-11T19:54:55.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.286+0000 7f8d46e00700 1 -- 172.21.15.97:0/2024910237 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d4019fd00 con 0x7f8d40077dd0 2024-03-11T19:54:55.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.287+0000 7f8d3cc00700 1 -- 172.21.15.97:0/2024910237 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8d30096840 con 0x7f8d40077dd0 2024-03-11T19:54:55.293 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-11T19:54:55.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.290+0000 7f8d46e00700 1 -- 172.21.15.97:0/2024910237 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8d240a1ac0 msgr2=0x7f8d240a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:55.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.290+0000 7f8d46e00700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8d240a1ac0 0x7f8d240a3f80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8d40077050 tx=0x7f8d2c009310 comp rx=0 tx=0).stop 2024-03-11T19:54:55.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.290+0000 7f8d46e00700 1 -- 172.21.15.97:0/2024910237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d40077dd0 msgr2=0x7f8d401a4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:55.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.290+0000 7f8d46e00700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d40077dd0 0x7f8d401a4650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000e9a0 tx=0x7f8d3000ecb0 comp rx=0 tx=0).stop 2024-03-11T19:54:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.290+0000 7f8d46e00700 1 -- 172.21.15.97:0/2024910237 shutdown_connections 2024-03-11T19:54:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.290+0000 7f8d46e00700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8d240a1ac0 0x7f8d240a3f80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.290+0000 7f8d46e00700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d40112ae0 0x7f8d4019e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.290+0000 7f8d46e00700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d40077dd0 0x7f8d401a4650 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.290+0000 7f8d46e00700 1 --2- 172.21.15.97:0/2024910237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8d40078780 0x7f8d401a4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.290+0000 7f8d46e00700 1 -- 172.21.15.97:0/2024910237 >> 172.21.15.97:0/2024910237 conn(0x7f8d400ff910 msgr2=0x7f8d40101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.290+0000 7f8d46e00700 1 -- 172.21.15.97:0/2024910237 shutdown_connections 2024-03-11T19:54:55.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:55.290+0000 7f8d46e00700 1 -- 172.21.15.97:0/2024910237 wait complete. 2024-03-11T19:54:55.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:55 smithi097 ceph-mon[31898]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:55.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:55 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2024910237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:54:55.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:55 smithi114 ceph-mon[36376]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:55.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:55 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2024910237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:54:56.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:55 smithi161 ceph-mon[36332]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:56.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:55 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2024910237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:54:56.187 INFO:tasks.cephadm:Adding mds.a on smithi097 2024-03-11T19:54:56.187 INFO:tasks.cephadm:Adding mds.d on smithi097 2024-03-11T19:54:56.187 INFO:tasks.cephadm:Adding mds.g on smithi097 2024-03-11T19:54:56.187 INFO:tasks.cephadm:Adding mds.j on smithi097 2024-03-11T19:54:56.187 INFO:tasks.cephadm:Adding mds.b on smithi114 2024-03-11T19:54:56.187 INFO:tasks.cephadm:Adding mds.e on smithi114 2024-03-11T19:54:56.188 INFO:tasks.cephadm:Adding mds.h on smithi114 2024-03-11T19:54:56.188 INFO:tasks.cephadm:Adding mds.k on smithi114 2024-03-11T19:54:56.188 INFO:tasks.cephadm:Adding mds.c on smithi161 2024-03-11T19:54:56.188 INFO:tasks.cephadm:Adding mds.f on smithi161 2024-03-11T19:54:56.188 INFO:tasks.cephadm:Adding mds.i on smithi161 2024-03-11T19:54:56.188 INFO:tasks.cephadm:Adding mds.l on smithi161 2024-03-11T19:54:56.188 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph orch apply mds all '12;smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi114=b;smithi114=e;smithi114=h;smithi114=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l' 2024-03-11T19:54:56.438 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T19:54:57.915 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:57 smithi161 ceph-mon[36332]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:57.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:57 smithi097 ceph-mon[31898]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:57.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:57 smithi114 ceph-mon[36376]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:58.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.151+0000 7fc83fcda700 1 -- 172.21.15.161:0/1386019905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8381032c0 msgr2=0x7fc8381036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:58.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.151+0000 7fc83fcda700 1 --2- 172.21.15.161:0/1386019905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8381032c0 0x7fc8381036e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc834009990 tx=0x7fc834009ca0 comp rx=0 tx=0).stop 2024-03-11T19:54:58.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.151+0000 7fc83fcda700 1 -- 172.21.15.161:0/1386019905 shutdown_connections 2024-03-11T19:54:58.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.151+0000 7fc83fcda700 1 --2- 172.21.15.161:0/1386019905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc838100700 0x7fc83810a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:58.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.151+0000 7fc83fcda700 1 --2- 172.21.15.161:0/1386019905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8381032c0 0x7fc8381036e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:58.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.151+0000 7fc83fcda700 1 --2- 172.21.15.161:0/1386019905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8380ffd40 0x7fc8381001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:58.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.151+0000 7fc83fcda700 1 -- 172.21.15.161:0/1386019905 >> 172.21.15.161:0/1386019905 conn(0x7fc8380fb8c0 msgr2=0x7fc8380fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:58.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.151+0000 7fc83fcda700 1 -- 172.21.15.161:0/1386019905 shutdown_connections 2024-03-11T19:54:58.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.152+0000 7fc83fcda700 1 -- 172.21.15.161:0/1386019905 wait complete. 2024-03-11T19:54:58.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.152+0000 7fc83fcda700 1 Processor -- start 2024-03-11T19:54:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.152+0000 7fc83fcda700 1 -- start start 2024-03-11T19:54:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.152+0000 7fc83fcda700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8380ffd40 0x7fc838197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.152+0000 7fc83fcda700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc838100700 0x7fc8381982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.152+0000 7fc83fcda700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8381032c0 0x7fc83819f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.152+0000 7fc83fcda700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83819f8f0 con 0x7fc8381032c0 2024-03-11T19:54:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.152+0000 7fc83fcda700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83819fa30 con 0x7fc838100700 2024-03-11T19:54:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.152+0000 7fc83fcda700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83819fbc0 con 0x7fc8380ffd40 2024-03-11T19:54:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.152+0000 7fc83da76700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8380ffd40 0x7fc838197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.152+0000 7fc83da76700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8380ffd40 0x7fc838197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37280/0 (socket says 172.21.15.161:37280) 2024-03-11T19:54:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.152+0000 7fc83da76700 1 -- 172.21.15.161:0/4047541779 learned_addr learned my addr 172.21.15.161:0/4047541779 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-11T19:54:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.153+0000 7fc83d275700 1 --2- 172.21.15.161:0/4047541779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc838100700 0x7fc8381982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.153+0000 7fc83da76700 1 -- 172.21.15.161:0/4047541779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc838100700 msgr2=0x7fc8381982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:58.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.153+0000 7fc83da76700 1 --2- 172.21.15.161:0/4047541779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc838100700 0x7fc8381982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.153+0000 7fc83da76700 1 -- 172.21.15.161:0/4047541779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8381032c0 msgr2=0x7fc83819f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.153+0000 7fc83da76700 1 --2- 172.21.15.161:0/4047541779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8381032c0 0x7fc83819f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.153+0000 7fc83da76700 1 -- 172.21.15.161:0/4047541779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8340095f0 con 0x7fc8380ffd40 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.153+0000 7fc83da76700 1 --2- 172.21.15.161:0/4047541779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8380ffd40 0x7fc838197d60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc83400f670 tx=0x7fc83400f6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.153+0000 7fc82effd700 1 -- 172.21.15.161:0/4047541779 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83401d070 con 0x7fc8380ffd40 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.153+0000 7fc82effd700 1 -- 172.21.15.161:0/4047541779 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc83400fb80 con 0x7fc8380ffd40 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.153+0000 7fc83fcda700 1 -- 172.21.15.161:0/4047541779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83819fe40 con 0x7fc8380ffd40 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.153+0000 7fc82effd700 1 -- 172.21.15.161:0/4047541779 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc834017650 con 0x7fc8380ffd40 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.153+0000 7fc83fcda700 1 -- 172.21.15.161:0/4047541779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8381a0220 con 0x7fc8380ffd40 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.154+0000 7fc83fcda700 1 -- 172.21.15.161:0/4047541779 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc838109240 con 0x7fc8380ffd40 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.155+0000 7fc82effd700 1 -- 172.21.15.161:0/4047541779 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc83400fcf0 con 0x7fc8380ffd40 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.156+0000 7fc82effd700 1 --2- 172.21.15.161:0/4047541779 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc8240a1ac0 0x7fc8240a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.156+0000 7fc82effd700 1 -- 172.21.15.161:0/4047541779 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc8340ceb90 con 0x7fc8380ffd40 2024-03-11T19:54:58.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.156+0000 7fc83d275700 1 --2- 172.21.15.161:0/4047541779 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc8240a1ac0 0x7fc8240a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:58.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.157+0000 7fc83d275700 1 --2- 172.21.15.161:0/4047541779 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc8240a1ac0 0x7fc8240a3f80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc83806a360 tx=0x7fc828009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:58.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.159+0000 7fc82effd700 1 -- 172.21.15.161:0/4047541779 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc8340982d0 con 0x7fc8380ffd40 2024-03-11T19:54:58.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.365+0000 7fc83fcda700 1 -- 172.21.15.161:0/4047541779 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi114=b;smithi114=e;smithi114=h;smithi114=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fc83802dab0 con 0x7fc8240a1ac0 2024-03-11T19:54:58.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.382+0000 7fc82effd700 1 -- 172.21.15.161:0/4047541779 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fc83802dab0 con 0x7fc8240a1ac0 2024-03-11T19:54:58.385 INFO:teuthology.orchestra.run.smithi161.stdout:Scheduled mds.all update... 2024-03-11T19:54:58.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.383+0000 7fc83fcda700 1 -- 172.21.15.161:0/4047541779 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc8240a1ac0 msgr2=0x7fc8240a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:58.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.383+0000 7fc83fcda700 1 --2- 172.21.15.161:0/4047541779 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc8240a1ac0 0x7fc8240a3f80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc83806a360 tx=0x7fc828009190 comp rx=0 tx=0).stop 2024-03-11T19:54:58.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.383+0000 7fc83fcda700 1 -- 172.21.15.161:0/4047541779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8380ffd40 msgr2=0x7fc838197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:58.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.383+0000 7fc83fcda700 1 --2- 172.21.15.161:0/4047541779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8380ffd40 0x7fc838197d60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc83400f670 tx=0x7fc83400f6a0 comp rx=0 tx=0).stop 2024-03-11T19:54:58.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.383+0000 7fc83fcda700 1 -- 172.21.15.161:0/4047541779 shutdown_connections 2024-03-11T19:54:58.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.383+0000 7fc83fcda700 1 --2- 172.21.15.161:0/4047541779 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc8240a1ac0 0x7fc8240a3f80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:58.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.383+0000 7fc83fcda700 1 --2- 172.21.15.161:0/4047541779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc8381032c0 0x7fc83819f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:58.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.383+0000 7fc83fcda700 1 --2- 172.21.15.161:0/4047541779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc8380ffd40 0x7fc838197d60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:58.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.383+0000 7fc83fcda700 1 --2- 172.21.15.161:0/4047541779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc838100700 0x7fc8381982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:58.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.383+0000 7fc83fcda700 1 -- 172.21.15.161:0/4047541779 >> 172.21.15.161:0/4047541779 conn(0x7fc8380fb8c0 msgr2=0x7fc838106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:58.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.383+0000 7fc83fcda700 1 -- 172.21.15.161:0/4047541779 shutdown_connections 2024-03-11T19:54:58.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-11T19:54:58.383+0000 7fc83fcda700 1 -- 172.21.15.161:0/4047541779 wait complete. 2024-03-11T19:54:59.034 DEBUG:teuthology.orchestra.run.smithi097:mds.a> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.a.service 2024-03-11T19:54:59.042 DEBUG:teuthology.orchestra.run.smithi097:mds.d> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.d.service 2024-03-11T19:54:59.087 DEBUG:teuthology.orchestra.run.smithi097:mds.g> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.g.service 2024-03-11T19:54:59.134 DEBUG:teuthology.orchestra.run.smithi097:mds.j> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.j.service 2024-03-11T19:54:59.179 DEBUG:teuthology.orchestra.run.smithi114:mds.b> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.b.service 2024-03-11T19:54:59.181 DEBUG:teuthology.orchestra.run.smithi114:mds.e> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.e.service 2024-03-11T19:54:59.224 DEBUG:teuthology.orchestra.run.smithi114:mds.h> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.h.service 2024-03-11T19:54:59.266 DEBUG:teuthology.orchestra.run.smithi114:mds.k> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.k.service 2024-03-11T19:54:59.310 DEBUG:teuthology.orchestra.run.smithi161:mds.c> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.c.service 2024-03-11T19:54:59.315 DEBUG:teuthology.orchestra.run.smithi161:mds.f> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.f.service 2024-03-11T19:54:59.358 DEBUG:teuthology.orchestra.run.smithi161:mds.i> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.i.service 2024-03-11T19:54:59.401 DEBUG:teuthology.orchestra.run.smithi161:mds.l> sudo journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.l.service 2024-03-11T19:54:59.444 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-11T19:54:59.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:54:59.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:59 smithi114 ceph-mon[36376]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi114=b;smithi114=e;smithi114=h;smithi114=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:54:59.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:59 smithi114 ceph-mon[36376]: Saving service mds.all spec with placement smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi114=b;smithi114=e;smithi114=h;smithi114=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l;count:12 2024-03-11T19:54:59.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:59 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:59.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:59 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:59.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:54:59 smithi114 ceph-mon[36376]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:59.653 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:59 smithi161 ceph-mon[36332]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi114=b;smithi114=e;smithi114=h;smithi114=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:54:59.653 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:59 smithi161 ceph-mon[36332]: Saving service mds.all spec with placement smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi114=b;smithi114=e;smithi114=h;smithi114=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l;count:12 2024-03-11T19:54:59.653 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:59 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:59.653 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:59 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:59.654 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:54:59 smithi161 ceph-mon[36332]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:59.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:59 smithi097 ceph-mon[31898]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi114=b;smithi114=e;smithi114=h;smithi114=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:54:59.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:59 smithi097 ceph-mon[31898]: Saving service mds.all spec with placement smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi114=b;smithi114=e;smithi114=h;smithi114=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l;count:12 2024-03-11T19:54:59.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:59 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:54:59.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:59 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:54:59.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:54:59 smithi097 ceph-mon[31898]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:54:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.676+0000 7ff46ca00640 1 -- 172.21.15.97:0/1307983175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4681033c0 msgr2=0x7ff4681037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.676+0000 7ff46ca00640 1 --2- 172.21.15.97:0/1307983175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4681033c0 0x7ff4681037e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff458009900 tx=0x7ff45802bc30 comp rx=0 tx=0).stop 2024-03-11T19:54:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.676+0000 7ff46ca00640 1 -- 172.21.15.97:0/1307983175 shutdown_connections 2024-03-11T19:54:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.676+0000 7ff46ca00640 1 --2- 172.21.15.97:0/1307983175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff468104f70 0x7ff46810b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.676+0000 7ff46ca00640 1 --2- 172.21.15.97:0/1307983175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4681045b0 0x7ff468104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.676+0000 7ff46ca00640 1 --2- 172.21.15.97:0/1307983175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4681033c0 0x7ff4681037e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.676+0000 7ff46ca00640 1 -- 172.21.15.97:0/1307983175 >> 172.21.15.97:0/1307983175 conn(0x7ff4680fe970 msgr2=0x7ff468100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.676+0000 7ff46ca00640 1 -- 172.21.15.97:0/1307983175 shutdown_connections 2024-03-11T19:54:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.676+0000 7ff46ca00640 1 -- 172.21.15.97:0/1307983175 wait complete. 2024-03-11T19:54:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.677+0000 7ff46ca00640 1 Processor -- start 2024-03-11T19:54:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.677+0000 7ff46ca00640 1 -- start start 2024-03-11T19:54:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff46ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4681033c0 0x7ff468199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff46ca00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4681045b0 0x7ff4681999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff46ca00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff468104f70 0x7ff4681a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff46ca00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4681a0f70 con 0x7ff4681045b0 2024-03-11T19:54:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff46ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4681a10e0 con 0x7ff4681033c0 2024-03-11T19:54:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff46ca00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4681a12a0 con 0x7ff468104f70 2024-03-11T19:54:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff465400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4681045b0 0x7ff4681999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff466800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff468104f70 0x7ff4681a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff466800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff468104f70 0x7ff4681a0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:59626/0 (socket says 172.21.15.97:59626) 2024-03-11T19:54:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff466800640 1 -- 172.21.15.97:0/2227966394 learned_addr learned my addr 172.21.15.97:0/2227966394 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff465400640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4681045b0 0x7ff4681999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41148/0 (socket says 172.21.15.97:41148) 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff465e00640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4681033c0 0x7ff468199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff466800640 1 -- 172.21.15.97:0/2227966394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4681033c0 msgr2=0x7ff468199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff466800640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4681033c0 0x7ff468199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff466800640 1 -- 172.21.15.97:0/2227966394 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4681045b0 msgr2=0x7ff4681999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff466800640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4681045b0 0x7ff4681999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.678+0000 7ff466800640 1 -- 172.21.15.97:0/2227966394 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff460009490 con 0x7ff468104f70 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.679+0000 7ff465400640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4681045b0 0x7ff4681999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.679+0000 7ff466800640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff468104f70 0x7ff4681a0a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff46000e9b0 tx=0x7ff46000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.679+0000 7ff45ea00640 1 -- 172.21.15.97:0/2227966394 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46000cd80 con 0x7ff468104f70 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.679+0000 7ff45ea00640 1 -- 172.21.15.97:0/2227966394 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4600035f0 con 0x7ff468104f70 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.679+0000 7ff46ca00640 1 -- 172.21.15.97:0/2227966394 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff458009560 con 0x7ff468104f70 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.679+0000 7ff45ea00640 1 -- 172.21.15.97:0/2227966394 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46000cee0 con 0x7ff468104f70 2024-03-11T19:54:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.679+0000 7ff46ca00640 1 -- 172.21.15.97:0/2227966394 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4681a18b0 con 0x7ff468104f70 2024-03-11T19:54:59.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.680+0000 7ff46ca00640 1 -- 172.21.15.97:0/2227966394 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46810a760 con 0x7ff468104f70 2024-03-11T19:54:59.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.682+0000 7ff45ea00640 1 -- 172.21.15.97:0/2227966394 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff460003ad0 con 0x7ff468104f70 2024-03-11T19:54:59.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.684+0000 7ff45ea00640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff4480a1a30 0x7ff4480a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:54:59.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.684+0000 7ff45ea00640 1 -- 172.21.15.97:0/2227966394 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff460014070 con 0x7ff468104f70 2024-03-11T19:54:59.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.684+0000 7ff465e00640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff4480a1a30 0x7ff4480a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:54:59.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.685+0000 7ff465e00640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff4480a1a30 0x7ff4480a3ef0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff458005ab0 tx=0x7ff458005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:54:59.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.692+0000 7ff45ea00640 1 -- 172.21.15.97:0/2227966394 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff460096a70 con 0x7ff468104f70 2024-03-11T19:54:59.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.890+0000 7ff46ca00640 1 -- 172.21.15.97:0/2227966394 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff46810eda0 con 0x7ff468104f70 2024-03-11T19:54:59.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.892+0000 7ff45ea00640 1 -- 172.21.15.97:0/2227966394 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18480 (secure 0 0 0) 0x7ff46009a130 con 0x7ff468104f70 2024-03-11T19:54:59.897 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:54:59.897 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":81,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:54:44.381499+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:54:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.895+0000 7ff46ca00640 1 -- 172.21.15.97:0/2227966394 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff4480a1a30 msgr2=0x7ff4480a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.895+0000 7ff46ca00640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff4480a1a30 0x7ff4480a3ef0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff458005ab0 tx=0x7ff458005fd0 comp rx=0 tx=0).stop 2024-03-11T19:54:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.895+0000 7ff46ca00640 1 -- 172.21.15.97:0/2227966394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff468104f70 msgr2=0x7ff4681a0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:54:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.895+0000 7ff46ca00640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff468104f70 0x7ff4681a0a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff46000e9b0 tx=0x7ff46000ee70 comp rx=0 tx=0).stop 2024-03-11T19:54:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.895+0000 7ff46ca00640 1 -- 172.21.15.97:0/2227966394 shutdown_connections 2024-03-11T19:54:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.895+0000 7ff46ca00640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff468104f70 0x7ff4681a0a30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.895+0000 7ff46ca00640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff4480a1a30 0x7ff4480a3ef0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:59.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.895+0000 7ff46ca00640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4681045b0 0x7ff4681999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.895+0000 7ff46ca00640 1 --2- 172.21.15.97:0/2227966394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4681033c0 0x7ff468199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:54:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.895+0000 7ff46ca00640 1 -- 172.21.15.97:0/2227966394 >> 172.21.15.97:0/2227966394 conn(0x7ff4680fe970 msgr2=0x7ff468107930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:54:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.896+0000 7ff46ca00640 1 -- 172.21.15.97:0/2227966394 shutdown_connections 2024-03-11T19:54:59.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:54:59.896+0000 7ff46ca00640 1 -- 172.21.15.97:0/2227966394 wait complete. 2024-03-11T19:54:59.920 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:50:29.051985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:54:59.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:55:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.127+0000 7fd5c2000640 1 -- 172.21.15.97:0/4276761072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5bc104490 msgr2=0x7fd5bc104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.127+0000 7fd5c2000640 1 --2- 172.21.15.97:0/4276761072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5bc104490 0x7fd5bc104910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009900 tx=0x7fd5a802bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.127+0000 7fd5c2000640 1 -- 172.21.15.97:0/4276761072 shutdown_connections 2024-03-11T19:55:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.127+0000 7fd5c2000640 1 --2- 172.21.15.97:0/4276761072 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5bc104e50 0x7fd5bc10b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.127+0000 7fd5c2000640 1 --2- 172.21.15.97:0/4276761072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5bc104490 0x7fd5bc104910 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.127+0000 7fd5c2000640 1 --2- 172.21.15.97:0/4276761072 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5bc1032a0 0x7fd5bc1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.127+0000 7fd5c2000640 1 -- 172.21.15.97:0/4276761072 >> 172.21.15.97:0/4276761072 conn(0x7fd5bc0fe830 msgr2=0x7fd5bc100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:00.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.128+0000 7fd5c2000640 1 -- 172.21.15.97:0/4276761072 shutdown_connections 2024-03-11T19:55:00.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.128+0000 7fd5c2000640 1 -- 172.21.15.97:0/4276761072 wait complete. 2024-03-11T19:55:00.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.129+0000 7fd5c2000640 1 Processor -- start 2024-03-11T19:55:00.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.129+0000 7fd5c2000640 1 -- start start 2024-03-11T19:55:00.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.129+0000 7fd5c2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5bc1032a0 0x7fd5bc199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.129+0000 7fd5c2000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5bc104490 0x7fd5bc199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.129+0000 7fd5c2000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5bc104e50 0x7fd5bc1a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.129+0000 7fd5c2000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc1a0e00 con 0x7fd5bc104e50 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.130+0000 7fd5c2000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc1a0f70 con 0x7fd5bc1032a0 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.130+0000 7fd5c2000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc1a1130 con 0x7fd5bc104490 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.130+0000 7fd5c0c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5bc1032a0 0x7fd5bc199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.130+0000 7fd5bbe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5bc104490 0x7fd5bc199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.130+0000 7fd5c0c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5bc1032a0 0x7fd5bc199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:57296/0 (socket says 172.21.15.97:57296) 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.130+0000 7fd5c0c00640 1 -- 172.21.15.97:0/3730854286 learned_addr learned my addr 172.21.15.97:0/3730854286 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.130+0000 7fd5c1600640 1 --2- 172.21.15.97:0/3730854286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5bc104e50 0x7fd5bc1a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.130+0000 7fd5bbe00640 1 -- 172.21.15.97:0/3730854286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5bc1032a0 msgr2=0x7fd5bc199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.130+0000 7fd5bbe00640 1 --2- 172.21.15.97:0/3730854286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5bc1032a0 0x7fd5bc199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.130+0000 7fd5bbe00640 1 -- 172.21.15.97:0/3730854286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5bc104e50 msgr2=0x7fd5bc1a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:00.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.130+0000 7fd5bbe00640 1 --2- 172.21.15.97:0/3730854286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5bc104e50 0x7fd5bc1a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.130+0000 7fd5bbe00640 1 -- 172.21.15.97:0/3730854286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a8009560 con 0x7fd5bc104490 2024-03-11T19:55:00.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.130+0000 7fd5bbe00640 1 --2- 172.21.15.97:0/3730854286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5bc104490 0x7fd5bc199850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8000c00 tx=0x7fd5a8005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:00.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.131+0000 7fd5b9600640 1 -- 172.21.15.97:0/3730854286 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a80043d0 con 0x7fd5bc104490 2024-03-11T19:55:00.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.131+0000 7fd5c2000640 1 -- 172.21.15.97:0/3730854286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5bc1a13b0 con 0x7fd5bc104490 2024-03-11T19:55:00.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.131+0000 7fd5b9600640 1 -- 172.21.15.97:0/3730854286 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5a80052d0 con 0x7fd5bc104490 2024-03-11T19:55:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.131+0000 7fd5b9600640 1 -- 172.21.15.97:0/3730854286 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8003d40 con 0x7fd5bc104490 2024-03-11T19:55:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.131+0000 7fd5c2000640 1 -- 172.21.15.97:0/3730854286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5bc1a18e0 con 0x7fd5bc104490 2024-03-11T19:55:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.132+0000 7fd5c2000640 1 -- 172.21.15.97:0/3730854286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5bc10a5e0 con 0x7fd5bc104490 2024-03-11T19:55:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.134+0000 7fd5b9600640 1 -- 172.21.15.97:0/3730854286 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd5a8005440 con 0x7fd5bc104490 2024-03-11T19:55:00.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.135+0000 7fd5b9600640 1 --2- 172.21.15.97:0/3730854286 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd5a00a1a30 0x7fd5a00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:00.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.135+0000 7fd5b9600640 1 -- 172.21.15.97:0/3730854286 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd5a80ef0d0 con 0x7fd5bc104490 2024-03-11T19:55:00.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.135+0000 7fd5c0c00640 1 --2- 172.21.15.97:0/3730854286 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd5a00a1a30 0x7fd5a00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:00.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.136+0000 7fd5c0c00640 1 --2- 172.21.15.97:0/3730854286 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd5a00a1a30 0x7fd5a00a3ef0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac0058c0 tx=0x7fd5ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:00.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.139+0000 7fd5b9600640 1 -- 172.21.15.97:0/3730854286 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd5a80b86e0 con 0x7fd5bc104490 2024-03-11T19:55:00.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.335+0000 7fd5c2000640 1 -- 172.21.15.97:0/3730854286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd5bc10a050 con 0x7fd5bc104490 2024-03-11T19:55:00.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.336+0000 7fd5b9600640 1 -- 172.21.15.97:0/3730854286 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd5a80bbda0 con 0x7fd5bc104490 2024-03-11T19:55:00.340 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-11T19:55:00.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.339+0000 7fd5c2000640 1 -- 172.21.15.97:0/3730854286 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd5a00a1a30 msgr2=0x7fd5a00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:00.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.339+0000 7fd5c2000640 1 --2- 172.21.15.97:0/3730854286 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd5a00a1a30 0x7fd5a00a3ef0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac0058c0 tx=0x7fd5ac005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:00.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.339+0000 7fd5c2000640 1 -- 172.21.15.97:0/3730854286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5bc104490 msgr2=0x7fd5bc199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:00.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.339+0000 7fd5c2000640 1 --2- 172.21.15.97:0/3730854286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5bc104490 0x7fd5bc199850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8000c00 tx=0x7fd5a8005e50 comp rx=0 tx=0).stop 2024-03-11T19:55:00.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.339+0000 7fd5c2000640 1 -- 172.21.15.97:0/3730854286 shutdown_connections 2024-03-11T19:55:00.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.339+0000 7fd5c2000640 1 --2- 172.21.15.97:0/3730854286 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd5a00a1a30 0x7fd5a00a3ef0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.339+0000 7fd5c2000640 1 --2- 172.21.15.97:0/3730854286 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd5bc104e50 0x7fd5bc1a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.339+0000 7fd5c2000640 1 --2- 172.21.15.97:0/3730854286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5bc104490 0x7fd5bc199850 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.339+0000 7fd5c2000640 1 --2- 172.21.15.97:0/3730854286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5bc1032a0 0x7fd5bc199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.339+0000 7fd5c2000640 1 -- 172.21.15.97:0/3730854286 >> 172.21.15.97:0/3730854286 conn(0x7fd5bc0fe830 msgr2=0x7fd5bc1077b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:00.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.339+0000 7fd5c2000640 1 -- 172.21.15.97:0/3730854286 shutdown_connections 2024-03-11T19:55:00.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.339+0000 7fd5c2000640 1 -- 172.21.15.97:0/3730854286 wait complete. 2024-03-11T19:55:00.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:00.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:00.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:55:00.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:00.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:00.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:00.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:00 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:00.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:00 smithi161 ceph-mon[36332]: Deploying daemon mds.l on smithi161 2024-03-11T19:55:00.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:00 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2227966394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:00.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:00 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3730854286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:55:00.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.568+0000 7fdcf1600640 1 -- 172.21.15.97:0/914860488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcec076d80 msgr2=0x7fdcec0771e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:00.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.568+0000 7fdcf1600640 1 --2- 172.21.15.97:0/914860488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcec076d80 0x7fdcec0771e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4009ca0 tx=0x7fdcd402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.568+0000 7fdcf1600640 1 -- 172.21.15.97:0/914860488 shutdown_connections 2024-03-11T19:55:00.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.568+0000 7fdcf1600640 1 --2- 172.21.15.97:0/914860488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdcec077720 0x7fdcec077fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.568+0000 7fdcf1600640 1 --2- 172.21.15.97:0/914860488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcec076d80 0x7fdcec0771e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.568+0000 7fdcf1600640 1 --2- 172.21.15.97:0/914860488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcec075b40 0x7fdcec075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.568+0000 7fdcf1600640 1 -- 172.21.15.97:0/914860488 >> 172.21.15.97:0/914860488 conn(0x7fdcec0fe970 msgr2=0x7fdcec100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:00.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.568+0000 7fdcf1600640 1 -- 172.21.15.97:0/914860488 shutdown_connections 2024-03-11T19:55:00.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.568+0000 7fdcf1600640 1 -- 172.21.15.97:0/914860488 wait complete. 2024-03-11T19:55:00.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.569+0000 7fdcf1600640 1 Processor -- start 2024-03-11T19:55:00.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.569+0000 7fdcf1600640 1 -- start start 2024-03-11T19:55:00.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdcf1600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcec075b40 0x7fdcec19d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:00.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdcf1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdcec076d80 0x7fdcec19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:00.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdcf1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcec077720 0x7fdcec1a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:00.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdcf1600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec1a5310 con 0x7fdcec076d80 2024-03-11T19:55:00.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdcf1600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec1a5480 con 0x7fdcec077720 2024-03-11T19:55:00.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdcf1600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec1a5640 con 0x7fdcec075b40 2024-03-11T19:55:00.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdcea000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdcec076d80 0x7fdcec19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:00.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdceaa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcec075b40 0x7fdcec19d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:00.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdceaa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcec075b40 0x7fdcec19d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:59666/0 (socket says 172.21.15.97:59666) 2024-03-11T19:55:00.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdceb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcec077720 0x7fdcec1a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:00.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdceaa00640 1 -- 172.21.15.97:0/617476039 learned_addr learned my addr 172.21.15.97:0/617476039 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:00.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdceaa00640 1 -- 172.21.15.97:0/617476039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcec077720 msgr2=0x7fdcec1a4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:00.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdceaa00640 1 --2- 172.21.15.97:0/617476039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcec077720 0x7fdcec1a4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdceaa00640 1 -- 172.21.15.97:0/617476039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdcec076d80 msgr2=0x7fdcec19dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:00.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.570+0000 7fdceaa00640 1 --2- 172.21.15.97:0/617476039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdcec076d80 0x7fdcec19dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.571+0000 7fdceaa00640 1 -- 172.21.15.97:0/617476039 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce4009490 con 0x7fdcec075b40 2024-03-11T19:55:00.575 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.571+0000 7fdceaa00640 1 --2- 172.21.15.97:0/617476039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcec075b40 0x7fdcec19d820 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdce400ea60 tx=0x7fdce400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:00.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.571+0000 7fdce3400640 1 -- 172.21.15.97:0/617476039 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce400cde0 con 0x7fdcec075b40 2024-03-11T19:55:00.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.571+0000 7fdce3400640 1 -- 172.21.15.97:0/617476039 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdce40035f0 con 0x7fdcec075b40 2024-03-11T19:55:00.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.571+0000 7fdce3400640 1 -- 172.21.15.97:0/617476039 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4010690 con 0x7fdcec075b40 2024-03-11T19:55:00.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.571+0000 7fdcf1600640 1 -- 172.21.15.97:0/617476039 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd4009900 con 0x7fdcec075b40 2024-03-11T19:55:00.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.571+0000 7fdcf1600640 1 -- 172.21.15.97:0/617476039 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcec1a5cc0 con 0x7fdcec075b40 2024-03-11T19:55:00.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.572+0000 7fdcf1600640 1 -- 172.21.15.97:0/617476039 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcec10ff10 con 0x7fdcec075b40 2024-03-11T19:55:00.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.574+0000 7fdce3400640 1 -- 172.21.15.97:0/617476039 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fdce4003af0 con 0x7fdcec075b40 2024-03-11T19:55:00.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.575+0000 7fdce3400640 1 --2- 172.21.15.97:0/617476039 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdcd00a1ab0 0x7fdcd00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:00.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.576+0000 7fdce3400640 1 -- 172.21.15.97:0/617476039 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdce4014070 con 0x7fdcec075b40 2024-03-11T19:55:00.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.576+0000 7fdcea000640 1 --2- 172.21.15.97:0/617476039 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdcd00a1ab0 0x7fdcd00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:00.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.576+0000 7fdcea000640 1 --2- 172.21.15.97:0/617476039 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdcd00a1ab0 0x7fdcd00a3f70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4006010 tx=0x7fdcd40326b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:00.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.579+0000 7fdce3400640 1 -- 172.21.15.97:0/617476039 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdce4096a10 con 0x7fdcec075b40 2024-03-11T19:55:00.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:00.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:55:00.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:00.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:00.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:00.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:00 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:00.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:00 smithi097 ceph-mon[31898]: Deploying daemon mds.l on smithi161 2024-03-11T19:55:00.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:00 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2227966394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:00.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:00 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3730854286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:55:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:55:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:00 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:00 smithi114 ceph-mon[36376]: Deploying daemon mds.l on smithi161 2024-03-11T19:55:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:00 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2227966394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:00 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3730854286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:55:00.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.817+0000 7fdcf1600640 1 -- 172.21.15.97:0/617476039 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcec113180 con 0x7fdcec075b40 2024-03-11T19:55:00.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.818+0000 7fdce3400640 1 -- 172.21.15.97:0/617476039 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7fdce409a0d0 con 0x7fdcec075b40 2024-03-11T19:55:00.822 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:00.822 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-11T19:55:00.822 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 1 2024-03-11T19:55:00.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.820+0000 7fdcf1600640 1 -- 172.21.15.97:0/617476039 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdcd00a1ab0 msgr2=0x7fdcd00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:00.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.820+0000 7fdcf1600640 1 --2- 172.21.15.97:0/617476039 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdcd00a1ab0 0x7fdcd00a3f70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4006010 tx=0x7fdcd40326b0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.820+0000 7fdcf1600640 1 -- 172.21.15.97:0/617476039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcec075b40 msgr2=0x7fdcec19d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:00.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.820+0000 7fdcf1600640 1 --2- 172.21.15.97:0/617476039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcec075b40 0x7fdcec19d820 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdce400ea60 tx=0x7fdce400ef20 comp rx=0 tx=0).stop 2024-03-11T19:55:00.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.820+0000 7fdcf1600640 1 -- 172.21.15.97:0/617476039 shutdown_connections 2024-03-11T19:55:00.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.820+0000 7fdcf1600640 1 --2- 172.21.15.97:0/617476039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcec077720 0x7fdcec1a4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.820+0000 7fdcf1600640 1 --2- 172.21.15.97:0/617476039 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdcd00a1ab0 0x7fdcd00a3f70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.820+0000 7fdcf1600640 1 --2- 172.21.15.97:0/617476039 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdcec076d80 0x7fdcec19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.820+0000 7fdcf1600640 1 --2- 172.21.15.97:0/617476039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcec075b40 0x7fdcec19d820 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:00.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.820+0000 7fdcf1600640 1 -- 172.21.15.97:0/617476039 >> 172.21.15.97:0/617476039 conn(0x7fdcec0fe970 msgr2=0x7fdcec100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:00.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.820+0000 7fdcf1600640 1 -- 172.21.15.97:0/617476039 shutdown_connections 2024-03-11T19:55:00.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:00.821+0000 7fdcf1600640 1 -- 172.21.15.97:0/617476039 wait complete. 2024-03-11T19:55:01.496 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:01 smithi161 ceph-mon[36332]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:01.496 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:01 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/617476039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:01.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:01 smithi097 ceph-mon[31898]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:01.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:01 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/617476039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:01 smithi114 ceph-mon[36376]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:01 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/617476039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:02.789 INFO:journalctl@ceph.mds.l.smithi161.stdout:Mar 11 19:55:02 smithi161 systemd[1]: Started Ceph mds.l for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:55:02.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:03.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.061+0000 7f005e800640 1 -- 172.21.15.97:0/708840315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0058104f70 msgr2=0x7f005810b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:03.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.061+0000 7f005e800640 1 --2- 172.21.15.97:0/708840315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0058104f70 0x7f005810b800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0050009ca0 tx=0x7f005002c100 comp rx=0 tx=0).stop 2024-03-11T19:55:03.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.062+0000 7f005e800640 1 -- 172.21.15.97:0/708840315 shutdown_connections 2024-03-11T19:55:03.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.062+0000 7f005e800640 1 --2- 172.21.15.97:0/708840315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0058104f70 0x7f005810b800 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:03.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.062+0000 7f005e800640 1 --2- 172.21.15.97:0/708840315 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00581045b0 0x7f0058104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:03.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.062+0000 7f005e800640 1 --2- 172.21.15.97:0/708840315 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00581033c0 0x7f00581037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.062+0000 7f005e800640 1 -- 172.21.15.97:0/708840315 >> 172.21.15.97:0/708840315 conn(0x7f00580fe930 msgr2=0x7f0058100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.062+0000 7f005e800640 1 -- 172.21.15.97:0/708840315 shutdown_connections 2024-03-11T19:55:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.062+0000 7f005e800640 1 -- 172.21.15.97:0/708840315 wait complete. 2024-03-11T19:55:03.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.063+0000 7f005e800640 1 Processor -- start 2024-03-11T19:55:03.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.063+0000 7f005e800640 1 -- start start 2024-03-11T19:55:03.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.063+0000 7f005e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00581033c0 0x7f00581993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:03.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.063+0000 7f005e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00581045b0 0x7f0058199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:03.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.063+0000 7f005e800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0058104f70 0x7f00581a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.063+0000 7f005e800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00581a0ee0 con 0x7f00581033c0 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.063+0000 7f005e800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00581a1050 con 0x7f00581045b0 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.063+0000 7f005e800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00581a1210 con 0x7f0058104f70 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f0057400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00581033c0 0x7f00581993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f004ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00581045b0 0x7f0058199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f004ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00581045b0 0x7f0058199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:57330/0 (socket says 172.21.15.97:57330) 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f004ea00640 1 -- 172.21.15.97:0/191540944 learned_addr learned my addr 172.21.15.97:0/191540944 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f0057e00640 1 --2- 172.21.15.97:0/191540944 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0058104f70 0x7f00581a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f004ea00640 1 -- 172.21.15.97:0/191540944 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0058104f70 msgr2=0x7f00581a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f004ea00640 1 --2- 172.21.15.97:0/191540944 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0058104f70 0x7f00581a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f004ea00640 1 -- 172.21.15.97:0/191540944 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00581033c0 msgr2=0x7f00581993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f004ea00640 1 --2- 172.21.15.97:0/191540944 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00581033c0 0x7f00581993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f004ea00640 1 -- 172.21.15.97:0/191540944 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0050009900 con 0x7f00581045b0 2024-03-11T19:55:03.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f004ea00640 1 --2- 172.21.15.97:0/191540944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00581045b0 0x7f0058199930 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f004000e9f0 tx=0x7f004000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:03.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f0054c00640 1 -- 172.21.15.97:0/191540944 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0040009700 con 0x7f00581045b0 2024-03-11T19:55:03.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f0054c00640 1 -- 172.21.15.97:0/191540944 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f004000cd80 con 0x7f00581045b0 2024-03-11T19:55:03.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.064+0000 7f005e800640 1 -- 172.21.15.97:0/191540944 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00581a14f0 con 0x7f00581045b0 2024-03-11T19:55:03.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.065+0000 7f0054c00640 1 -- 172.21.15.97:0/191540944 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0040018920 con 0x7f00581045b0 2024-03-11T19:55:03.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.065+0000 7f005e800640 1 -- 172.21.15.97:0/191540944 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00581a1a50 con 0x7f00581045b0 2024-03-11T19:55:03.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.066+0000 7f005e800640 1 -- 172.21.15.97:0/191540944 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f005810a760 con 0x7f00581045b0 2024-03-11T19:55:03.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.067+0000 7f0054c00640 1 -- 172.21.15.97:0/191540944 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0040018a80 con 0x7f00581045b0 2024-03-11T19:55:03.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.069+0000 7f0054c00640 1 --2- 172.21.15.97:0/191540944 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f003c0a1ab0 0x7f003c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:03.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.069+0000 7f0054c00640 1 -- 172.21.15.97:0/191540944 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0040014070 con 0x7f00581045b0 2024-03-11T19:55:03.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.069+0000 7f0057400640 1 --2- 172.21.15.97:0/191540944 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f003c0a1ab0 0x7f003c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:03.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.070+0000 7f0057400640 1 --2- 172.21.15.97:0/191540944 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f003c0a1ab0 0x7f003c0a3f70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0048005fd0 tx=0x7f004800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:03.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.073+0000 7f0054c00640 1 -- 172.21.15.97:0/191540944 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0040052d20 con 0x7f00581045b0 2024-03-11T19:55:03.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.306+0000 7f005e800640 1 -- 172.21.15.97:0/191540944 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f005802ce40 con 0x7f00581045b0 2024-03-11T19:55:03.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.307+0000 7f0054c00640 1 -- 172.21.15.97:0/191540944 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7f00400969b0 con 0x7f00581045b0 2024-03-11T19:55:03.310 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:03.310 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-11T19:55:03.311 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 2 2024-03-11T19:55:03.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.310+0000 7f005e800640 1 -- 172.21.15.97:0/191540944 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f003c0a1ab0 msgr2=0x7f003c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:03.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.310+0000 7f005e800640 1 --2- 172.21.15.97:0/191540944 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f003c0a1ab0 0x7f003c0a3f70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0048005fd0 tx=0x7f004800b390 comp rx=0 tx=0).stop 2024-03-11T19:55:03.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.310+0000 7f005e800640 1 -- 172.21.15.97:0/191540944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00581045b0 msgr2=0x7f0058199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:03.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.310+0000 7f005e800640 1 --2- 172.21.15.97:0/191540944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00581045b0 0x7f0058199930 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f004000e9f0 tx=0x7f004000eeb0 comp rx=0 tx=0).stop 2024-03-11T19:55:03.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.310+0000 7f005e800640 1 -- 172.21.15.97:0/191540944 shutdown_connections 2024-03-11T19:55:03.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.310+0000 7f005e800640 1 --2- 172.21.15.97:0/191540944 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0058104f70 0x7f00581a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:03.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.310+0000 7f005e800640 1 --2- 172.21.15.97:0/191540944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00581045b0 0x7f0058199930 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:03.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.310+0000 7f005e800640 1 --2- 172.21.15.97:0/191540944 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f003c0a1ab0 0x7f003c0a3f70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:03.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.310+0000 7f005e800640 1 --2- 172.21.15.97:0/191540944 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00581033c0 0x7f00581993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:03.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.310+0000 7f005e800640 1 -- 172.21.15.97:0/191540944 >> 172.21.15.97:0/191540944 conn(0x7f00580fe930 msgr2=0x7f0058107930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:03.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.310+0000 7f005e800640 1 -- 172.21.15.97:0/191540944 shutdown_connections 2024-03-11T19:55:03.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:03.310+0000 7f005e800640 1 -- 172.21.15.97:0/191540944 wait complete. 2024-03-11T19:55:03.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:03 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.161:6832/2213740234,v1:172.21.15.161:6833/2213740234] up:boot 2024-03-11T19:55:03.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:03 smithi114 ceph-mon[36376]: fsmap 1 up:standby 2024-03-11T19:55:03.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:03 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T19:55:03.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:03 smithi114 ceph-mon[36376]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:03.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:03 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:03.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:03 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:03.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:03 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:03.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:03 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:03.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:03 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:03.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:03 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:03.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:03 smithi114 ceph-mon[36376]: Deploying daemon mds.k on smithi114 2024-03-11T19:55:03.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:03 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/191540944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:03.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:03 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.161:6832/2213740234,v1:172.21.15.161:6833/2213740234] up:boot 2024-03-11T19:55:03.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:03 smithi097 ceph-mon[31898]: fsmap 1 up:standby 2024-03-11T19:55:03.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T19:55:03.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:03 smithi097 ceph-mon[31898]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:03.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:03.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:03.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:03.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:03.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:03.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:03 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:03.672 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:03 smithi097 ceph-mon[31898]: Deploying daemon mds.k on smithi114 2024-03-11T19:55:03.673 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:03 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/191540944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:03.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:03 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.161:6832/2213740234,v1:172.21.15.161:6833/2213740234] up:boot 2024-03-11T19:55:03.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:03 smithi161 ceph-mon[36332]: fsmap 1 up:standby 2024-03-11T19:55:03.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:03 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T19:55:03.789 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:03 smithi161 ceph-mon[36332]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:03.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:03 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:03.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:03 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:03.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:03 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:03.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:03 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:03.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:03 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:03.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:03 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:03.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:03 smithi161 ceph-mon[36332]: Deploying daemon mds.k on smithi114 2024-03-11T19:55:03.790 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:03 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/191540944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:04.993 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 19:55:04 smithi114 systemd[1]: Starting Ceph mds.k for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T19:55:05.328 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 19:55:05 smithi114 podman[65721]: 2024-03-11 19:55:05.063452849 +0000 UTC m=+0.112588198 container create 80803eebc596ba60064a0fa12ffff3713a88a20654aea32ce6d8c1d30f511266 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T19:55:05.328 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 19:55:05 smithi114 podman[65721]: 2024-03-11 19:55:04.968372535 +0000 UTC m=+0.017507874 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:55:05.328 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 19:55:05 smithi114 podman[65721]: 2024-03-11 19:55:05.299505902 +0000 UTC m=+0.348641232 container init 80803eebc596ba60064a0fa12ffff3713a88a20654aea32ce6d8c1d30f511266 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-k, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T19:55:05.328 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 19:55:05 smithi114 podman[65721]: 2024-03-11 19:55:05.302166804 +0000 UTC m=+0.351302143 container start 80803eebc596ba60064a0fa12ffff3713a88a20654aea32ce6d8c1d30f511266 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T19:55:05.334 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:05.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.555+0000 7f164e600640 1 -- 172.21.15.97:0/1287132518 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16481033c0 msgr2=0x7f16481037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:05.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.555+0000 7f164e600640 1 --2- 172.21.15.97:0/1287132518 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16481033c0 0x7f16481037e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1638009900 tx=0x7f163802bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:05.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.555+0000 7f164e600640 1 -- 172.21.15.97:0/1287132518 shutdown_connections 2024-03-11T19:55:05.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.555+0000 7f164e600640 1 --2- 172.21.15.97:0/1287132518 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1648104f70 0x7f164810b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:05.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.555+0000 7f164e600640 1 --2- 172.21.15.97:0/1287132518 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16481045b0 0x7f1648104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:05.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.555+0000 7f164e600640 1 --2- 172.21.15.97:0/1287132518 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16481033c0 0x7f16481037e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:05.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.555+0000 7f164e600640 1 -- 172.21.15.97:0/1287132518 >> 172.21.15.97:0/1287132518 conn(0x7f16480fe970 msgr2=0x7f1648100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:05.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.555+0000 7f164e600640 1 -- 172.21.15.97:0/1287132518 shutdown_connections 2024-03-11T19:55:05.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.555+0000 7f164e600640 1 -- 172.21.15.97:0/1287132518 wait complete. 2024-03-11T19:55:05.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.556+0000 7f164e600640 1 Processor -- start 2024-03-11T19:55:05.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.556+0000 7f164e600640 1 -- start start 2024-03-11T19:55:05.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f164e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16481033c0 0x7f16481994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:05.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f164e600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16481045b0 0x7f1648199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:05.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f164e600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1648104f70 0x7f16481a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:05.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f164e600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16481a0fb0 con 0x7f16481045b0 2024-03-11T19:55:05.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f164e600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16481a1120 con 0x7f16481033c0 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f164e600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16481a12e0 con 0x7f1648104f70 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f1647400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16481033c0 0x7f16481994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f1647400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16481033c0 0x7f16481994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:40876/0 (socket says 172.21.15.97:40876) 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f1647400640 1 -- 172.21.15.97:0/3038777810 learned_addr learned my addr 172.21.15.97:0/3038777810 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f1646a00640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16481045b0 0x7f1648199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f1647e00640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1648104f70 0x7f16481a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f1647400640 1 -- 172.21.15.97:0/3038777810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1648104f70 msgr2=0x7f16481a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f1647400640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1648104f70 0x7f16481a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f1647400640 1 -- 172.21.15.97:0/3038777810 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16481045b0 msgr2=0x7f1648199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f1647400640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16481045b0 0x7f1648199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.557+0000 7f1647400640 1 -- 172.21.15.97:0/3038777810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1640009840 con 0x7f16481033c0 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.558+0000 7f1647400640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16481033c0 0x7f16481994c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1638000c00 tx=0x7f1638005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.558+0000 7f163fe00640 1 -- 172.21.15.97:0/3038777810 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16380043d0 con 0x7f16481033c0 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.558+0000 7f163fe00640 1 -- 172.21.15.97:0/3038777810 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16380052d0 con 0x7f16481033c0 2024-03-11T19:55:05.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.558+0000 7f163fe00640 1 -- 172.21.15.97:0/3038777810 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638003c30 con 0x7f16481033c0 2024-03-11T19:55:05.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.558+0000 7f164e600640 1 -- 172.21.15.97:0/3038777810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1638009560 con 0x7f16481033c0 2024-03-11T19:55:05.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.558+0000 7f164e600640 1 -- 172.21.15.97:0/3038777810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16481a1930 con 0x7f16481033c0 2024-03-11T19:55:05.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.559+0000 7f164e600640 1 -- 172.21.15.97:0/3038777810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164810a6e0 con 0x7f16481033c0 2024-03-11T19:55:05.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.560+0000 7f163fe00640 1 -- 172.21.15.97:0/3038777810 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1638005440 con 0x7f16481033c0 2024-03-11T19:55:05.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.562+0000 7f163fe00640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f162c0a1ab0 0x7f162c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:05.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.562+0000 7f163fe00640 1 -- 172.21.15.97:0/3038777810 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f16380ef770 con 0x7f16481033c0 2024-03-11T19:55:05.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.562+0000 7f1646a00640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f162c0a1ab0 0x7f162c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:05.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.563+0000 7f1646a00640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f162c0a1ab0 0x7f162c0a3f70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f164819a9d0 tx=0x7f1634005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:05.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.566+0000 7f163fe00640 1 -- 172.21.15.97:0/3038777810 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f16380b8ea0 con 0x7f16481033c0 2024-03-11T19:55:05.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:05 smithi114 ceph-mon[36376]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:05.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:05.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:05.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:05.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:05.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:05.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:05 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:05.743 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 19:55:05 smithi114 ceph-mds[65740]: -- 172.21.15.114:0/1233132275 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5561846561a0 con 0x55618462cc00 2024-03-11T19:55:05.743 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 19:55:05 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-k[65736]: starting mds.k at 2024-03-11T19:55:05.743 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 19:55:05 smithi114 bash[65721]: 80803eebc596ba60064a0fa12ffff3713a88a20654aea32ce6d8c1d30f511266 2024-03-11T19:55:05.744 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 19:55:05 smithi114 systemd[1]: Started Ceph mds.k for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:55:05.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.804+0000 7f164e600640 1 -- 172.21.15.97:0/3038777810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f164810f0e0 con 0x7f16481033c0 2024-03-11T19:55:05.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.805+0000 7f163fe00640 1 -- 172.21.15.97:0/3038777810 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7f1638043860 con 0x7f16481033c0 2024-03-11T19:55:05.808 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:05.809 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-11T19:55:05.809 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 3 2024-03-11T19:55:05.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.808+0000 7f164e600640 1 -- 172.21.15.97:0/3038777810 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f162c0a1ab0 msgr2=0x7f162c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:05.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.808+0000 7f164e600640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f162c0a1ab0 0x7f162c0a3f70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f164819a9d0 tx=0x7f1634005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:05.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.808+0000 7f164e600640 1 -- 172.21.15.97:0/3038777810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16481033c0 msgr2=0x7f16481994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:05.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.808+0000 7f164e600640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16481033c0 0x7f16481994c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1638000c00 tx=0x7f1638005e80 comp rx=0 tx=0).stop 2024-03-11T19:55:05.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.808+0000 7f164e600640 1 -- 172.21.15.97:0/3038777810 shutdown_connections 2024-03-11T19:55:05.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.808+0000 7f164e600640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1648104f70 0x7f16481a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:05.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.808+0000 7f164e600640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f162c0a1ab0 0x7f162c0a3f70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:05.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.808+0000 7f164e600640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f16481045b0 0x7f1648199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:05.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.808+0000 7f164e600640 1 --2- 172.21.15.97:0/3038777810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16481033c0 0x7f16481994c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:05.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.808+0000 7f164e600640 1 -- 172.21.15.97:0/3038777810 >> 172.21.15.97:0/3038777810 conn(0x7f16480fe970 msgr2=0x7f16481078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:05.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.808+0000 7f164e600640 1 -- 172.21.15.97:0/3038777810 shutdown_connections 2024-03-11T19:55:05.812 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:05.808+0000 7f164e600640 1 -- 172.21.15.97:0/3038777810 wait complete. 2024-03-11T19:55:05.827 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:05 smithi097 ceph-mon[31898]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:05.827 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:05.827 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:05.827 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:05.828 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:05.828 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:05.828 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:05 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:06.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:05 smithi161 ceph-mon[36332]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:06.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:06.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:06.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:06.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:06.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:06.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:05 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:06.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:06 smithi097 ceph-mon[31898]: Deploying daemon mds.j on smithi097 2024-03-11T19:55:06.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:06 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] up:boot 2024-03-11T19:55:06.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:06 smithi097 ceph-mon[31898]: fsmap 2 up:standby 2024-03-11T19:55:06.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:06 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T19:55:06.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:06 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3038777810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:06.720 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:06 smithi114 ceph-mon[36376]: Deploying daemon mds.j on smithi097 2024-03-11T19:55:06.720 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:06 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] up:boot 2024-03-11T19:55:06.720 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:06 smithi114 ceph-mon[36376]: fsmap 2 up:standby 2024-03-11T19:55:06.720 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:06 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T19:55:06.720 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:06 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3038777810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:07.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:06 smithi161 ceph-mon[36332]: Deploying daemon mds.j on smithi097 2024-03-11T19:55:07.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:06 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] up:boot 2024-03-11T19:55:07.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:06 smithi161 ceph-mon[36332]: fsmap 2 up:standby 2024-03-11T19:55:07.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:06 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T19:55:07.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:06 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3038777810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:07.670 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:07 smithi097 ceph-mon[31898]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:07.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:07.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:07 smithi114 ceph-mon[36376]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:08.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:07 smithi161 ceph-mon[36332]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:08.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.256+0000 7fb6bc400640 1 -- 172.21.15.97:0/1662810868 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6b4106e80 msgr2=0x7fb6b41092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:08.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.256+0000 7fb6bc400640 1 --2- 172.21.15.97:0/1662810868 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6b4106e80 0x7fb6b41092d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac009ca0 tx=0x7fb6ac02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:55:08.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.257+0000 7fb6bc400640 1 -- 172.21.15.97:0/1662810868 shutdown_connections 2024-03-11T19:55:08.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.257+0000 7fb6bc400640 1 --2- 172.21.15.97:0/1662810868 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6b4106e80 0x7fb6b41092d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:08.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.257+0000 7fb6bc400640 1 --2- 172.21.15.97:0/1662810868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6b4104550 0x7fb6b4106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:08.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.257+0000 7fb6bc400640 1 --2- 172.21.15.97:0/1662810868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6b4101c20 0x7fb6b4104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:08.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.258+0000 7fb6bc400640 1 -- 172.21.15.97:0/1662810868 >> 172.21.15.97:0/1662810868 conn(0x7fb6b40fb610 msgr2=0x7fb6b40fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:08.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.258+0000 7fb6bc400640 1 -- 172.21.15.97:0/1662810868 shutdown_connections 2024-03-11T19:55:08.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.258+0000 7fb6bc400640 1 -- 172.21.15.97:0/1662810868 wait complete. 2024-03-11T19:55:08.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.259+0000 7fb6bc400640 1 Processor -- start 2024-03-11T19:55:08.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.259+0000 7fb6bc400640 1 -- start start 2024-03-11T19:55:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.260+0000 7fb6bc400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6b4101c20 0x7fb6b4197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.260+0000 7fb6bc400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6b4104550 0x7fb6b4197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.260+0000 7fb6bc400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6b4106e80 0x7fb6b419e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.260+0000 7fb6bc400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b419ed20 con 0x7fb6b4106e80 2024-03-11T19:55:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.260+0000 7fb6bc400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b419ee90 con 0x7fb6b4104550 2024-03-11T19:55:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.260+0000 7fb6bc400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b419f050 con 0x7fb6b4101c20 2024-03-11T19:55:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.260+0000 7fb6ba400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6b4106e80 0x7fb6b419e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.260+0000 7fb6ba400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6b4106e80 0x7fb6b419e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37322/0 (socket says 172.21.15.97:37322) 2024-03-11T19:55:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.260+0000 7fb6ba400640 1 -- 172.21.15.97:0/3547407597 learned_addr learned my addr 172.21.15.97:0/3547407597 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.260+0000 7fb6b9a00640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6b4101c20 0x7fb6b4197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:08.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.260+0000 7fb6b9000640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6b4104550 0x7fb6b4197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:08.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.261+0000 7fb6ba400640 1 -- 172.21.15.97:0/3547407597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6b4101c20 msgr2=0x7fb6b4197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:08.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.261+0000 7fb6ba400640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6b4101c20 0x7fb6b4197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:08.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.261+0000 7fb6ba400640 1 -- 172.21.15.97:0/3547407597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6b4104550 msgr2=0x7fb6b4197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:08.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.261+0000 7fb6ba400640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6b4104550 0x7fb6b4197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:08.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.261+0000 7fb6ba400640 1 -- 172.21.15.97:0/3547407597 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a8009490 con 0x7fb6b4106e80 2024-03-11T19:55:08.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.261+0000 7fb6b9000640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6b4104550 0x7fb6b4197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:08.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.261+0000 7fb6b9a00640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6b4101c20 0x7fb6b4197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:08.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.261+0000 7fb6ba400640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6b4106e80 0x7fb6b419e7e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac009bf0 tx=0x7fb6ac034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:08.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.261+0000 7fb6b2000640 1 -- 172.21.15.97:0/3547407597 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac0323b0 con 0x7fb6b4106e80 2024-03-11T19:55:08.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.261+0000 7fb6b2000640 1 -- 172.21.15.97:0/3547407597 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6ac003e50 con 0x7fb6b4106e80 2024-03-11T19:55:08.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.261+0000 7fb6bc400640 1 -- 172.21.15.97:0/3547407597 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ac009900 con 0x7fb6b4106e80 2024-03-11T19:55:08.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.261+0000 7fb6b2000640 1 -- 172.21.15.97:0/3547407597 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac040450 con 0x7fb6b4106e80 2024-03-11T19:55:08.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.261+0000 7fb6bc400640 1 -- 172.21.15.97:0/3547407597 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b419f5a0 con 0x7fb6b4106e80 2024-03-11T19:55:08.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.263+0000 7fb6bc400640 1 -- 172.21.15.97:0/3547407597 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b40fd100 con 0x7fb6b4106e80 2024-03-11T19:55:08.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.265+0000 7fb6b2000640 1 -- 172.21.15.97:0/3547407597 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb6ac00a040 con 0x7fb6b4106e80 2024-03-11T19:55:08.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.266+0000 7fb6b2000640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb69c0a1a80 0x7fb69c0a3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:08.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.266+0000 7fb6b2000640 1 -- 172.21.15.97:0/3547407597 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb6ac0ef9c0 con 0x7fb6b4106e80 2024-03-11T19:55:08.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.267+0000 7fb6b9a00640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb69c0a1a80 0x7fb69c0a3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:08.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.268+0000 7fb6b9a00640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb69c0a1a80 0x7fb69c0a3f40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0009fc0 tx=0x7fb6a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:08.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.275+0000 7fb6b2000640 1 -- 172.21.15.97:0/3547407597 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb6ac0b8fd0 con 0x7fb6b4106e80 2024-03-11T19:55:08.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.524+0000 7fb6bc400640 1 -- 172.21.15.97:0/3547407597 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6b402ce40 con 0x7fb6b4106e80 2024-03-11T19:55:08.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.524+0000 7fb6b2000640 1 -- 172.21.15.97:0/3547407597 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7fb6ac0bc690 con 0x7fb6b4106e80 2024-03-11T19:55:08.528 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:08.528 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-11T19:55:08.528 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 3 2024-03-11T19:55:08.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.526+0000 7fb6bc400640 1 -- 172.21.15.97:0/3547407597 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb69c0a1a80 msgr2=0x7fb69c0a3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:08.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.526+0000 7fb6bc400640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb69c0a1a80 0x7fb69c0a3f40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0009fc0 tx=0x7fb6a0005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:08.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.527+0000 7fb6bc400640 1 -- 172.21.15.97:0/3547407597 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6b4106e80 msgr2=0x7fb6b419e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:08.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.527+0000 7fb6bc400640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6b4106e80 0x7fb6b419e7e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac009bf0 tx=0x7fb6ac034cc0 comp rx=0 tx=0).stop 2024-03-11T19:55:08.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.527+0000 7fb6bc400640 1 -- 172.21.15.97:0/3547407597 shutdown_connections 2024-03-11T19:55:08.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.527+0000 7fb6bc400640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb69c0a1a80 0x7fb69c0a3f40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:08.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.527+0000 7fb6bc400640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb6b4106e80 0x7fb6b419e7e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:08.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.527+0000 7fb6bc400640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6b4104550 0x7fb6b4197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:08.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.527+0000 7fb6bc400640 1 --2- 172.21.15.97:0/3547407597 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6b4101c20 0x7fb6b4197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:08.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.527+0000 7fb6bc400640 1 -- 172.21.15.97:0/3547407597 >> 172.21.15.97:0/3547407597 conn(0x7fb6b40fb610 msgr2=0x7fb6b4102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:08.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.527+0000 7fb6bc400640 1 -- 172.21.15.97:0/3547407597 shutdown_connections 2024-03-11T19:55:08.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:08.527+0000 7fb6bc400640 1 -- 172.21.15.97:0/3547407597 wait complete. 2024-03-11T19:55:08.824 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:08 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3547407597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:08.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:08 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3547407597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:09.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:08 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3547407597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:09.160 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 19:55:08 smithi097 podman[72401]: 2024-03-11 19:55:08.860790894 +0000 UTC m=+0.168663236 container create 80123fcb5e22646809d9a0330bcf9322fcc9a03c9401c8ca999bee1f56101614 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-j, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:55:09.161 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 19:55:09 smithi097 podman[72401]: 2024-03-11 19:55:09.156596163 +0000 UTC m=+0.464468489 container init 80123fcb5e22646809d9a0330bcf9322fcc9a03c9401c8ca999bee1f56101614 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-j, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-03-11T19:55:09.421 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 19:55:09 smithi097 podman[72401]: 2024-03-11 19:55:09.16769196 +0000 UTC m=+0.475564251 container start 80123fcb5e22646809d9a0330bcf9322fcc9a03c9401c8ca999bee1f56101614 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.build-date=20240307) 2024-03-11T19:55:09.421 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 19:55:09 smithi097 ceph-mds[72420]: -- 172.21.15.97:0/1680769104 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c0667841a0 con 0x55c06675ac00 2024-03-11T19:55:09.421 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 19:55:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-j[72416]: starting mds.j at 2024-03-11T19:55:09.421 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 19:55:09 smithi097 bash[72401]: 80123fcb5e22646809d9a0330bcf9322fcc9a03c9401c8ca999bee1f56101614 2024-03-11T19:55:09.422 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 19:55:09 smithi097 systemd[1]: Started Ceph mds.j for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:55:09.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:09 smithi097 ceph-mon[31898]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:09.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:09 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:09.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:09 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:09.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:09 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:09.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:09 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:09.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:09 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:09.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:09 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:09.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:09 smithi114 ceph-mon[36376]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:09 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:09 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:09 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:09 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:09 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:09 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:10.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:09 smithi161 ceph-mon[36332]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:10.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:09 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:10.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:09 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:10.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:09 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:10.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:09 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:10.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:09 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:10.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:09 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:10.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:10.633 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:10 smithi097 ceph-mon[31898]: Deploying daemon mds.d on smithi097 2024-03-11T19:55:10.634 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:10 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.97:6834/207363921,v1:172.21.15.97:6835/207363921] up:boot 2024-03-11T19:55:10.634 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:10 smithi097 ceph-mon[31898]: fsmap 3 up:standby 2024-03-11T19:55:10.634 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:10 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T19:55:10.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.770+0000 7fd136c00640 1 -- 172.21.15.97:0/1475384092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1300ffe80 msgr2=0x7fd1301002a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:10.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.770+0000 7fd136c00640 1 --2- 172.21.15.97:0/1475384092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1300ffe80 0x7fd1301002a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd124009900 tx=0x7fd12402bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:10.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.770+0000 7fd136c00640 1 -- 172.21.15.97:0/1475384092 shutdown_connections 2024-03-11T19:55:10.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.770+0000 7fd136c00640 1 --2- 172.21.15.97:0/1475384092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd130101a20 0x7fd130108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:10.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.770+0000 7fd136c00640 1 --2- 172.21.15.97:0/1475384092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd130101060 0x7fd1301014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:10.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.770+0000 7fd136c00640 1 --2- 172.21.15.97:0/1475384092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1300ffe80 0x7fd1301002a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:10.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.770+0000 7fd136c00640 1 -- 172.21.15.97:0/1475384092 >> 172.21.15.97:0/1475384092 conn(0x7fd1300fb460 msgr2=0x7fd1300fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:10.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.770+0000 7fd136c00640 1 -- 172.21.15.97:0/1475384092 shutdown_connections 2024-03-11T19:55:10.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.770+0000 7fd136c00640 1 -- 172.21.15.97:0/1475384092 wait complete. 2024-03-11T19:55:10.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.771+0000 7fd136c00640 1 Processor -- start 2024-03-11T19:55:10.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.771+0000 7fd136c00640 1 -- start start 2024-03-11T19:55:10.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.771+0000 7fd136c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1300ffe80 0x7fd130199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:10.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.771+0000 7fd136c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd130101060 0x7fd1301997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:10.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.771+0000 7fd136c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd130101a20 0x7fd1301a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:10.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.771+0000 7fd136c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1301a0d50 con 0x7fd130101a20 2024-03-11T19:55:10.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.771+0000 7fd136c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1301a0ec0 con 0x7fd130101060 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.771+0000 7fd136c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1301a1080 con 0x7fd1300ffe80 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.772+0000 7fd136200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd130101a20 0x7fd1301a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.772+0000 7fd136200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd130101a20 0x7fd1301a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37360/0 (socket says 172.21.15.97:37360) 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.772+0000 7fd136200640 1 -- 172.21.15.97:0/1050182516 learned_addr learned my addr 172.21.15.97:0/1050182516 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.772+0000 7fd135800640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1300ffe80 0x7fd130199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.772+0000 7fd134e00640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd130101060 0x7fd1301997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.772+0000 7fd136200640 1 -- 172.21.15.97:0/1050182516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1300ffe80 msgr2=0x7fd130199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.772+0000 7fd136200640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1300ffe80 0x7fd130199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.772+0000 7fd136200640 1 -- 172.21.15.97:0/1050182516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd130101060 msgr2=0x7fd1301997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.772+0000 7fd136200640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd130101060 0x7fd1301997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.772+0000 7fd136200640 1 -- 172.21.15.97:0/1050182516 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd124009560 con 0x7fd130101a20 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.772+0000 7fd136200640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd130101a20 0x7fd1301a0810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd12800eb10 tx=0x7fd12800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.773+0000 7fd12e000640 1 -- 172.21.15.97:0/1050182516 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd128004b20 con 0x7fd130101a20 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.773+0000 7fd12e000640 1 -- 172.21.15.97:0/1050182516 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd128004c80 con 0x7fd130101a20 2024-03-11T19:55:10.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.773+0000 7fd12e000640 1 -- 172.21.15.97:0/1050182516 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd128010650 con 0x7fd130101a20 2024-03-11T19:55:10.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.773+0000 7fd136c00640 1 -- 172.21.15.97:0/1050182516 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1301a1360 con 0x7fd130101a20 2024-03-11T19:55:10.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.773+0000 7fd136c00640 1 -- 172.21.15.97:0/1050182516 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1301a1870 con 0x7fd130101a20 2024-03-11T19:55:10.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.774+0000 7fd136c00640 1 -- 172.21.15.97:0/1050182516 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13010b930 con 0x7fd130101a20 2024-03-11T19:55:10.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.776+0000 7fd12e000640 1 -- 172.21.15.97:0/1050182516 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd128005020 con 0x7fd130101a20 2024-03-11T19:55:10.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.777+0000 7fd12e000640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd1180a1ab0 0x7fd1180a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:10.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.777+0000 7fd12e000640 1 -- 172.21.15.97:0/1050182516 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd128014070 con 0x7fd130101a20 2024-03-11T19:55:10.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.778+0000 7fd135800640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd1180a1ab0 0x7fd1180a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:10.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.778+0000 7fd135800640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd1180a1ab0 0x7fd1180a3f70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd12402c240 tx=0x7fd12402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:10.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:10.782+0000 7fd12e000640 1 -- 172.21.15.97:0/1050182516 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd128096b70 con 0x7fd130101a20 2024-03-11T19:55:10.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:10 smithi114 ceph-mon[36376]: Deploying daemon mds.d on smithi097 2024-03-11T19:55:10.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:10 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.97:6834/207363921,v1:172.21.15.97:6835/207363921] up:boot 2024-03-11T19:55:10.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:10 smithi114 ceph-mon[36376]: fsmap 3 up:standby 2024-03-11T19:55:10.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:10 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T19:55:11.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.018+0000 7fd136c00640 1 -- 172.21.15.97:0/1050182516 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1301a1d70 con 0x7fd130101a20 2024-03-11T19:55:11.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.019+0000 7fd12e000640 1 -- 172.21.15.97:0/1050182516 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2910 (secure 0 0 0) 0x7fd12809a230 con 0x7fd130101a20 2024-03-11T19:55:11.022 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:11.022 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-11T19:55:11.022 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 4 2024-03-11T19:55:11.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.021+0000 7fd136c00640 1 -- 172.21.15.97:0/1050182516 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd1180a1ab0 msgr2=0x7fd1180a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:11.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.021+0000 7fd136c00640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd1180a1ab0 0x7fd1180a3f70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd12402c240 tx=0x7fd12402c7e0 comp rx=0 tx=0).stop 2024-03-11T19:55:11.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.021+0000 7fd136c00640 1 -- 172.21.15.97:0/1050182516 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd130101a20 msgr2=0x7fd1301a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:11.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.021+0000 7fd136c00640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd130101a20 0x7fd1301a0810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd12800eb10 tx=0x7fd12800c2d0 comp rx=0 tx=0).stop 2024-03-11T19:55:11.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.021+0000 7fd136c00640 1 -- 172.21.15.97:0/1050182516 shutdown_connections 2024-03-11T19:55:11.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.021+0000 7fd136c00640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd1180a1ab0 0x7fd1180a3f70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:11.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.021+0000 7fd136c00640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd130101a20 0x7fd1301a0810 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:11.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.021+0000 7fd136c00640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd130101060 0x7fd1301997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:11.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.021+0000 7fd136c00640 1 --2- 172.21.15.97:0/1050182516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1300ffe80 0x7fd130199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:11.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.021+0000 7fd136c00640 1 -- 172.21.15.97:0/1050182516 >> 172.21.15.97:0/1050182516 conn(0x7fd1300fb460 msgr2=0x7fd1300fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:11.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.021+0000 7fd136c00640 1 -- 172.21.15.97:0/1050182516 shutdown_connections 2024-03-11T19:55:11.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:11.022+0000 7fd136c00640 1 -- 172.21.15.97:0/1050182516 wait complete. 2024-03-11T19:55:11.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:10 smithi161 ceph-mon[36332]: Deploying daemon mds.d on smithi097 2024-03-11T19:55:11.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:10 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.97:6834/207363921,v1:172.21.15.97:6835/207363921] up:boot 2024-03-11T19:55:11.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:10 smithi161 ceph-mon[36332]: fsmap 3 up:standby 2024-03-11T19:55:11.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:10 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T19:55:11.785 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:11 smithi097 ceph-mon[31898]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:11.785 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:11 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1050182516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:11.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:11 smithi114 ceph-mon[36376]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:11.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:11 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1050182516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:12.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:11 smithi161 ceph-mon[36332]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:12.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:11 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1050182516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:13.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:13.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.250+0000 7ff753200640 1 -- 172.21.15.97:0/453003634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff74c1045b0 msgr2=0x7ff74c104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:13.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.250+0000 7ff753200640 1 --2- 172.21.15.97:0/453003634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff74c1045b0 0x7ff74c104a30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff738009900 tx=0x7ff73802bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:13.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.250+0000 7ff753200640 1 -- 172.21.15.97:0/453003634 shutdown_connections 2024-03-11T19:55:13.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.250+0000 7ff753200640 1 --2- 172.21.15.97:0/453003634 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff74c104f70 0x7ff74c10b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:13.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.250+0000 7ff753200640 1 --2- 172.21.15.97:0/453003634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff74c1045b0 0x7ff74c104a30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:13.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.250+0000 7ff753200640 1 --2- 172.21.15.97:0/453003634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff74c1033c0 0x7ff74c1037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:13.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.251+0000 7ff753200640 1 -- 172.21.15.97:0/453003634 >> 172.21.15.97:0/453003634 conn(0x7ff74c0fe970 msgr2=0x7ff74c100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:13.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.251+0000 7ff753200640 1 -- 172.21.15.97:0/453003634 shutdown_connections 2024-03-11T19:55:13.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.251+0000 7ff753200640 1 -- 172.21.15.97:0/453003634 wait complete. 2024-03-11T19:55:13.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.252+0000 7ff753200640 1 Processor -- start 2024-03-11T19:55:13.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.252+0000 7ff753200640 1 -- start start 2024-03-11T19:55:13.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff753200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff74c1033c0 0x7ff74c199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:13.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff753200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff74c1045b0 0x7ff74c1999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:13.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff753200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff74c104f70 0x7ff74c1a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:13.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff753200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c1a0f50 con 0x7ff74c1033c0 2024-03-11T19:55:13.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff753200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c1a10c0 con 0x7ff74c1045b0 2024-03-11T19:55:13.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff753200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c1a1280 con 0x7ff74c104f70 2024-03-11T19:55:13.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff74be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff74c1033c0 0x7ff74c199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:13.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff74be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff74c1033c0 0x7ff74c199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37384/0 (socket says 172.21.15.97:37384) 2024-03-11T19:55:13.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff74be00640 1 -- 172.21.15.97:0/671905257 learned_addr learned my addr 172.21.15.97:0/671905257 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:13.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff74b400640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff74c1045b0 0x7ff74c1999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:13.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff751200640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff74c104f70 0x7ff74c1a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:13.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff74be00640 1 -- 172.21.15.97:0/671905257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff74c104f70 msgr2=0x7ff74c1a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:13.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff74be00640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff74c104f70 0x7ff74c1a0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:13.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff74be00640 1 -- 172.21.15.97:0/671905257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff74c1045b0 msgr2=0x7ff74c1999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:13.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff74be00640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff74c1045b0 0x7ff74c1999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:13.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.253+0000 7ff74be00640 1 -- 172.21.15.97:0/671905257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73c009490 con 0x7ff74c1033c0 2024-03-11T19:55:13.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.254+0000 7ff74be00640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff74c1033c0 0x7ff74c199460 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00e9c0 tx=0x7ff73c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:13.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.254+0000 7ff748c00640 1 -- 172.21.15.97:0/671905257 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c00cde0 con 0x7ff74c1033c0 2024-03-11T19:55:13.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.254+0000 7ff748c00640 1 -- 172.21.15.97:0/671905257 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff73c010910 con 0x7ff74c1033c0 2024-03-11T19:55:13.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.254+0000 7ff753200640 1 -- 172.21.15.97:0/671905257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff738009560 con 0x7ff74c1033c0 2024-03-11T19:55:13.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.254+0000 7ff748c00640 1 -- 172.21.15.97:0/671905257 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c018a10 con 0x7ff74c1033c0 2024-03-11T19:55:13.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.254+0000 7ff753200640 1 -- 172.21.15.97:0/671905257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74c1a18b0 con 0x7ff74c1033c0 2024-03-11T19:55:13.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.255+0000 7ff753200640 1 -- 172.21.15.97:0/671905257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff74c10a6e0 con 0x7ff74c1033c0 2024-03-11T19:55:13.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.257+0000 7ff748c00640 1 -- 172.21.15.97:0/671905257 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff73c010a80 con 0x7ff74c1033c0 2024-03-11T19:55:13.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.259+0000 7ff748c00640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff7300a1a30 0x7ff7300a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:13.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.259+0000 7ff748c00640 1 -- 172.21.15.97:0/671905257 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff73c014070 con 0x7ff74c1033c0 2024-03-11T19:55:13.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.259+0000 7ff74b400640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff7300a1a30 0x7ff7300a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:13.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.260+0000 7ff74b400640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff7300a1a30 0x7ff7300a3ef0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff738009230 tx=0x7ff738005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:13.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.261+0000 7ff748c00640 1 -- 172.21.15.97:0/671905257 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff73c0972f0 con 0x7ff74c1033c0 2024-03-11T19:55:13.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.510+0000 7ff753200640 1 -- 172.21.15.97:0/671905257 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff74c10eef0 con 0x7ff74c1033c0 2024-03-11T19:55:13.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.511+0000 7ff748c00640 1 -- 172.21.15.97:0/671905257 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2910 (secure 0 0 0) 0x7ff73c09a9b0 con 0x7ff74c1033c0 2024-03-11T19:55:13.514 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:13.514 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-11T19:55:13.515 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 4 2024-03-11T19:55:13.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.514+0000 7ff753200640 1 -- 172.21.15.97:0/671905257 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff7300a1a30 msgr2=0x7ff7300a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:13.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.514+0000 7ff753200640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff7300a1a30 0x7ff7300a3ef0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff738009230 tx=0x7ff738005fd0 comp rx=0 tx=0).stop 2024-03-11T19:55:13.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.514+0000 7ff753200640 1 -- 172.21.15.97:0/671905257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff74c1033c0 msgr2=0x7ff74c199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:13.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.514+0000 7ff753200640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff74c1033c0 0x7ff74c199460 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00e9c0 tx=0x7ff73c00ee80 comp rx=0 tx=0).stop 2024-03-11T19:55:13.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.514+0000 7ff753200640 1 -- 172.21.15.97:0/671905257 shutdown_connections 2024-03-11T19:55:13.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.514+0000 7ff753200640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff74c104f70 0x7ff74c1a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:13.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.514+0000 7ff753200640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff74c1045b0 0x7ff74c1999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:13.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.514+0000 7ff753200640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff7300a1a30 0x7ff7300a3ef0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:13.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.514+0000 7ff753200640 1 --2- 172.21.15.97:0/671905257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff74c1033c0 0x7ff74c199460 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:13.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.514+0000 7ff753200640 1 -- 172.21.15.97:0/671905257 >> 172.21.15.97:0/671905257 conn(0x7ff74c0fe970 msgr2=0x7ff74c1078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:13.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.514+0000 7ff753200640 1 -- 172.21.15.97:0/671905257 shutdown_connections 2024-03-11T19:55:13.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:13.514+0000 7ff753200640 1 -- 172.21.15.97:0/671905257 wait complete. 2024-03-11T19:55:13.532 INFO:journalctl@ceph.mds.d.smithi097.stdout:Mar 11 19:55:13 smithi097 systemd[1]: Started Ceph mds.d for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:55:13.828 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:13 smithi097 ceph-mon[31898]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:13.829 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:13 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:13.829 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:13 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:13.829 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:13 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:13.829 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:13 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:13.829 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:13 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:13.829 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:13 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:13.829 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:13 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/671905257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:13.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:13 smithi114 ceph-mon[36376]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:13.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:13 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:13.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:13 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:13.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:13 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:13.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:13 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:13.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:13 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:13.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:13 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:13.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:13 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/671905257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:14.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:13 smithi161 ceph-mon[36332]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:14.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:13 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:14.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:13 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:14.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:13 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:14.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:13 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:14.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:13 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:14.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:13 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:14.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:13 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/671905257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:14.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:14 smithi114 ceph-mon[36376]: Deploying daemon mds.e on smithi114 2024-03-11T19:55:14.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:14 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.97:6836/267259401,v1:172.21.15.97:6837/267259401] up:boot 2024-03-11T19:55:14.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:14 smithi114 ceph-mon[36376]: fsmap 4 up:standby 2024-03-11T19:55:14.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:14 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T19:55:14.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:14 smithi097 ceph-mon[31898]: Deploying daemon mds.e on smithi114 2024-03-11T19:55:14.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:14 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.97:6836/267259401,v1:172.21.15.97:6837/267259401] up:boot 2024-03-11T19:55:14.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:14 smithi097 ceph-mon[31898]: fsmap 4 up:standby 2024-03-11T19:55:14.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:14 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T19:55:15.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:14 smithi161 ceph-mon[36332]: Deploying daemon mds.e on smithi114 2024-03-11T19:55:15.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:14 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.97:6836/267259401,v1:172.21.15.97:6837/267259401] up:boot 2024-03-11T19:55:15.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:14 smithi161 ceph-mon[36332]: fsmap 4 up:standby 2024-03-11T19:55:15.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:14 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T19:55:15.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:15.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:15 smithi114 ceph-mon[36376]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:15.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.747+0000 7fdf11200640 1 -- 172.21.15.97:0/4245556232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c1033c0 msgr2=0x7fdf0c1037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:15.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.747+0000 7fdf11200640 1 --2- 172.21.15.97:0/4245556232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c1033c0 0x7fdf0c1037e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009900 tx=0x7fdefc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:15.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.747+0000 7fdf11200640 1 -- 172.21.15.97:0/4245556232 shutdown_connections 2024-03-11T19:55:15.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.747+0000 7fdf11200640 1 --2- 172.21.15.97:0/4245556232 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c104f70 0x7fdf0c10b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:15.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.747+0000 7fdf11200640 1 --2- 172.21.15.97:0/4245556232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf0c1045b0 0x7fdf0c104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:15.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.747+0000 7fdf11200640 1 --2- 172.21.15.97:0/4245556232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c1033c0 0x7fdf0c1037e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:15.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.747+0000 7fdf11200640 1 -- 172.21.15.97:0/4245556232 >> 172.21.15.97:0/4245556232 conn(0x7fdf0c0fe970 msgr2=0x7fdf0c100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:15.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.747+0000 7fdf11200640 1 -- 172.21.15.97:0/4245556232 shutdown_connections 2024-03-11T19:55:15.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.747+0000 7fdf11200640 1 -- 172.21.15.97:0/4245556232 wait complete. 2024-03-11T19:55:15.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.748+0000 7fdf11200640 1 Processor -- start 2024-03-11T19:55:15.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.748+0000 7fdf11200640 1 -- start start 2024-03-11T19:55:15.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.748+0000 7fdf11200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c1033c0 0x7fdf0c199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:15.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.748+0000 7fdf11200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf0c1045b0 0x7fdf0c1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:15.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.748+0000 7fdf11200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c104f70 0x7fdf0c1a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.748+0000 7fdf11200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c1a0f70 con 0x7fdf0c104f70 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.748+0000 7fdf11200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c1a10e0 con 0x7fdf0c1033c0 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.748+0000 7fdf11200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c1a12a0 con 0x7fdf0c1045b0 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.749+0000 7fdf09e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c1033c0 0x7fdf0c199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.749+0000 7fdf09e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c1033c0 0x7fdf0c199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:53454/0 (socket says 172.21.15.97:53454) 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.749+0000 7fdf09e00640 1 -- 172.21.15.97:0/3011538460 learned_addr learned my addr 172.21.15.97:0/3011538460 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.749+0000 7fdf0a800640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c104f70 0x7fdf0c1a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.749+0000 7fdf09400640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf0c1045b0 0x7fdf0c1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.749+0000 7fdf09e00640 1 -- 172.21.15.97:0/3011538460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf0c1045b0 msgr2=0x7fdf0c1999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.749+0000 7fdf09e00640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf0c1045b0 0x7fdf0c1999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.749+0000 7fdf09e00640 1 -- 172.21.15.97:0/3011538460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c104f70 msgr2=0x7fdf0c1a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.749+0000 7fdf09e00640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c104f70 0x7fdf0c1a0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.749+0000 7fdf09e00640 1 -- 172.21.15.97:0/3011538460 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc009560 con 0x7fdf0c1033c0 2024-03-11T19:55:15.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.749+0000 7fdf0a800640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c104f70 0x7fdf0c1a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:55:15.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.749+0000 7fdf09400640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf0c1045b0 0x7fdf0c1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:55:15.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.749+0000 7fdf09e00640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c1033c0 0x7fdf0c199480 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdefc000c00 tx=0x7fdefc0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:15.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.750+0000 7fdf02a00640 1 -- 172.21.15.97:0/3011538460 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc0044e0 con 0x7fdf0c1033c0 2024-03-11T19:55:15.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.750+0000 7fdf02a00640 1 -- 172.21.15.97:0/3011538460 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdefc0054e0 con 0x7fdf0c1033c0 2024-03-11T19:55:15.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.750+0000 7fdf11200640 1 -- 172.21.15.97:0/3011538460 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0c1a1520 con 0x7fdf0c1033c0 2024-03-11T19:55:15.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.750+0000 7fdf02a00640 1 -- 172.21.15.97:0/3011538460 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc03e450 con 0x7fdf0c1033c0 2024-03-11T19:55:15.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.750+0000 7fdf11200640 1 -- 172.21.15.97:0/3011538460 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0c1a1a80 con 0x7fdf0c1033c0 2024-03-11T19:55:15.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.751+0000 7fdf11200640 1 -- 172.21.15.97:0/3011538460 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0c10a760 con 0x7fdf0c1033c0 2024-03-11T19:55:15.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.752+0000 7fdf02a00640 1 -- 172.21.15.97:0/3011538460 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fdefc0035f0 con 0x7fdf0c1033c0 2024-03-11T19:55:15.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.754+0000 7fdf02a00640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdef00a1ab0 0x7fdef00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:15.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.754+0000 7fdf02a00640 1 -- 172.21.15.97:0/3011538460 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdefc0efa10 con 0x7fdf0c1033c0 2024-03-11T19:55:15.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.754+0000 7fdf09400640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdef00a1ab0 0x7fdef00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:15.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.755+0000 7fdf09400640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdef00a1ab0 0x7fdef00a3f70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdef80058c0 tx=0x7fdef8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:15.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:15.758+0000 7fdf02a00640 1 -- 172.21.15.97:0/3011538460 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdefc0b9020 con 0x7fdf0c1033c0 2024-03-11T19:55:15.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:15 smithi097 ceph-mon[31898]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:16.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.002+0000 7fdf11200640 1 -- 172.21.15.97:0/3011538460 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf0c02cdd0 con 0x7fdf0c1033c0 2024-03-11T19:55:16.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.003+0000 7fdf02a00640 1 -- 172.21.15.97:0/3011538460 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3698 (secure 0 0 0) 0x7fdefc043e90 con 0x7fdf0c1033c0 2024-03-11T19:55:16.006 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:16.006 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24499,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[]} 2024-03-11T19:55:16.007 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 5 2024-03-11T19:55:16.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.006+0000 7fdf11200640 1 -- 172.21.15.97:0/3011538460 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdef00a1ab0 msgr2=0x7fdef00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:16.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.006+0000 7fdf11200640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdef00a1ab0 0x7fdef00a3f70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdef80058c0 tx=0x7fdef8005850 comp rx=0 tx=0).stop 2024-03-11T19:55:16.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.006+0000 7fdf11200640 1 -- 172.21.15.97:0/3011538460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c1033c0 msgr2=0x7fdf0c199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:16.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.006+0000 7fdf11200640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c1033c0 0x7fdf0c199480 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdefc000c00 tx=0x7fdefc0042f0 comp rx=0 tx=0).stop 2024-03-11T19:55:16.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.006+0000 7fdf11200640 1 -- 172.21.15.97:0/3011538460 shutdown_connections 2024-03-11T19:55:16.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.006+0000 7fdf11200640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdef00a1ab0 0x7fdef00a3f70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:16.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.006+0000 7fdf11200640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdf0c104f70 0x7fdf0c1a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:16.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.006+0000 7fdf11200640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdf0c1045b0 0x7fdf0c1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:16.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.006+0000 7fdf11200640 1 --2- 172.21.15.97:0/3011538460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf0c1033c0 0x7fdf0c199480 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:16.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.006+0000 7fdf11200640 1 -- 172.21.15.97:0/3011538460 >> 172.21.15.97:0/3011538460 conn(0x7fdf0c0fe970 msgr2=0x7fdf0c107930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:16.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.006+0000 7fdf11200640 1 -- 172.21.15.97:0/3011538460 shutdown_connections 2024-03-11T19:55:16.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:16.006+0000 7fdf11200640 1 -- 172.21.15.97:0/3011538460 wait complete. 2024-03-11T19:55:16.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:15 smithi161 ceph-mon[36332]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:16.492 INFO:journalctl@ceph.mds.e.smithi114.stdout:Mar 11 19:55:16 smithi114 systemd[1]: Started Ceph mds.e for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:55:16.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:16 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3011538460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:16.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:16 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:16.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:16 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:16.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:16 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:16.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:16 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:16.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:16 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:16.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:16 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:16.938 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:16 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3011538460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:16.938 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:16 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:16.939 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:16 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:16.939 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:16 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:16.939 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:16 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:16.939 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:16 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:16.939 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:16 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:16.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:16 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3011538460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:16.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:16 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:16.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:16 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:16.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:16 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:16.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:16 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:16.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:16 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:16.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:16 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:17.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:17 smithi097 ceph-mon[31898]: Deploying daemon mds.f on smithi161 2024-03-11T19:55:17.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:17 smithi097 ceph-mon[31898]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:17.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:17 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.114:6834/2623690462,v1:172.21.15.114:6835/2623690462] up:boot 2024-03-11T19:55:17.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:17 smithi097 ceph-mon[31898]: fsmap 5 up:standby 2024-03-11T19:55:17.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:55:17.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:17 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:55:17.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:17 smithi114 ceph-mon[36376]: Deploying daemon mds.f on smithi161 2024-03-11T19:55:17.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:17 smithi114 ceph-mon[36376]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:17 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.114:6834/2623690462,v1:172.21.15.114:6835/2623690462] up:boot 2024-03-11T19:55:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:17 smithi114 ceph-mon[36376]: fsmap 5 up:standby 2024-03-11T19:55:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:55:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:17 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:55:18.029 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:18.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:17 smithi161 ceph-mon[36332]: Deploying daemon mds.f on smithi161 2024-03-11T19:55:18.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:17 smithi161 ceph-mon[36332]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:18.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:17 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.114:6834/2623690462,v1:172.21.15.114:6835/2623690462] up:boot 2024-03-11T19:55:18.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:17 smithi161 ceph-mon[36332]: fsmap 5 up:standby 2024-03-11T19:55:18.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:55:18.040 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:17 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:55:18.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.243+0000 7fa019c00640 1 -- 172.21.15.97:0/147419722 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa014104d80 msgr2=0x7fa01410b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:18.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.243+0000 7fa019c00640 1 --2- 172.21.15.97:0/147419722 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa014104d80 0x7fa01410b610 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009ca0 tx=0x7fa00c02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:55:18.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.243+0000 7fa019c00640 1 -- 172.21.15.97:0/147419722 shutdown_connections 2024-03-11T19:55:18.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.243+0000 7fa019c00640 1 --2- 172.21.15.97:0/147419722 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa014104d80 0x7fa01410b610 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:18.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.243+0000 7fa019c00640 1 --2- 172.21.15.97:0/147419722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0141043c0 0x7fa014104840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:18.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.243+0000 7fa019c00640 1 --2- 172.21.15.97:0/147419722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0141031d0 0x7fa0141035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:18.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.243+0000 7fa019c00640 1 -- 172.21.15.97:0/147419722 >> 172.21.15.97:0/147419722 conn(0x7fa0140fe760 msgr2=0x7fa014100b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:18.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.243+0000 7fa019c00640 1 -- 172.21.15.97:0/147419722 shutdown_connections 2024-03-11T19:55:18.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.243+0000 7fa019c00640 1 -- 172.21.15.97:0/147419722 wait complete. 2024-03-11T19:55:18.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.244+0000 7fa019c00640 1 Processor -- start 2024-03-11T19:55:18.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa019c00640 1 -- start start 2024-03-11T19:55:18.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa019c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0141031d0 0x7fa014199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:18.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa019c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0141043c0 0x7fa014199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:18.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa019c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa014104d80 0x7fa0141a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:18.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa019c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0141a0cf0 con 0x7fa014104d80 2024-03-11T19:55:18.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa019c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0141a0e60 con 0x7fa0141031d0 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa019c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0141a1020 con 0x7fa0141043c0 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa013400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0141043c0 0x7fa014199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa013400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0141043c0 0x7fa014199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:48006/0 (socket says 172.21.15.97:48006) 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa019200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa014104d80 0x7fa0141a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa013400640 1 -- 172.21.15.97:0/3686892368 learned_addr learned my addr 172.21.15.97:0/3686892368 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa013e00640 1 --2- 172.21.15.97:0/3686892368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0141031d0 0x7fa014199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa013400640 1 -- 172.21.15.97:0/3686892368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0141031d0 msgr2=0x7fa014199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa013400640 1 --2- 172.21.15.97:0/3686892368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0141031d0 0x7fa014199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa013400640 1 -- 172.21.15.97:0/3686892368 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa014104d80 msgr2=0x7fa0141a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.245+0000 7fa013400640 1 --2- 172.21.15.97:0/3686892368 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa014104d80 0x7fa0141a07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.246+0000 7fa013400640 1 -- 172.21.15.97:0/3686892368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00c009900 con 0x7fa0141043c0 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.246+0000 7fa013400640 1 --2- 172.21.15.97:0/3686892368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0141043c0 0x7fa014199740 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa00000e9f0 tx=0x7fa00000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.246+0000 7fa010c00640 1 -- 172.21.15.97:0/3686892368 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa000009720 con 0x7fa0141043c0 2024-03-11T19:55:18.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.246+0000 7fa010c00640 1 -- 172.21.15.97:0/3686892368 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa00000cd80 con 0x7fa0141043c0 2024-03-11T19:55:18.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.246+0000 7fa019c00640 1 -- 172.21.15.97:0/3686892368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0141a1300 con 0x7fa0141043c0 2024-03-11T19:55:18.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.246+0000 7fa010c00640 1 -- 172.21.15.97:0/3686892368 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa000010690 con 0x7fa0141043c0 2024-03-11T19:55:18.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.246+0000 7fa019c00640 1 -- 172.21.15.97:0/3686892368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0141a17c0 con 0x7fa0141043c0 2024-03-11T19:55:18.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.247+0000 7fa019c00640 1 -- 172.21.15.97:0/3686892368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01410a490 con 0x7fa0141043c0 2024-03-11T19:55:18.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.250+0000 7fa010c00640 1 -- 172.21.15.97:0/3686892368 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa0000046c0 con 0x7fa0141043c0 2024-03-11T19:55:18.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.251+0000 7fa010c00640 1 --2- 172.21.15.97:0/3686892368 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9ff80a1a30 0x7f9ff80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:18.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.251+0000 7fa010c00640 1 -- 172.21.15.97:0/3686892368 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa000014070 con 0x7fa0141043c0 2024-03-11T19:55:18.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.251+0000 7fa013e00640 1 --2- 172.21.15.97:0/3686892368 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9ff80a1a30 0x7f9ff80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:18.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.252+0000 7fa013e00640 1 --2- 172.21.15.97:0/3686892368 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9ff80a1a30 0x7f9ff80a3ef0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa014104220 tx=0x7fa004005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:18.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.255+0000 7fa010c00640 1 -- 172.21.15.97:0/3686892368 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa000096a30 con 0x7fa0141043c0 2024-03-11T19:55:18.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.492+0000 7fa019c00640 1 -- 172.21.15.97:0/3686892368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa01410ebe0 con 0x7fa0141043c0 2024-03-11T19:55:18.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.494+0000 7fa010c00640 1 -- 172.21.15.97:0/3686892368 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4492 (secure 0 0 0) 0x7fa00009a0f0 con 0x7fa0141043c0 2024-03-11T19:55:18.497 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:18.497 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24499,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-11T19:55:18.497 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 6 2024-03-11T19:55:18.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.497+0000 7fa019c00640 1 -- 172.21.15.97:0/3686892368 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9ff80a1a30 msgr2=0x7f9ff80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:18.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.497+0000 7fa019c00640 1 --2- 172.21.15.97:0/3686892368 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9ff80a1a30 0x7f9ff80a3ef0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa014104220 tx=0x7fa004005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:18.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.497+0000 7fa019c00640 1 -- 172.21.15.97:0/3686892368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0141043c0 msgr2=0x7fa014199740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:18.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.497+0000 7fa019c00640 1 --2- 172.21.15.97:0/3686892368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0141043c0 0x7fa014199740 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa00000e9f0 tx=0x7fa00000edf0 comp rx=0 tx=0).stop 2024-03-11T19:55:18.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.497+0000 7fa019c00640 1 -- 172.21.15.97:0/3686892368 shutdown_connections 2024-03-11T19:55:18.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.497+0000 7fa019c00640 1 --2- 172.21.15.97:0/3686892368 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9ff80a1a30 0x7f9ff80a3ef0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:18.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.497+0000 7fa019c00640 1 --2- 172.21.15.97:0/3686892368 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa014104d80 0x7fa0141a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:18.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.497+0000 7fa019c00640 1 --2- 172.21.15.97:0/3686892368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0141043c0 0x7fa014199740 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:18.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.497+0000 7fa019c00640 1 --2- 172.21.15.97:0/3686892368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0141031d0 0x7fa014199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:18.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.497+0000 7fa019c00640 1 -- 172.21.15.97:0/3686892368 >> 172.21.15.97:0/3686892368 conn(0x7fa0140fe760 msgr2=0x7fa014107660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:18.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.497+0000 7fa019c00640 1 -- 172.21.15.97:0/3686892368 shutdown_connections 2024-03-11T19:55:18.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:18.497+0000 7fa019c00640 1 -- 172.21.15.97:0/3686892368 wait complete. 2024-03-11T19:55:18.636 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 19:55:18 smithi161 systemd[1]: Starting Ceph mds.f for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T19:55:18.887 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:18 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3686892368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:18.887 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 19:55:18 smithi161 podman[66925]: 2024-03-11 19:55:18.64274568 +0000 UTC m=+0.018178588 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:55:18.887 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 19:55:18 smithi161 podman[66925]: 2024-03-11 19:55:18.767452304 +0000 UTC m=+0.142885209 container create 7764197c160f3108a63d0eef42a22565f7a84d02f6c8d8e704cd70e5d7e1e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-11T19:55:18.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:18 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3686892368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:18.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:18 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3686892368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:19.288 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 19:55:19 smithi161 podman[66925]: 2024-03-11 19:55:19.028604946 +0000 UTC m=+0.404037863 container init 7764197c160f3108a63d0eef42a22565f7a84d02f6c8d8e704cd70e5d7e1e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T19:55:19.289 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 19:55:19 smithi161 podman[66925]: 2024-03-11 19:55:19.031212334 +0000 UTC m=+0.406645241 container start 7764197c160f3108a63d0eef42a22565f7a84d02f6c8d8e704cd70e5d7e1e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-f, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:55:19.289 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 19:55:19 smithi161 ceph-mds[66944]: -- 172.21.15.161:0/2507069200 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a9b6c1c1a0 con 0x55a9b6bf3000 2024-03-11T19:55:19.289 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 19:55:19 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-f[66940]: starting mds.f at 2024-03-11T19:55:19.289 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 19:55:19 smithi161 bash[66925]: 7764197c160f3108a63d0eef42a22565f7a84d02f6c8d8e704cd70e5d7e1e3a3 2024-03-11T19:55:19.289 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 19:55:19 smithi161 systemd[1]: Started Ceph mds.f for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:55:19.890 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:19 smithi161 ceph-mon[36332]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:19.890 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:19.891 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:19.891 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:19.891 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:19.891 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:19.891 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:19 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:19.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:19 smithi097 ceph-mon[31898]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:19.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:19.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:19.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:19.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:19.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:19.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:19 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:19.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:19 smithi114 ceph-mon[36376]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:19.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:19.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:19.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:19.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:19.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:19.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:19 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:20.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:20.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.727+0000 7f782d000640 1 -- 172.21.15.97:0/2328494670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78281077c0 msgr2=0x7f7828107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:20.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.727+0000 7f782d000640 1 --2- 172.21.15.97:0/2328494670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78281077c0 0x7f7828107be0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7818009900 tx=0x7f781802bc50 comp rx=0 tx=0).stop 2024-03-11T19:55:20.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.727+0000 7f782d000640 1 -- 172.21.15.97:0/2328494670 shutdown_connections 2024-03-11T19:55:20.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.727+0000 7f782d000640 1 --2- 172.21.15.97:0/2328494670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7828109370 0x7f782810fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:20.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.727+0000 7f782d000640 1 --2- 172.21.15.97:0/2328494670 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f78281089b0 0x7f7828108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:20.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.727+0000 7f782d000640 1 --2- 172.21.15.97:0/2328494670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78281077c0 0x7f7828107be0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:20.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.727+0000 7f782d000640 1 -- 172.21.15.97:0/2328494670 >> 172.21.15.97:0/2328494670 conn(0x7f7828076790 msgr2=0x7f7828078bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:20.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.727+0000 7f782d000640 1 -- 172.21.15.97:0/2328494670 shutdown_connections 2024-03-11T19:55:20.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.727+0000 7f782d000640 1 -- 172.21.15.97:0/2328494670 wait complete. 2024-03-11T19:55:20.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.728+0000 7f782d000640 1 Processor -- start 2024-03-11T19:55:20.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.729+0000 7f782d000640 1 -- start start 2024-03-11T19:55:20.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.729+0000 7f782d000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78281077c0 0x7f782819d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:20.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.729+0000 7f782d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78281089b0 0x7f782819ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:20.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.729+0000 7f782d000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7828109370 0x7f78281a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:20.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.729+0000 7f782d000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78281a5380 con 0x7f78281077c0 2024-03-11T19:55:20.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.729+0000 7f782d000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78281a54f0 con 0x7f78281089b0 2024-03-11T19:55:20.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.729+0000 7f782d000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78281a56b0 con 0x7f7828109370 2024-03-11T19:55:20.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.729+0000 7f7825e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78281077c0 0x7f782819d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:20.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.729+0000 7f7825e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78281077c0 0x7f782819d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60500/0 (socket says 172.21.15.97:60500) 2024-03-11T19:55:20.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.729+0000 7f7825e00640 1 -- 172.21.15.97:0/3959008361 learned_addr learned my addr 172.21.15.97:0/3959008361 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:20.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.729+0000 7f7826800640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7828109370 0x7f78281a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:20.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.729+0000 7f7825400640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78281089b0 0x7f782819ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:20.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.730+0000 7f7825e00640 1 -- 172.21.15.97:0/3959008361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7828109370 msgr2=0x7f78281a4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:20.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.730+0000 7f7825e00640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7828109370 0x7f78281a4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:20.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.730+0000 7f7825e00640 1 -- 172.21.15.97:0/3959008361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78281089b0 msgr2=0x7f782819ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:20.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.730+0000 7f7825e00640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78281089b0 0x7f782819ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:20.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.730+0000 7f7825e00640 1 -- 172.21.15.97:0/3959008361 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7818009560 con 0x7f78281077c0 2024-03-11T19:55:20.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.730+0000 7f7826800640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7828109370 0x7f78281a4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:20.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.730+0000 7f7825e00640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78281077c0 0x7f782819d890 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7818005ab0 tx=0x7f7818005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:20.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.730+0000 7f781ea00640 1 -- 172.21.15.97:0/3959008361 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78180043d0 con 0x7f78281077c0 2024-03-11T19:55:20.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.731+0000 7f781ea00640 1 -- 172.21.15.97:0/3959008361 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78180052d0 con 0x7f78281077c0 2024-03-11T19:55:20.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.731+0000 7f781ea00640 1 -- 172.21.15.97:0/3959008361 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818003d40 con 0x7f78281077c0 2024-03-11T19:55:20.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.731+0000 7f782d000640 1 -- 172.21.15.97:0/3959008361 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78281a5930 con 0x7f78281077c0 2024-03-11T19:55:20.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.731+0000 7f782d000640 1 -- 172.21.15.97:0/3959008361 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78281a5df0 con 0x7f78281077c0 2024-03-11T19:55:20.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.732+0000 7f782d000640 1 -- 172.21.15.97:0/3959008361 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782810eae0 con 0x7f78281077c0 2024-03-11T19:55:20.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.734+0000 7f781ea00640 1 -- 172.21.15.97:0/3959008361 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7818005440 con 0x7f78281077c0 2024-03-11T19:55:20.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.735+0000 7f781ea00640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f780c0a1b30 0x7f780c0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:20.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.735+0000 7f781ea00640 1 -- 172.21.15.97:0/3959008361 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f78180ef690 con 0x7f78281077c0 2024-03-11T19:55:20.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.735+0000 7f7825400640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f780c0a1b30 0x7f780c0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:20.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.736+0000 7f7825400640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f780c0a1b30 0x7f780c0a3ff0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f782819eda0 tx=0x7f7814005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:20.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.738+0000 7f781ea00640 1 -- 172.21.15.97:0/3959008361 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f78180b8ca0 con 0x7f78281077c0 2024-03-11T19:55:20.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:20 smithi097 ceph-mon[31898]: Deploying daemon mds.c on smithi161 2024-03-11T19:55:20.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:20 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.161:6834/3358622196,v1:172.21.15.161:6835/3358622196] up:boot 2024-03-11T19:55:20.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:20 smithi097 ceph-mon[31898]: fsmap 6 up:standby 2024-03-11T19:55:20.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:20 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T19:55:20.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.973+0000 7f782d000640 1 -- 172.21.15.97:0/3959008361 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f782810e550 con 0x7f78281077c0 2024-03-11T19:55:20.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.974+0000 7f781ea00640 1 -- 172.21.15.97:0/3959008361 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5286 (secure 0 0 0) 0x7f78180dc030 con 0x7f78281077c0 2024-03-11T19:55:20.977 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:20.977 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24499,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-11T19:55:20.978 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 7 2024-03-11T19:55:20.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.977+0000 7f782d000640 1 -- 172.21.15.97:0/3959008361 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f780c0a1b30 msgr2=0x7f780c0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:20.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.977+0000 7f782d000640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f780c0a1b30 0x7f780c0a3ff0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f782819eda0 tx=0x7f7814005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:20.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.977+0000 7f782d000640 1 -- 172.21.15.97:0/3959008361 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78281077c0 msgr2=0x7f782819d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:20.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.977+0000 7f782d000640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78281077c0 0x7f782819d890 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7818005ab0 tx=0x7f7818005e50 comp rx=0 tx=0).stop 2024-03-11T19:55:20.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.977+0000 7f782d000640 1 -- 172.21.15.97:0/3959008361 shutdown_connections 2024-03-11T19:55:20.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.977+0000 7f782d000640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7828109370 0x7f78281a4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:20.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.977+0000 7f782d000640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78281089b0 0x7f782819ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:20.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.977+0000 7f782d000640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f780c0a1b30 0x7f780c0a3ff0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:20.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.977+0000 7f782d000640 1 --2- 172.21.15.97:0/3959008361 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f78281077c0 0x7f782819d890 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:20.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.977+0000 7f782d000640 1 -- 172.21.15.97:0/3959008361 >> 172.21.15.97:0/3959008361 conn(0x7f7828076790 msgr2=0x7f782810bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:20.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.977+0000 7f782d000640 1 -- 172.21.15.97:0/3959008361 shutdown_connections 2024-03-11T19:55:20.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:20.977+0000 7f782d000640 1 -- 172.21.15.97:0/3959008361 wait complete. 2024-03-11T19:55:20.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:20 smithi114 ceph-mon[36376]: Deploying daemon mds.c on smithi161 2024-03-11T19:55:20.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:20 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.161:6834/3358622196,v1:172.21.15.161:6835/3358622196] up:boot 2024-03-11T19:55:20.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:20 smithi114 ceph-mon[36376]: fsmap 6 up:standby 2024-03-11T19:55:20.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:20 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T19:55:20.998 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:20 smithi161 ceph-mon[36332]: Deploying daemon mds.c on smithi161 2024-03-11T19:55:20.998 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:20 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.161:6834/3358622196,v1:172.21.15.161:6835/3358622196] up:boot 2024-03-11T19:55:20.998 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:20 smithi161 ceph-mon[36332]: fsmap 6 up:standby 2024-03-11T19:55:20.998 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:20 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T19:55:21.657 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 19:55:21 smithi161 systemd[1]: Starting Ceph mds.c for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T19:55:21.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:21 smithi097 ceph-mon[31898]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:21.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:21 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3959008361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:21.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:21 smithi161 ceph-mon[36332]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:21.942 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:21 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3959008361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:21.942 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 19:55:21 smithi161 podman[67230]: 2024-03-11 19:55:21.590769415 +0000 UTC m=+0.017805865 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:55:21.942 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 19:55:21 smithi161 podman[67230]: 2024-03-11 19:55:21.715179939 +0000 UTC m=+0.142216431 container create 1a128a11cdb93320cb689a669d9bb3a4d32aeab2c086a2fc751ba604b4575491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307) 2024-03-11T19:55:21.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:21 smithi114 ceph-mon[36376]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:21.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:21 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3959008361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:22.289 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 19:55:22 smithi161 podman[67230]: 2024-03-11 19:55:22.06796355 +0000 UTC m=+0.495000011 container init 1a128a11cdb93320cb689a669d9bb3a4d32aeab2c086a2fc751ba604b4575491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-c, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T19:55:22.289 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 19:55:22 smithi161 podman[67230]: 2024-03-11 19:55:22.070643159 +0000 UTC m=+0.497679609 container start 1a128a11cdb93320cb689a669d9bb3a4d32aeab2c086a2fc751ba604b4575491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-c, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T19:55:22.289 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 19:55:22 smithi161 ceph-mds[67251]: -- 172.21.15.161:0/2194883787 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56429b7b21a0 con 0x56429b788800 2024-03-11T19:55:22.289 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 19:55:22 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-c[67247]: starting mds.c at 2024-03-11T19:55:22.289 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 19:55:22 smithi161 bash[67230]: 1a128a11cdb93320cb689a669d9bb3a4d32aeab2c086a2fc751ba604b4575491 2024-03-11T19:55:22.289 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 19:55:22 smithi161 systemd[1]: Started Ceph mds.c for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:55:23.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:23.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.218+0000 7f663a200640 1 -- 172.21.15.97:0/46280211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66341033c0 msgr2=0x7f66341037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.218+0000 7f663a200640 1 --2- 172.21.15.97:0/46280211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66341033c0 0x7f66341037e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6624009900 tx=0x7f662402bc40 comp rx=0 tx=0).stop 2024-03-11T19:55:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.219+0000 7f663a200640 1 -- 172.21.15.97:0/46280211 shutdown_connections 2024-03-11T19:55:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.219+0000 7f663a200640 1 --2- 172.21.15.97:0/46280211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6634104f70 0x7f663410b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.219+0000 7f663a200640 1 --2- 172.21.15.97:0/46280211 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66341045b0 0x7f6634104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.219+0000 7f663a200640 1 --2- 172.21.15.97:0/46280211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66341033c0 0x7f66341037e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.219+0000 7f663a200640 1 -- 172.21.15.97:0/46280211 >> 172.21.15.97:0/46280211 conn(0x7f66340fe970 msgr2=0x7f6634100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.219+0000 7f663a200640 1 -- 172.21.15.97:0/46280211 shutdown_connections 2024-03-11T19:55:23.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.219+0000 7f663a200640 1 -- 172.21.15.97:0/46280211 wait complete. 2024-03-11T19:55:23.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.220+0000 7f663a200640 1 Processor -- start 2024-03-11T19:55:23.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.220+0000 7f663a200640 1 -- start start 2024-03-11T19:55:23.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.220+0000 7f663a200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66341033c0 0x7f66341994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:23.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.220+0000 7f663a200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66341045b0 0x7f66341999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:23.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.220+0000 7f663a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6634104f70 0x7f66341a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:23.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.220+0000 7f663a200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66341a0f90 con 0x7f6634104f70 2024-03-11T19:55:23.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.220+0000 7f663a200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66341a1100 con 0x7f66341033c0 2024-03-11T19:55:23.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.220+0000 7f663a200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66341a12c0 con 0x7f66341045b0 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.220+0000 7f6633400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66341033c0 0x7f66341994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.220+0000 7f6633400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66341033c0 0x7f66341994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:53512/0 (socket says 172.21.15.97:53512) 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.220+0000 7f6633e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6634104f70 0x7f66341a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.220+0000 7f6633400640 1 -- 172.21.15.97:0/1068343276 learned_addr learned my addr 172.21.15.97:0/1068343276 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.220+0000 7f662aa00640 1 --2- 172.21.15.97:0/1068343276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66341045b0 0x7f66341999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.221+0000 7f6633400640 1 -- 172.21.15.97:0/1068343276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66341045b0 msgr2=0x7f66341999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.221+0000 7f6633400640 1 --2- 172.21.15.97:0/1068343276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66341045b0 0x7f66341999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.221+0000 7f6633400640 1 -- 172.21.15.97:0/1068343276 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6634104f70 msgr2=0x7f66341a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.221+0000 7f6633400640 1 --2- 172.21.15.97:0/1068343276 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6634104f70 0x7f66341a0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.221+0000 7f6633400640 1 -- 172.21.15.97:0/1068343276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c009490 con 0x7f66341033c0 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.221+0000 7f6633400640 1 --2- 172.21.15.97:0/1068343276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66341033c0 0x7f66341994a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6624005ab0 tx=0x7f6624005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.221+0000 7f6630c00640 1 -- 172.21.15.97:0/1068343276 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66240043d0 con 0x7f66341033c0 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.221+0000 7f663a200640 1 -- 172.21.15.97:0/1068343276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662c009840 con 0x7f66341033c0 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.221+0000 7f6630c00640 1 -- 172.21.15.97:0/1068343276 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66240052d0 con 0x7f66341033c0 2024-03-11T19:55:23.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.221+0000 7f6630c00640 1 -- 172.21.15.97:0/1068343276 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6624003cd0 con 0x7f66341033c0 2024-03-11T19:55:23.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.222+0000 7f663a200640 1 -- 172.21.15.97:0/1068343276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6624009560 con 0x7f66341033c0 2024-03-11T19:55:23.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.223+0000 7f663a200640 1 -- 172.21.15.97:0/1068343276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663410a760 con 0x7f66341033c0 2024-03-11T19:55:23.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.224+0000 7f6630c00640 1 -- 172.21.15.97:0/1068343276 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6624005440 con 0x7f66341033c0 2024-03-11T19:55:23.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.226+0000 7f6630c00640 1 --2- 172.21.15.97:0/1068343276 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66180a1a30 0x7f66180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:23.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.226+0000 7f6630c00640 1 -- 172.21.15.97:0/1068343276 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f66240ef120 con 0x7f66341033c0 2024-03-11T19:55:23.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.226+0000 7f662aa00640 1 --2- 172.21.15.97:0/1068343276 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66180a1a30 0x7f66180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:23.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.226+0000 7f662aa00640 1 --2- 172.21.15.97:0/1068343276 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66180a1a30 0x7f66180a3ef0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f663419a9b0 tx=0x7f661c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:23.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.230+0000 7f6630c00640 1 -- 172.21.15.97:0/1068343276 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f66240b8730 con 0x7f66341033c0 2024-03-11T19:55:23.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:23.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:23.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:23.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:23.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:23.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:23.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:23 smithi097 ceph-mon[31898]: Deploying daemon mds.h on smithi114 2024-03-11T19:55:23.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:23 smithi097 ceph-mon[31898]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:23.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:23 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.161:6836/1600156948,v1:172.21.15.161:6837/1600156948] up:boot 2024-03-11T19:55:23.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:23 smithi097 ceph-mon[31898]: fsmap 7 up:standby 2024-03-11T19:55:23.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:23 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T19:55:23.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.466+0000 7f663a200640 1 -- 172.21.15.97:0/1068343276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f663410edd0 con 0x7f66341033c0 2024-03-11T19:55:23.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.467+0000 7f6630c00640 1 -- 172.21.15.97:0/1068343276 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6080 (secure 0 0 0) 0x7f66240bbdf0 con 0x7f66341033c0 2024-03-11T19:55:23.470 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:23.470 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24499,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-11T19:55:23.470 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 8 2024-03-11T19:55:23.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.469+0000 7f663a200640 1 -- 172.21.15.97:0/1068343276 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66180a1a30 msgr2=0x7f66180a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:23.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.469+0000 7f663a200640 1 --2- 172.21.15.97:0/1068343276 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66180a1a30 0x7f66180a3ef0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f663419a9b0 tx=0x7f661c005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:23.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.469+0000 7f663a200640 1 -- 172.21.15.97:0/1068343276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66341033c0 msgr2=0x7f66341994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:23.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.469+0000 7f663a200640 1 --2- 172.21.15.97:0/1068343276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66341033c0 0x7f66341994a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6624005ab0 tx=0x7f6624005e50 comp rx=0 tx=0).stop 2024-03-11T19:55:23.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.470+0000 7f663a200640 1 -- 172.21.15.97:0/1068343276 shutdown_connections 2024-03-11T19:55:23.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.470+0000 7f663a200640 1 --2- 172.21.15.97:0/1068343276 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66180a1a30 0x7f66180a3ef0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:23.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.470+0000 7f663a200640 1 --2- 172.21.15.97:0/1068343276 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6634104f70 0x7f66341a0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:23.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.470+0000 7f663a200640 1 --2- 172.21.15.97:0/1068343276 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66341045b0 0x7f66341999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:23.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.470+0000 7f663a200640 1 --2- 172.21.15.97:0/1068343276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66341033c0 0x7f66341994a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:23.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.470+0000 7f663a200640 1 -- 172.21.15.97:0/1068343276 >> 172.21.15.97:0/1068343276 conn(0x7f66340fe970 msgr2=0x7f6634107930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:23.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.470+0000 7f663a200640 1 -- 172.21.15.97:0/1068343276 shutdown_connections 2024-03-11T19:55:23.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:23.470+0000 7f663a200640 1 -- 172.21.15.97:0/1068343276 wait complete. 2024-03-11T19:55:23.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:23.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:23.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:23.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:23.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:23.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:23.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:23 smithi114 ceph-mon[36376]: Deploying daemon mds.h on smithi114 2024-03-11T19:55:23.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:23 smithi114 ceph-mon[36376]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:23.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:23 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.161:6836/1600156948,v1:172.21.15.161:6837/1600156948] up:boot 2024-03-11T19:55:23.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:23 smithi114 ceph-mon[36376]: fsmap 7 up:standby 2024-03-11T19:55:23.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:23 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T19:55:23.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:23.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:23.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:23.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:23.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:23.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:23.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:23 smithi161 ceph-mon[36332]: Deploying daemon mds.h on smithi114 2024-03-11T19:55:23.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:23 smithi161 ceph-mon[36332]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:23.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:23 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.161:6836/1600156948,v1:172.21.15.161:6837/1600156948] up:boot 2024-03-11T19:55:23.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:23 smithi161 ceph-mon[36332]: fsmap 7 up:standby 2024-03-11T19:55:23.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:23 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T19:55:24.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:24 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1068343276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:24.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:24 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1068343276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:24.644 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 19:55:24 smithi114 systemd[1]: Starting Ceph mds.h for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T19:55:24.670 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:24 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1068343276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:24.921 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 19:55:24 smithi114 podman[66355]: 2024-03-11 19:55:24.543559145 +0000 UTC m=+0.017239673 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:55:24.921 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 19:55:24 smithi114 podman[66355]: 2024-03-11 19:55:24.674415616 +0000 UTC m=+0.148096138 container create cbd981c5cca1f85f50cab2d02ff65d5397c4f0bbb74110e69558b41413269e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-h, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-94a5397, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:55:25.242 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 19:55:24 smithi114 podman[66355]: 2024-03-11 19:55:24.918965471 +0000 UTC m=+0.392646012 container init cbd981c5cca1f85f50cab2d02ff65d5397c4f0bbb74110e69558b41413269e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-h, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:55:25.242 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 19:55:24 smithi114 podman[66355]: 2024-03-11 19:55:24.921985755 +0000 UTC m=+0.395666282 container start cbd981c5cca1f85f50cab2d02ff65d5397c4f0bbb74110e69558b41413269e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-h, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-11T19:55:25.242 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 19:55:24 smithi114 ceph-mds[66374]: -- 172.21.15.114:0/1848213262 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c96f0181a0 con 0x55c96efef000 2024-03-11T19:55:25.242 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 19:55:24 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-h[66370]: starting mds.h at 2024-03-11T19:55:25.242 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 19:55:25 smithi114 bash[66355]: cbd981c5cca1f85f50cab2d02ff65d5397c4f0bbb74110e69558b41413269e09 2024-03-11T19:55:25.243 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 19:55:25 smithi114 systemd[1]: Started Ceph mds.h for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:55:25.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:25.576 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:25 smithi097 ceph-mon[31898]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:25.576 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:25 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:25.576 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:25 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:25.576 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:25 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:25.576 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:25 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:25.576 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:25 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:25.577 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:25 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:25.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.703+0000 7fa26d800640 1 -- 172.21.15.97:0/3828370630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2681089b0 msgr2=0x7fa268108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:25.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.703+0000 7fa26d800640 1 --2- 172.21.15.97:0/3828370630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2681089b0 0x7fa268108e30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa250009860 tx=0x7fa25002bb70 comp rx=0 tx=0).stop 2024-03-11T19:55:25.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.704+0000 7fa26d800640 1 -- 172.21.15.97:0/3828370630 shutdown_connections 2024-03-11T19:55:25.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.704+0000 7fa26d800640 1 --2- 172.21.15.97:0/3828370630 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa268109370 0x7fa26810fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:25.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.704+0000 7fa26d800640 1 --2- 172.21.15.97:0/3828370630 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa2681089b0 0x7fa268108e30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:25.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.704+0000 7fa26d800640 1 --2- 172.21.15.97:0/3828370630 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2681077c0 0x7fa268107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:25.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.704+0000 7fa26d800640 1 -- 172.21.15.97:0/3828370630 >> 172.21.15.97:0/3828370630 conn(0x7fa268076790 msgr2=0x7fa268078bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:25.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.704+0000 7fa26d800640 1 -- 172.21.15.97:0/3828370630 shutdown_connections 2024-03-11T19:55:25.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.704+0000 7fa26d800640 1 -- 172.21.15.97:0/3828370630 wait complete. 2024-03-11T19:55:25.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.705+0000 7fa26d800640 1 Processor -- start 2024-03-11T19:55:25.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.705+0000 7fa26d800640 1 -- start start 2024-03-11T19:55:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.705+0000 7fa26d800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2681077c0 0x7fa26819d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.705+0000 7fa26d800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2681089b0 0x7fa26819dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.705+0000 7fa26d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268109370 0x7fa2681a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.705+0000 7fa26d800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681a5300 con 0x7fa2681089b0 2024-03-11T19:55:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.705+0000 7fa26d800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681a5470 con 0x7fa268109370 2024-03-11T19:55:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.705+0000 7fa26d800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681a5630 con 0x7fa2681077c0 2024-03-11T19:55:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.705+0000 7fa25fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2681089b0 0x7fa26819dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.705+0000 7fa266a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2681077c0 0x7fa26819d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.705+0000 7fa25fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2681089b0 0x7fa26819dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47308/0 (socket says 172.21.15.97:47308) 2024-03-11T19:55:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.705+0000 7fa25fe00640 1 -- 172.21.15.97:0/158731438 learned_addr learned my addr 172.21.15.97:0/158731438 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.706+0000 7fa267400640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268109370 0x7fa2681a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:25.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.706+0000 7fa266a00640 1 -- 172.21.15.97:0/158731438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268109370 msgr2=0x7fa2681a4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:25.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.706+0000 7fa266a00640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268109370 0x7fa2681a4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:25.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.706+0000 7fa266a00640 1 -- 172.21.15.97:0/158731438 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2681089b0 msgr2=0x7fa26819dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:25.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.706+0000 7fa266a00640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2681089b0 0x7fa26819dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:25.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.706+0000 7fa266a00640 1 -- 172.21.15.97:0/158731438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa250009490 con 0x7fa2681077c0 2024-03-11T19:55:25.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.706+0000 7fa267400640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268109370 0x7fa2681a4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:55:25.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.706+0000 7fa266a00640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2681077c0 0x7fa26819d810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa258005fd0 tx=0x7fa25800ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:25.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.706+0000 7fa25f400640 1 -- 172.21.15.97:0/158731438 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25800ccc0 con 0x7fa2681077c0 2024-03-11T19:55:25.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.706+0000 7fa25f400640 1 -- 172.21.15.97:0/158731438 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa25800ee70 con 0x7fa2681077c0 2024-03-11T19:55:25.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.706+0000 7fa26d800640 1 -- 172.21.15.97:0/158731438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2681a5910 con 0x7fa2681077c0 2024-03-11T19:55:25.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.706+0000 7fa25f400640 1 -- 172.21.15.97:0/158731438 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258010430 con 0x7fa2681077c0 2024-03-11T19:55:25.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.707+0000 7fa26d800640 1 -- 172.21.15.97:0/158731438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2681a5e20 con 0x7fa2681077c0 2024-03-11T19:55:25.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.708+0000 7fa26d800640 1 -- 172.21.15.97:0/158731438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26810ea80 con 0x7fa2681077c0 2024-03-11T19:55:25.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.709+0000 7fa25f400640 1 -- 172.21.15.97:0/158731438 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa258003ee0 con 0x7fa2681077c0 2024-03-11T19:55:25.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.711+0000 7fa25f400640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa24c0a1ab0 0x7fa24c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:25.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.711+0000 7fa25f400640 1 -- 172.21.15.97:0/158731438 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa258014070 con 0x7fa2681077c0 2024-03-11T19:55:25.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.711+0000 7fa25fe00640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa24c0a1ab0 0x7fa24c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:25.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.711+0000 7fa25fe00640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa24c0a1ab0 0x7fa24c0a3f70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa26819ed20 tx=0x7fa2500058b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:25.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.714+0000 7fa25f400640 1 -- 172.21.15.97:0/158731438 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa2580968e0 con 0x7fa2681077c0 2024-03-11T19:55:25.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:25 smithi114 ceph-mon[36376]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:25.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:25 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:25.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:25 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:25.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:25 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:25.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:25 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:25.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:25 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:25.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:25 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:25.770 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:25 smithi161 ceph-mon[36332]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:25.771 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:25 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:25.771 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:25 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:25.771 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:25 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:25.771 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:25 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:25.771 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:25 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:25.771 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:25 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:25.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.971+0000 7fa26d800640 1 -- 172.21.15.97:0/158731438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2681131a0 con 0x7fa2681077c0 2024-03-11T19:55:25.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.972+0000 7fa25f400640 1 -- 172.21.15.97:0/158731438 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6874 (secure 0 0 0) 0x7fa258099fa0 con 0x7fa2681077c0 2024-03-11T19:55:25.975 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:25.975 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24499,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-11T19:55:25.975 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 9 2024-03-11T19:55:25.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.975+0000 7fa26d800640 1 -- 172.21.15.97:0/158731438 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa24c0a1ab0 msgr2=0x7fa24c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:25.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.975+0000 7fa26d800640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa24c0a1ab0 0x7fa24c0a3f70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa26819ed20 tx=0x7fa2500058b0 comp rx=0 tx=0).stop 2024-03-11T19:55:25.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.975+0000 7fa26d800640 1 -- 172.21.15.97:0/158731438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2681077c0 msgr2=0x7fa26819d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:25.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.975+0000 7fa26d800640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2681077c0 0x7fa26819d810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa258005fd0 tx=0x7fa25800ec20 comp rx=0 tx=0).stop 2024-03-11T19:55:25.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.975+0000 7fa26d800640 1 -- 172.21.15.97:0/158731438 shutdown_connections 2024-03-11T19:55:25.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.975+0000 7fa26d800640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa268109370 0x7fa2681a4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:25.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.975+0000 7fa26d800640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa24c0a1ab0 0x7fa24c0a3f70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:25.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.975+0000 7fa26d800640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa2681089b0 0x7fa26819dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:25.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.975+0000 7fa26d800640 1 --2- 172.21.15.97:0/158731438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2681077c0 0x7fa26819d810 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:25.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.975+0000 7fa26d800640 1 -- 172.21.15.97:0/158731438 >> 172.21.15.97:0/158731438 conn(0x7fa268076790 msgr2=0x7fa26810bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:25.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.975+0000 7fa26d800640 1 -- 172.21.15.97:0/158731438 shutdown_connections 2024-03-11T19:55:25.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:25.975+0000 7fa26d800640 1 -- 172.21.15.97:0/158731438 wait complete. 2024-03-11T19:55:26.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:26 smithi161 ceph-mon[36332]: Deploying daemon mds.i on smithi161 2024-03-11T19:55:26.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:26 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] up:boot 2024-03-11T19:55:26.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:26 smithi161 ceph-mon[36332]: fsmap 8 up:standby 2024-03-11T19:55:26.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:26 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T19:55:26.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:26 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/158731438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:26.670 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:26 smithi097 ceph-mon[31898]: Deploying daemon mds.i on smithi161 2024-03-11T19:55:26.670 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:26 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] up:boot 2024-03-11T19:55:26.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:26 smithi097 ceph-mon[31898]: fsmap 8 up:standby 2024-03-11T19:55:26.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:26 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T19:55:26.671 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:26 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/158731438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:26.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:26 smithi114 ceph-mon[36376]: Deploying daemon mds.i on smithi161 2024-03-11T19:55:26.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:26 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] up:boot 2024-03-11T19:55:26.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:26 smithi114 ceph-mon[36376]: fsmap 8 up:standby 2024-03-11T19:55:26.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:26 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T19:55:26.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:26 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/158731438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:27.590 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:27 smithi161 ceph-mon[36332]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:27.590 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 19:55:27 smithi161 systemd[1]: Starting Ceph mds.i for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T19:55:27.670 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:27 smithi097 ceph-mon[31898]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:27.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:27 smithi114 ceph-mon[36376]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:27.851 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 19:55:27 smithi161 podman[67541]: 2024-03-11 19:55:27.490031748 +0000 UTC m=+0.017999113 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:55:27.851 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 19:55:27 smithi161 podman[67541]: 2024-03-11 19:55:27.595802526 +0000 UTC m=+0.123769898 container create 2e4833f4daa648d3f915e8621d8fa3dc19e910bfeb7af0ec823850b96f63a99b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-i, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-11T19:55:27.851 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 19:55:27 smithi161 podman[67541]: 2024-03-11 19:55:27.848924324 +0000 UTC m=+0.376891674 container init 2e4833f4daa648d3f915e8621d8fa3dc19e910bfeb7af0ec823850b96f63a99b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-i, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-11T19:55:27.997 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:28.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.212+0000 7fa516000640 1 -- 172.21.15.97:0/1527006677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa510100ff0 msgr2=0x7fa510101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:28.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.212+0000 7fa516000640 1 --2- 172.21.15.97:0/1527006677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa510100ff0 0x7fa510101410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa500009900 tx=0x7fa50002bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:28.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.213+0000 7fa516000640 1 -- 172.21.15.97:0/1527006677 shutdown_connections 2024-03-11T19:55:28.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.213+0000 7fa516000640 1 --2- 172.21.15.97:0/1527006677 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa510102ba0 0x7fa510109430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:28.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.213+0000 7fa516000640 1 --2- 172.21.15.97:0/1527006677 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa5101021e0 0x7fa510102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:28.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.213+0000 7fa516000640 1 --2- 172.21.15.97:0/1527006677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa510100ff0 0x7fa510101410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:28.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.213+0000 7fa516000640 1 -- 172.21.15.97:0/1527006677 >> 172.21.15.97:0/1527006677 conn(0x7fa5100fc560 msgr2=0x7fa5100fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:28.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.213+0000 7fa516000640 1 -- 172.21.15.97:0/1527006677 shutdown_connections 2024-03-11T19:55:28.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.213+0000 7fa516000640 1 -- 172.21.15.97:0/1527006677 wait complete. 2024-03-11T19:55:28.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.214+0000 7fa516000640 1 Processor -- start 2024-03-11T19:55:28.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.214+0000 7fa516000640 1 -- start start 2024-03-11T19:55:28.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.214+0000 7fa516000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa510100ff0 0x7fa5101992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:28.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.214+0000 7fa516000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5101021e0 0x7fa510199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:28.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.214+0000 7fa516000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa510102ba0 0x7fa5101a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.214+0000 7fa516000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5101a0db0 con 0x7fa510102ba0 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.214+0000 7fa516000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5101a0f20 con 0x7fa510100ff0 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.214+0000 7fa516000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5101a10e0 con 0x7fa5101021e0 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.214+0000 7fa507e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5101021e0 0x7fa510199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.214+0000 7fa507e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5101021e0 0x7fa510199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:58752/0 (socket says 172.21.15.97:58752) 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.214+0000 7fa507e00640 1 -- 172.21.15.97:0/1888314117 learned_addr learned my addr 172.21.15.97:0/1888314117 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.214+0000 7fa514c00640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa510100ff0 0x7fa5101992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.214+0000 7fa515600640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa510102ba0 0x7fa5101a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.215+0000 7fa507e00640 1 -- 172.21.15.97:0/1888314117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa510100ff0 msgr2=0x7fa5101992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.215+0000 7fa507e00640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa510100ff0 0x7fa5101992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.215+0000 7fa507e00640 1 -- 172.21.15.97:0/1888314117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa510102ba0 msgr2=0x7fa5101a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.215+0000 7fa507e00640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa510102ba0 0x7fa5101a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.215+0000 7fa507e00640 1 -- 172.21.15.97:0/1888314117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa500009560 con 0x7fa5101021e0 2024-03-11T19:55:28.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.215+0000 7fa507e00640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5101021e0 0x7fa510199800 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800e9f0 tx=0x7fa4f800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:28.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.215+0000 7fa50e000640 1 -- 172.21.15.97:0/1888314117 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8009700 con 0x7fa5101021e0 2024-03-11T19:55:28.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.215+0000 7fa50e000640 1 -- 172.21.15.97:0/1888314117 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4f800cd80 con 0x7fa5101021e0 2024-03-11T19:55:28.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.215+0000 7fa50e000640 1 -- 172.21.15.97:0/1888314117 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8018920 con 0x7fa5101021e0 2024-03-11T19:55:28.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.215+0000 7fa516000640 1 -- 172.21.15.97:0/1888314117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5101a13c0 con 0x7fa5101021e0 2024-03-11T19:55:28.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.216+0000 7fa516000640 1 -- 172.21.15.97:0/1888314117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5101a1920 con 0x7fa5101021e0 2024-03-11T19:55:28.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.217+0000 7fa516000640 1 -- 172.21.15.97:0/1888314117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51010ca30 con 0x7fa5101021e0 2024-03-11T19:55:28.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.218+0000 7fa50e000640 1 -- 172.21.15.97:0/1888314117 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa4f8018a80 con 0x7fa5101021e0 2024-03-11T19:55:28.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.220+0000 7fa50e000640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa4f40a1ab0 0x7fa4f40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:28.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.220+0000 7fa50e000640 1 -- 172.21.15.97:0/1888314117 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa4f8014070 con 0x7fa5101021e0 2024-03-11T19:55:28.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.220+0000 7fa514c00640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa4f40a1ab0 0x7fa4f40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:28.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.221+0000 7fa514c00640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa4f40a1ab0 0x7fa4f40a3f70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa50002c240 tx=0x7fa50002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:28.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.224+0000 7fa50e000640 1 -- 172.21.15.97:0/1888314117 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa4f8052d20 con 0x7fa5101021e0 2024-03-11T19:55:28.266 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 19:55:27 smithi161 podman[67541]: 2024-03-11 19:55:27.851568821 +0000 UTC m=+0.379536175 container start 2e4833f4daa648d3f915e8621d8fa3dc19e910bfeb7af0ec823850b96f63a99b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-i, RELEASE=squid-94a5397, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-11T19:55:28.266 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 19:55:27 smithi161 ceph-mds[67560]: -- 172.21.15.161:0/1496613397 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c256ff41a0 con 0x55c256fcb000 2024-03-11T19:55:28.266 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 19:55:27 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-i[67556]: starting mds.i at 2024-03-11T19:55:28.266 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 19:55:27 smithi161 bash[67541]: 2e4833f4daa648d3f915e8621d8fa3dc19e910bfeb7af0ec823850b96f63a99b 2024-03-11T19:55:28.266 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 19:55:27 smithi161 systemd[1]: Started Ceph mds.i for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:55:28.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.462+0000 7fa516000640 1 -- 172.21.15.97:0/1888314117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5101a1cf0 con 0x7fa5101021e0 2024-03-11T19:55:28.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.463+0000 7fa50e000640 1 -- 172.21.15.97:0/1888314117 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7667 (secure 0 0 0) 0x7fa4f80969b0 con 0x7fa5101021e0 2024-03-11T19:55:28.466 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:28.466 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24499,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-11T19:55:28.466 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 10 2024-03-11T19:55:28.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.466+0000 7fa516000640 1 -- 172.21.15.97:0/1888314117 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa4f40a1ab0 msgr2=0x7fa4f40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:28.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.466+0000 7fa516000640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa4f40a1ab0 0x7fa4f40a3f70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa50002c240 tx=0x7fa50002c7e0 comp rx=0 tx=0).stop 2024-03-11T19:55:28.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.466+0000 7fa516000640 1 -- 172.21.15.97:0/1888314117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5101021e0 msgr2=0x7fa510199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:28.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.466+0000 7fa516000640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5101021e0 0x7fa510199800 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800e9f0 tx=0x7fa4f800eeb0 comp rx=0 tx=0).stop 2024-03-11T19:55:28.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.466+0000 7fa516000640 1 -- 172.21.15.97:0/1888314117 shutdown_connections 2024-03-11T19:55:28.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.466+0000 7fa516000640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa4f40a1ab0 0x7fa4f40a3f70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:28.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.466+0000 7fa516000640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa510102ba0 0x7fa5101a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:28.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.466+0000 7fa516000640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5101021e0 0x7fa510199800 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:28.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.466+0000 7fa516000640 1 --2- 172.21.15.97:0/1888314117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa510100ff0 0x7fa5101992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:28.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.466+0000 7fa516000640 1 -- 172.21.15.97:0/1888314117 >> 172.21.15.97:0/1888314117 conn(0x7fa5100fc560 msgr2=0x7fa5100733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:28.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.466+0000 7fa516000640 1 -- 172.21.15.97:0/1888314117 shutdown_connections 2024-03-11T19:55:28.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:28.466+0000 7fa516000640 1 -- 172.21.15.97:0/1888314117 wait complete. 2024-03-11T19:55:29.232 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:29.281 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:29.282 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:29.282 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:29.282 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:29.282 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:29.282 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:29 smithi097 ceph-mon[31898]: Deploying daemon mds.g on smithi097 2024-03-11T19:55:29.282 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:29 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] up:boot 2024-03-11T19:55:29.282 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:29 smithi097 ceph-mon[31898]: fsmap 9 up:standby 2024-03-11T19:55:29.282 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:55:29.282 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:29 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:55:29.282 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:29 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1888314117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:29.283 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:29 smithi097 ceph-mon[31898]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:29.288 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:29.288 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:29.288 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:29.288 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:29.288 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:29.289 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:29.289 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:29 smithi161 ceph-mon[36332]: Deploying daemon mds.g on smithi097 2024-03-11T19:55:29.289 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:29 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] up:boot 2024-03-11T19:55:29.289 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:29 smithi161 ceph-mon[36332]: fsmap 9 up:standby 2024-03-11T19:55:29.289 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:55:29.289 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:29 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:55:29.289 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:29 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1888314117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:29.289 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:29 smithi161 ceph-mon[36332]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:29.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:29.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:29.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:29.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:29.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:29.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:29.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:29 smithi114 ceph-mon[36376]: Deploying daemon mds.g on smithi097 2024-03-11T19:55:29.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:29 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] up:boot 2024-03-11T19:55:29.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:29 smithi114 ceph-mon[36376]: fsmap 9 up:standby 2024-03-11T19:55:29.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:55:29.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:29 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:55:29.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:29 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1888314117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:29.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:29 smithi114 ceph-mon[36376]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:30.490 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:30.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.920+0000 7f3972400640 1 -- 172.21.15.97:0/2832830101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c101a50 msgr2=0x7f396c103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:30.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.920+0000 7f3972400640 1 --2- 172.21.15.97:0/2832830101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c101a50 0x7f396c103e40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f395c009900 tx=0x7f395c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:30.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.920+0000 7f3972400640 1 -- 172.21.15.97:0/2832830101 shutdown_connections 2024-03-11T19:55:30.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.920+0000 7f3972400640 1 --2- 172.21.15.97:0/2832830101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f396c106cb0 0x7f396c109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:30.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.920+0000 7f3972400640 1 --2- 172.21.15.97:0/2832830101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c104380 0x7f396c106770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:30.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.920+0000 7f3972400640 1 --2- 172.21.15.97:0/2832830101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c101a50 0x7f396c103e40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:30.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.920+0000 7f3972400640 1 -- 172.21.15.97:0/2832830101 >> 172.21.15.97:0/2832830101 conn(0x7f396c0fb460 msgr2=0x7f396c0fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:30.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.921+0000 7f3972400640 1 -- 172.21.15.97:0/2832830101 shutdown_connections 2024-03-11T19:55:30.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.921+0000 7f3972400640 1 -- 172.21.15.97:0/2832830101 wait complete. 2024-03-11T19:55:30.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.922+0000 7f3972400640 1 Processor -- start 2024-03-11T19:55:30.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.922+0000 7f3972400640 1 -- start start 2024-03-11T19:55:30.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.922+0000 7f3972400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f396c101a50 0x7f396c1013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:30.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.922+0000 7f3972400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c104380 0x7f396c0ffa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:30.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.922+0000 7f3972400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c106cb0 0x7f396c0fff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:30.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.922+0000 7f3972400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c1006a0 con 0x7f396c106cb0 2024-03-11T19:55:30.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.922+0000 7f3972400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c100810 con 0x7f396c101a50 2024-03-11T19:55:30.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.922+0000 7f3972400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c1a1050 con 0x7f396c104380 2024-03-11T19:55:30.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.923+0000 7f396be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c104380 0x7f396c0ffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:30.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.923+0000 7f396be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c104380 0x7f396c0ffa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:58770/0 (socket says 172.21.15.97:58770) 2024-03-11T19:55:30.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.923+0000 7f396be00640 1 -- 172.21.15.97:0/3604944032 learned_addr learned my addr 172.21.15.97:0/3604944032 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:30.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.923+0000 7f3971000640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f396c101a50 0x7f396c1013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:30.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.923+0000 7f3971a00640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c106cb0 0x7f396c0fff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:30.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.923+0000 7f396be00640 1 -- 172.21.15.97:0/3604944032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f396c101a50 msgr2=0x7f396c1013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:30.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.923+0000 7f396be00640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f396c101a50 0x7f396c1013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:30.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.923+0000 7f396be00640 1 -- 172.21.15.97:0/3604944032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c106cb0 msgr2=0x7f396c0fff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:30.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.923+0000 7f396be00640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c106cb0 0x7f396c0fff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:30.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.923+0000 7f396be00640 1 -- 172.21.15.97:0/3604944032 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c009560 con 0x7f396c104380 2024-03-11T19:55:30.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.924+0000 7f396be00640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c104380 0x7f396c0ffa00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f395800c970 tx=0x7f395800ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:30.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.924+0000 7f3969600640 1 -- 172.21.15.97:0/3604944032 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958004440 con 0x7f396c104380 2024-03-11T19:55:30.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.924+0000 7f3969600640 1 -- 172.21.15.97:0/3604944032 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3958007d40 con 0x7f396c104380 2024-03-11T19:55:30.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.924+0000 7f3969600640 1 -- 172.21.15.97:0/3604944032 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395800f450 con 0x7f396c104380 2024-03-11T19:55:30.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.924+0000 7f3972400640 1 -- 172.21.15.97:0/3604944032 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396c1a1300 con 0x7f396c104380 2024-03-11T19:55:30.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.924+0000 7f3972400640 1 -- 172.21.15.97:0/3604944032 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c1a1810 con 0x7f396c104380 2024-03-11T19:55:30.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.926+0000 7f3972400640 1 -- 172.21.15.97:0/3604944032 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396c0fcf50 con 0x7f396c104380 2024-03-11T19:55:30.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.927+0000 7f3969600640 1 -- 172.21.15.97:0/3604944032 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3958003680 con 0x7f396c104380 2024-03-11T19:55:30.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.929+0000 7f3969600640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39500a1a30 0x7f39500a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:30.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.929+0000 7f3969600640 1 -- 172.21.15.97:0/3604944032 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f39580ccca0 con 0x7f396c104380 2024-03-11T19:55:30.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.929+0000 7f3971000640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39500a1a30 0x7f39500a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:30.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.930+0000 7f3971000640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39500a1a30 0x7f39500a3ef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f395c000c00 tx=0x7f395c02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:30.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:30.933+0000 7f3969600640 1 -- 172.21.15.97:0/3604944032 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3958096350 con 0x7f396c104380 2024-03-11T19:55:31.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.175+0000 7f3972400640 1 -- 172.21.15.97:0/3604944032 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f396c10cbc0 con 0x7f396c104380 2024-03-11T19:55:31.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.176+0000 7f3969600640 1 -- 172.21.15.97:0/3604944032 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7667 (secure 0 0 0) 0x7f3958099a10 con 0x7f396c104380 2024-03-11T19:55:31.179 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:31.179 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24499,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-11T19:55:31.180 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 10 2024-03-11T19:55:31.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.178+0000 7f3972400640 1 -- 172.21.15.97:0/3604944032 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39500a1a30 msgr2=0x7f39500a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:31.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.178+0000 7f3972400640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39500a1a30 0x7f39500a3ef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f395c000c00 tx=0x7f395c02c7e0 comp rx=0 tx=0).stop 2024-03-11T19:55:31.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.178+0000 7f3972400640 1 -- 172.21.15.97:0/3604944032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c104380 msgr2=0x7f396c0ffa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:31.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.178+0000 7f3972400640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c104380 0x7f396c0ffa00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f395800c970 tx=0x7f395800ce30 comp rx=0 tx=0).stop 2024-03-11T19:55:31.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.178+0000 7f3972400640 1 -- 172.21.15.97:0/3604944032 shutdown_connections 2024-03-11T19:55:31.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.178+0000 7f3972400640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39500a1a30 0x7f39500a3ef0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:31.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.178+0000 7f3972400640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f396c106cb0 0x7f396c0fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:31.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.178+0000 7f3972400640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f396c104380 0x7f396c0ffa00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:31.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.178+0000 7f3972400640 1 --2- 172.21.15.97:0/3604944032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f396c101a50 0x7f396c1013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:31.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.178+0000 7f3972400640 1 -- 172.21.15.97:0/3604944032 >> 172.21.15.97:0/3604944032 conn(0x7f396c0fb460 msgr2=0x7f396c1023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:31.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.178+0000 7f3972400640 1 -- 172.21.15.97:0/3604944032 shutdown_connections 2024-03-11T19:55:31.183 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:31.178+0000 7f3972400640 1 -- 172.21.15.97:0/3604944032 wait complete. 2024-03-11T19:55:31.824 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:31 smithi097 ceph-mon[31898]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:31.825 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:31 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3604944032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:31.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:31 smithi114 ceph-mon[36376]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:31.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:31 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3604944032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:32.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:31 smithi161 ceph-mon[36332]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:32.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:31 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3604944032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:32.170 INFO:journalctl@ceph.mds.g.smithi097.stdout:Mar 11 19:55:31 smithi097 systemd[1]: Started Ceph mds.g for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:55:33.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:33.295 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:33 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:33.295 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:33 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:33.295 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:33 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:33.295 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:33 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:33.296 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:33 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:33.296 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:33 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:33.296 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:33 smithi097 ceph-mon[31898]: Deploying daemon mds.a on smithi097 2024-03-11T19:55:33.296 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:33 smithi097 ceph-mon[31898]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:33.296 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:33 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.97:6838/2355075568,v1:172.21.15.97:6839/2355075568] up:boot 2024-03-11T19:55:33.296 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:33 smithi097 ceph-mon[31898]: fsmap 10 up:standby 2024-03-11T19:55:33.296 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:33 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T19:55:33.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.421+0000 7f4acd800640 1 -- 172.21.15.97:0/1559071584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac8077720 msgr2=0x7f4ac8077fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:33.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.421+0000 7f4acd800640 1 --2- 172.21.15.97:0/1559071584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac8077720 0x7f4ac8077fc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0009ca0 tx=0x7f4ac002c100 comp rx=0 tx=0).stop 2024-03-11T19:55:33.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.421+0000 7f4acd800640 1 -- 172.21.15.97:0/1559071584 shutdown_connections 2024-03-11T19:55:33.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.421+0000 7f4acd800640 1 --2- 172.21.15.97:0/1559071584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac8077720 0x7f4ac8077fc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:33.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.421+0000 7f4acd800640 1 --2- 172.21.15.97:0/1559071584 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ac8076d80 0x7f4ac80771e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:33.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.421+0000 7f4acd800640 1 --2- 172.21.15.97:0/1559071584 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ac8075b40 0x7f4ac8075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:33.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.421+0000 7f4acd800640 1 -- 172.21.15.97:0/1559071584 >> 172.21.15.97:0/1559071584 conn(0x7f4ac80fe930 msgr2=0x7f4ac8100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:33.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.421+0000 7f4acd800640 1 -- 172.21.15.97:0/1559071584 shutdown_connections 2024-03-11T19:55:33.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.421+0000 7f4acd800640 1 -- 172.21.15.97:0/1559071584 wait complete. 2024-03-11T19:55:33.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.422+0000 7f4acd800640 1 Processor -- start 2024-03-11T19:55:33.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.422+0000 7f4acd800640 1 -- start start 2024-03-11T19:55:33.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.422+0000 7f4acd800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ac8075b40 0x7f4ac819d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.422+0000 7f4acd800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac8076d80 0x7f4ac819dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.422+0000 7f4acd800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ac8077720 0x7f4ac81a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.422+0000 7f4acd800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac81a5300 con 0x7f4ac8075b40 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.422+0000 7f4acd800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac81a5470 con 0x7f4ac8076d80 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.422+0000 7f4acd800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac81a5630 con 0x7f4ac8077720 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.423+0000 7f4ac6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ac8075b40 0x7f4ac819d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.423+0000 7f4abfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac8076d80 0x7f4ac819dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.423+0000 7f4ac6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ac8075b40 0x7f4ac819d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47384/0 (socket says 172.21.15.97:47384) 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.423+0000 7f4abfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac8076d80 0x7f4ac819dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:45244/0 (socket says 172.21.15.97:45244) 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.423+0000 7f4ac6a00640 1 -- 172.21.15.97:0/2618557770 learned_addr learned my addr 172.21.15.97:0/2618557770 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.423+0000 7f4ac7400640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ac8077720 0x7f4ac81a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.423+0000 7f4ac6a00640 1 -- 172.21.15.97:0/2618557770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ac8077720 msgr2=0x7f4ac81a4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.423+0000 7f4ac6a00640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ac8077720 0x7f4ac81a4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.423+0000 7f4ac6a00640 1 -- 172.21.15.97:0/2618557770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac8076d80 msgr2=0x7f4ac819dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:33.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.423+0000 7f4ac6a00640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac8076d80 0x7f4ac819dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:33.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.423+0000 7f4ac6a00640 1 -- 172.21.15.97:0/2618557770 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac0009900 con 0x7f4ac8075b40 2024-03-11T19:55:33.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.423+0000 7f4abfe00640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac8076d80 0x7f4ac819dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:33.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.423+0000 7f4ac6a00640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ac8075b40 0x7f4ac819d810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4ab800e9f0 tx=0x7f4ab800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:33.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.424+0000 7f4abf400640 1 -- 172.21.15.97:0/2618557770 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8009700 con 0x7f4ac8075b40 2024-03-11T19:55:33.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.424+0000 7f4abf400640 1 -- 172.21.15.97:0/2618557770 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ab800cd80 con 0x7f4ac8075b40 2024-03-11T19:55:33.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.424+0000 7f4acd800640 1 -- 172.21.15.97:0/2618557770 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac81a5910 con 0x7f4ac8075b40 2024-03-11T19:55:33.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.424+0000 7f4abf400640 1 -- 172.21.15.97:0/2618557770 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8018920 con 0x7f4ac8075b40 2024-03-11T19:55:33.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.424+0000 7f4acd800640 1 -- 172.21.15.97:0/2618557770 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac81a5e20 con 0x7f4ac8075b40 2024-03-11T19:55:33.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.425+0000 7f4acd800640 1 -- 172.21.15.97:0/2618557770 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac810fec0 con 0x7f4ac8075b40 2024-03-11T19:55:33.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.427+0000 7f4abf400640 1 -- 172.21.15.97:0/2618557770 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4ab8018a80 con 0x7f4ac8075b40 2024-03-11T19:55:33.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.428+0000 7f4abf400640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4aac0a1a30 0x7f4aac0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:33.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.428+0000 7f4abf400640 1 -- 172.21.15.97:0/2618557770 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4ab8014070 con 0x7f4ac8075b40 2024-03-11T19:55:33.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.429+0000 7f4abfe00640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4aac0a1a30 0x7f4aac0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:33.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.429+0000 7f4abfe00640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4aac0a1a30 0x7f4aac0a3ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4ac819ed20 tx=0x7f4ab0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:33.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.433+0000 7f4abf400640 1 -- 172.21.15.97:0/2618557770 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4ab8052d20 con 0x7f4ac8075b40 2024-03-11T19:55:33.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:33 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:33.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:33 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:33.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:33 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:33.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:33 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:33.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:33 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:33.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:33 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:33.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:33 smithi114 ceph-mon[36376]: Deploying daemon mds.a on smithi097 2024-03-11T19:55:33.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:33 smithi114 ceph-mon[36376]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:33.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:33 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.97:6838/2355075568,v1:172.21.15.97:6839/2355075568] up:boot 2024-03-11T19:55:33.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:33 smithi114 ceph-mon[36376]: fsmap 10 up:standby 2024-03-11T19:55:33.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:33 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T19:55:33.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:33 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:33.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:33 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:33.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:33 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:33.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:33 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:33.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:33 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:33.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:33 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:33.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:33 smithi161 ceph-mon[36332]: Deploying daemon mds.a on smithi097 2024-03-11T19:55:33.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:33 smithi161 ceph-mon[36332]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:33.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:33 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.97:6838/2355075568,v1:172.21.15.97:6839/2355075568] up:boot 2024-03-11T19:55:33.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:33 smithi161 ceph-mon[36332]: fsmap 10 up:standby 2024-03-11T19:55:33.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:33 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T19:55:33.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.675+0000 7f4acd800640 1 -- 172.21.15.97:0/2618557770 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ac802cee0 con 0x7f4ac8075b40 2024-03-11T19:55:33.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.676+0000 7f4abf400640 1 -- 172.21.15.97:0/2618557770 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8459 (secure 0 0 0) 0x7f4ab8096a30 con 0x7f4ac8075b40 2024-03-11T19:55:33.680 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:33.680 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24499,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-11T19:55:33.680 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 11 2024-03-11T19:55:33.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.679+0000 7f4acd800640 1 -- 172.21.15.97:0/2618557770 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4aac0a1a30 msgr2=0x7f4aac0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:33.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.679+0000 7f4acd800640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4aac0a1a30 0x7f4aac0a3ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4ac819ed20 tx=0x7f4ab0005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:33.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.679+0000 7f4acd800640 1 -- 172.21.15.97:0/2618557770 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ac8075b40 msgr2=0x7f4ac819d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:33.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.679+0000 7f4acd800640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ac8075b40 0x7f4ac819d810 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4ab800e9f0 tx=0x7f4ab800eeb0 comp rx=0 tx=0).stop 2024-03-11T19:55:33.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.679+0000 7f4acd800640 1 -- 172.21.15.97:0/2618557770 shutdown_connections 2024-03-11T19:55:33.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.679+0000 7f4acd800640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ac8077720 0x7f4ac81a4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:33.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.679+0000 7f4acd800640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ac8076d80 0x7f4ac819dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:33.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.679+0000 7f4acd800640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4aac0a1a30 0x7f4aac0a3ef0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:33.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.679+0000 7f4acd800640 1 --2- 172.21.15.97:0/2618557770 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ac8075b40 0x7f4ac819d810 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:33.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.679+0000 7f4acd800640 1 -- 172.21.15.97:0/2618557770 >> 172.21.15.97:0/2618557770 conn(0x7f4ac80fe930 msgr2=0x7f4ac8100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:33.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.679+0000 7f4acd800640 1 -- 172.21.15.97:0/2618557770 shutdown_connections 2024-03-11T19:55:33.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:33.679+0000 7f4acd800640 1 -- 172.21.15.97:0/2618557770 wait complete. 2024-03-11T19:55:34.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:34 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2618557770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:34.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:34 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2618557770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:34.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:34 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2618557770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:35.352 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:35 smithi097 ceph-mon[31898]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:35.393 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:35 smithi114 ceph-mon[36376]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:35.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:35 smithi161 ceph-mon[36332]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:35.670 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 19:55:35 smithi097 systemd[1]: Starting Ceph mds.a for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T19:55:35.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:35.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.911+0000 7f2d10600640 1 -- 172.21.15.97:0/1222090551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d08073b80 msgr2=0x7f2d08073fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:35.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.911+0000 7f2d10600640 1 --2- 172.21.15.97:0/1222090551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d08073b80 0x7f2d08073fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4009900 tx=0x7f2cf402bcc0 comp rx=0 tx=0).stop 2024-03-11T19:55:35.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.911+0000 7f2d10600640 1 -- 172.21.15.97:0/1222090551 shutdown_connections 2024-03-11T19:55:35.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.911+0000 7f2d10600640 1 --2- 172.21.15.97:0/1222090551 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d08109050 0x7f2d0810b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:35.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.911+0000 7f2d10600640 1 --2- 172.21.15.97:0/1222090551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d08073b80 0x7f2d08073fe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:35.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.911+0000 7f2d10600640 1 --2- 172.21.15.97:0/1222090551 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d08075160 0x7f2d080735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:35.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.911+0000 7f2d10600640 1 -- 172.21.15.97:0/1222090551 >> 172.21.15.97:0/1222090551 conn(0x7f2d080fc730 msgr2=0x7f2d080feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:35.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.911+0000 7f2d10600640 1 -- 172.21.15.97:0/1222090551 shutdown_connections 2024-03-11T19:55:35.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.911+0000 7f2d10600640 1 -- 172.21.15.97:0/1222090551 wait complete. 2024-03-11T19:55:35.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.912+0000 7f2d10600640 1 Processor -- start 2024-03-11T19:55:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.912+0000 7f2d10600640 1 -- start start 2024-03-11T19:55:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d10600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d08073b80 0x7f2d08199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d10600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d08075160 0x7f2d08199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d10600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d08109050 0x7f2d081a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d10600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d081a0f10 con 0x7f2d08073b80 2024-03-11T19:55:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d10600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d081a1080 con 0x7f2d08075160 2024-03-11T19:55:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d10600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d081a1240 con 0x7f2d08109050 2024-03-11T19:55:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d0dc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d08073b80 0x7f2d08199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d0dc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d08073b80 0x7f2d08199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52100/0 (socket says 172.21.15.97:52100) 2024-03-11T19:55:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d0dc00640 1 -- 172.21.15.97:0/441911026 learned_addr learned my addr 172.21.15.97:0/441911026 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d0d200640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d08075160 0x7f2d08199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d0e600640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d08109050 0x7f2d081a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d0d200640 1 -- 172.21.15.97:0/441911026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d08109050 msgr2=0x7f2d081a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:35.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d0d200640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d08109050 0x7f2d081a09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:35.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d0d200640 1 -- 172.21.15.97:0/441911026 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d08073b80 msgr2=0x7f2d08199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:35.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d0d200640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d08073b80 0x7f2d08199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:35.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d0d200640 1 -- 172.21.15.97:0/441911026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf4009560 con 0x7f2d08075160 2024-03-11T19:55:35.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.913+0000 7f2d0dc00640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d08073b80 0x7f2d08199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:35.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.914+0000 7f2d0d200640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d08075160 0x7f2d08199960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2cf402c200 tx=0x7f2cf4005d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:35.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.914+0000 7f2d06000640 1 -- 172.21.15.97:0/441911026 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf402dc10 con 0x7f2d08075160 2024-03-11T19:55:35.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.914+0000 7f2d06000640 1 -- 172.21.15.97:0/441911026 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cf402dd70 con 0x7f2d08075160 2024-03-11T19:55:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.914+0000 7f2d06000640 1 -- 172.21.15.97:0/441911026 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4003bf0 con 0x7f2d08075160 2024-03-11T19:55:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.914+0000 7f2d10600640 1 -- 172.21.15.97:0/441911026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d081a14c0 con 0x7f2d08075160 2024-03-11T19:55:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.914+0000 7f2d10600640 1 -- 172.21.15.97:0/441911026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d081a1950 con 0x7f2d08075160 2024-03-11T19:55:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.916+0000 7f2d10600640 1 -- 172.21.15.97:0/441911026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d08107e10 con 0x7f2d08075160 2024-03-11T19:55:35.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.919+0000 7f2d06000640 1 -- 172.21.15.97:0/441911026 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2cf4003d50 con 0x7f2d08075160 2024-03-11T19:55:35.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.920+0000 7f2d06000640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2cec0a1ab0 0x7f2cec0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:35.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.920+0000 7f2d06000640 1 -- 172.21.15.97:0/441911026 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2cf40ef480 con 0x7f2d08075160 2024-03-11T19:55:35.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.920+0000 7f2d0dc00640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2cec0a1ab0 0x7f2cec0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:35.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.921+0000 7f2d0dc00640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2cec0a1ab0 0x7f2cec0a3f70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80049d0 tx=0x7f2cf8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:35.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:35.924+0000 7f2d06000640 1 -- 172.21.15.97:0/441911026 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2cf40b8bc0 con 0x7f2d08075160 2024-03-11T19:55:36.024 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 19:55:35 smithi097 podman[73729]: 2024-03-11 19:55:35.630103083 +0000 UTC m=+0.038148895 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:55:36.024 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 19:55:35 smithi097 podman[73729]: 2024-03-11 19:55:35.783930539 +0000 UTC m=+0.191976355 container create f4167ce20b3adef6134dea02ab9b4bbb3bde611795f344af866a0d11092717a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-a, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:55:36.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.168+0000 7f2d10600640 1 -- 172.21.15.97:0/441911026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d0810ee20 con 0x7f2d08075160 2024-03-11T19:55:36.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.168+0000 7f2d06000640 1 -- 172.21.15.97:0/441911026 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9251 (secure 0 0 0) 0x7f2cf40bc280 con 0x7f2d08075160 2024-03-11T19:55:36.172 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:36.172 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24499,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6841/2531110572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2531110572},{"type":"v1","addr":"172.21.15.97:6841","nonce":2531110572}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[]} 2024-03-11T19:55:36.172 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 12 2024-03-11T19:55:36.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.171+0000 7f2d10600640 1 -- 172.21.15.97:0/441911026 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2cec0a1ab0 msgr2=0x7f2cec0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:36.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.171+0000 7f2d10600640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2cec0a1ab0 0x7f2cec0a3f70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80049d0 tx=0x7f2cf8009500 comp rx=0 tx=0).stop 2024-03-11T19:55:36.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.171+0000 7f2d10600640 1 -- 172.21.15.97:0/441911026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d08075160 msgr2=0x7f2d08199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:36.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.171+0000 7f2d10600640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d08075160 0x7f2d08199960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2cf402c200 tx=0x7f2cf4005d70 comp rx=0 tx=0).stop 2024-03-11T19:55:36.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.172+0000 7f2d10600640 1 -- 172.21.15.97:0/441911026 shutdown_connections 2024-03-11T19:55:36.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.172+0000 7f2d10600640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d08109050 0x7f2d081a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:36.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.172+0000 7f2d10600640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d08075160 0x7f2d08199960 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:36.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.172+0000 7f2d10600640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2cec0a1ab0 0x7f2cec0a3f70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:36.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.172+0000 7f2d10600640 1 --2- 172.21.15.97:0/441911026 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2d08073b80 0x7f2d08199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:36.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.172+0000 7f2d10600640 1 -- 172.21.15.97:0/441911026 >> 172.21.15.97:0/441911026 conn(0x7f2d080fc730 msgr2=0x7f2d08104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:36.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.172+0000 7f2d10600640 1 -- 172.21.15.97:0/441911026 shutdown_connections 2024-03-11T19:55:36.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:36.172+0000 7f2d10600640 1 -- 172.21.15.97:0/441911026 wait complete. 2024-03-11T19:55:36.421 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 19:55:36 smithi097 podman[73729]: 2024-03-11 19:55:36.021421183 +0000 UTC m=+0.429467076 container init f4167ce20b3adef6134dea02ab9b4bbb3bde611795f344af866a0d11092717a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:55:36.421 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 19:55:36 smithi097 podman[73729]: 2024-03-11 19:55:36.032285946 +0000 UTC m=+0.440331784 container start f4167ce20b3adef6134dea02ab9b4bbb3bde611795f344af866a0d11092717a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-a, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T19:55:36.421 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 19:55:36 smithi097 ceph-mds[73787]: -- 172.21.15.97:0/1197266582 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5592b03ba1a0 con 0x5592b0390800 2024-03-11T19:55:36.421 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 19:55:36 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-a[73765]: starting mds.a at 2024-03-11T19:55:36.421 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 19:55:36 smithi097 bash[73729]: f4167ce20b3adef6134dea02ab9b4bbb3bde611795f344af866a0d11092717a5 2024-03-11T19:55:36.421 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 19:55:36 smithi097 systemd[1]: Started Ceph mds.a for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:55:37.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:37 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] up:boot 2024-03-11T19:55:37.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:37 smithi114 ceph-mon[36376]: fsmap 11 up:standby 2024-03-11T19:55:37.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:37 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T19:55:37.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:37 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/441911026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:37.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:37 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:37.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:37 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:37.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:37 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:37.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:37 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:37 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:37 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:37 smithi114 ceph-mon[36376]: Deploying daemon mds.b on smithi114 2024-03-11T19:55:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:37 smithi114 ceph-mon[36376]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:37.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:37 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] up:boot 2024-03-11T19:55:37.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:37 smithi097 ceph-mon[31898]: fsmap 11 up:standby 2024-03-11T19:55:37.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:37 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T19:55:37.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:37 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/441911026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:37.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:37 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:37.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:37 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:37.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:37 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:37.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:37 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:37.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:37 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:37.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:37 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:37.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:37 smithi097 ceph-mon[31898]: Deploying daemon mds.b on smithi114 2024-03-11T19:55:37.421 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:37 smithi097 ceph-mon[31898]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:37.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:37 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] up:boot 2024-03-11T19:55:37.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:37 smithi161 ceph-mon[36332]: fsmap 11 up:standby 2024-03-11T19:55:37.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:37 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T19:55:37.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:37 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/441911026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:37.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:37 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:37.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:37 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:37.539 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:37 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:37.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:37 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T19:55:37.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:37 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T19:55:37.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:37 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:37.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:37 smithi161 ceph-mon[36332]: Deploying daemon mds.b on smithi114 2024-03-11T19:55:37.540 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:37 smithi161 ceph-mon[36332]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:38.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.407+0000 7ff775a00640 1 -- 172.21.15.97:0/2973386964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7701045b0 msgr2=0x7ff770104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.407+0000 7ff775a00640 1 --2- 172.21.15.97:0/2973386964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7701045b0 0x7ff770104a30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff758009900 tx=0x7ff75802bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.407+0000 7ff775a00640 1 -- 172.21.15.97:0/2973386964 shutdown_connections 2024-03-11T19:55:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.407+0000 7ff775a00640 1 --2- 172.21.15.97:0/2973386964 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff770104f70 0x7ff77010b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.407+0000 7ff775a00640 1 --2- 172.21.15.97:0/2973386964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7701045b0 0x7ff770104a30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.407+0000 7ff775a00640 1 --2- 172.21.15.97:0/2973386964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7701033c0 0x7ff7701037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.407+0000 7ff775a00640 1 -- 172.21.15.97:0/2973386964 >> 172.21.15.97:0/2973386964 conn(0x7ff7700fe970 msgr2=0x7ff770100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.408+0000 7ff775a00640 1 -- 172.21.15.97:0/2973386964 shutdown_connections 2024-03-11T19:55:38.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.408+0000 7ff775a00640 1 -- 172.21.15.97:0/2973386964 wait complete. 2024-03-11T19:55:38.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.408+0000 7ff775a00640 1 Processor -- start 2024-03-11T19:55:38.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.409+0000 7ff775a00640 1 -- start start 2024-03-11T19:55:38.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.409+0000 7ff775a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7701033c0 0x7ff770199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:38.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.409+0000 7ff775a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7701045b0 0x7ff770199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:38.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.409+0000 7ff775a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff770104f70 0x7ff7701a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:38.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.409+0000 7ff775a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7701a0f20 con 0x7ff7701033c0 2024-03-11T19:55:38.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.409+0000 7ff775a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7701a1090 con 0x7ff7701045b0 2024-03-11T19:55:38.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.409+0000 7ff775a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7701a1250 con 0x7ff770104f70 2024-03-11T19:55:38.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.409+0000 7ff76ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7701033c0 0x7ff770199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:38.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.409+0000 7ff76ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7701033c0 0x7ff770199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52128/0 (socket says 172.21.15.97:52128) 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.409+0000 7ff76ea00640 1 -- 172.21.15.97:0/928831824 learned_addr learned my addr 172.21.15.97:0/928831824 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.409+0000 7ff767e00640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7701045b0 0x7ff770199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.409+0000 7ff76f400640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff770104f70 0x7ff7701a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.410+0000 7ff76ea00640 1 -- 172.21.15.97:0/928831824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff770104f70 msgr2=0x7ff7701a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.410+0000 7ff76ea00640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff770104f70 0x7ff7701a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.410+0000 7ff76ea00640 1 -- 172.21.15.97:0/928831824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7701045b0 msgr2=0x7ff770199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.410+0000 7ff76ea00640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7701045b0 0x7ff770199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.410+0000 7ff76ea00640 1 -- 172.21.15.97:0/928831824 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff758009560 con 0x7ff7701033c0 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.410+0000 7ff767e00640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7701045b0 0x7ff770199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.410+0000 7ff76ea00640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7701033c0 0x7ff770199430 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff76000e9c0 tx=0x7ff76000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.410+0000 7ff767400640 1 -- 172.21.15.97:0/928831824 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760009700 con 0x7ff7701033c0 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.410+0000 7ff775a00640 1 -- 172.21.15.97:0/928831824 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7701a1530 con 0x7ff7701033c0 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.410+0000 7ff767400640 1 -- 172.21.15.97:0/928831824 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff76000cd80 con 0x7ff7701033c0 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.410+0000 7ff767400640 1 -- 172.21.15.97:0/928831824 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760010430 con 0x7ff7701033c0 2024-03-11T19:55:38.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.410+0000 7ff775a00640 1 -- 172.21.15.97:0/928831824 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7701a1a10 con 0x7ff7701033c0 2024-03-11T19:55:38.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.411+0000 7ff775a00640 1 -- 172.21.15.97:0/928831824 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77010a700 con 0x7ff7701033c0 2024-03-11T19:55:38.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.413+0000 7ff767400640 1 -- 172.21.15.97:0/928831824 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff760003680 con 0x7ff7701033c0 2024-03-11T19:55:38.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.414+0000 7ff767400640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff7540a1a30 0x7ff7540a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:38.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.415+0000 7ff767400640 1 -- 172.21.15.97:0/928831824 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff760014070 con 0x7ff7701033c0 2024-03-11T19:55:38.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.415+0000 7ff767e00640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff7540a1a30 0x7ff7540a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:38.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.415+0000 7ff767e00640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff7540a1a30 0x7ff7540a3ef0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff758009230 tx=0x7ff758005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:38.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.419+0000 7ff767400640 1 -- 172.21.15.97:0/928831824 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff7600969c0 con 0x7ff7701033c0 2024-03-11T19:55:38.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.656+0000 7ff775a00640 1 -- 172.21.15.97:0/928831824 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff77010edd0 con 0x7ff7701033c0 2024-03-11T19:55:38.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.657+0000 7ff767400640 1 -- 172.21.15.97:0/928831824 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9251 (secure 0 0 0) 0x7ff76009a080 con 0x7ff7701033c0 2024-03-11T19:55:38.661 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:38.661 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24499,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6841/2531110572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2531110572},{"type":"v1","addr":"172.21.15.97:6841","nonce":2531110572}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[]} 2024-03-11T19:55:38.661 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 12 2024-03-11T19:55:38.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.660+0000 7ff775a00640 1 -- 172.21.15.97:0/928831824 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff7540a1a30 msgr2=0x7ff7540a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:38.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.660+0000 7ff775a00640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff7540a1a30 0x7ff7540a3ef0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff758009230 tx=0x7ff758005fd0 comp rx=0 tx=0).stop 2024-03-11T19:55:38.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.660+0000 7ff775a00640 1 -- 172.21.15.97:0/928831824 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7701033c0 msgr2=0x7ff770199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:38.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.660+0000 7ff775a00640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7701033c0 0x7ff770199430 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff76000e9c0 tx=0x7ff76000ee80 comp rx=0 tx=0).stop 2024-03-11T19:55:38.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.660+0000 7ff775a00640 1 -- 172.21.15.97:0/928831824 shutdown_connections 2024-03-11T19:55:38.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.660+0000 7ff775a00640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff770104f70 0x7ff7701a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:38.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.660+0000 7ff775a00640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7701045b0 0x7ff770199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:38.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.660+0000 7ff775a00640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff7540a1a30 0x7ff7540a3ef0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:38.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.660+0000 7ff775a00640 1 --2- 172.21.15.97:0/928831824 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff7701033c0 0x7ff770199430 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:38.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.660+0000 7ff775a00640 1 -- 172.21.15.97:0/928831824 >> 172.21.15.97:0/928831824 conn(0x7ff7700fe970 msgr2=0x7ff7701078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:38.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.660+0000 7ff775a00640 1 -- 172.21.15.97:0/928831824 shutdown_connections 2024-03-11T19:55:38.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:38.661+0000 7ff775a00640 1 -- 172.21.15.97:0/928831824 wait complete. 2024-03-11T19:55:38.736 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 19:55:38 smithi114 systemd[1]: Starting Ceph mds.b for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T19:55:38.992 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 19:55:38 smithi114 podman[66677]: 2024-03-11 19:55:38.63724567 +0000 UTC m=+0.017586795 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T19:55:38.992 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 19:55:38 smithi114 podman[66677]: 2024-03-11 19:55:38.754976972 +0000 UTC m=+0.135318097 container create 1d053fe741d3a18d257e4dc71628c7495208c1034a44539eb859847543088af1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T19:55:39.492 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 19:55:38 smithi114 podman[66677]: 2024-03-11 19:55:38.999247485 +0000 UTC m=+0.379588612 container init 1d053fe741d3a18d257e4dc71628c7495208c1034a44539eb859847543088af1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-11T19:55:39.492 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 19:55:39 smithi114 podman[66677]: 2024-03-11 19:55:39.001765665 +0000 UTC m=+0.382106791 container start 1d053fe741d3a18d257e4dc71628c7495208c1034a44539eb859847543088af1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T19:55:39.492 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 19:55:39 smithi114 ceph-mds[66697]: -- 172.21.15.114:0/1183969857 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5633831341a0 con 0x56338310ac00 2024-03-11T19:55:39.492 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 19:55:39 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b[66693]: starting mds.b at 2024-03-11T19:55:39.492 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 19:55:39 smithi114 bash[66677]: 1d053fe741d3a18d257e4dc71628c7495208c1034a44539eb859847543088af1 2024-03-11T19:55:39.492 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 19:55:39 smithi114 systemd[1]: Started Ceph mds.b for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T19:55:39.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:39 smithi097 ceph-mon[31898]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:39.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:39 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/928831824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:39.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:39.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:39.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:39.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:39.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:39.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:55:39.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:39.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:39 smithi114 ceph-mon[36376]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:39.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:39 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/928831824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:39.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:39.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:39.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:39.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:39.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:39.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:55:39.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:40.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:39 smithi161 ceph-mon[36332]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:40.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:39 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/928831824' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:40.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:40.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:40.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:40.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:40.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:40.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:55:40.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:40.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:40.869 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:40 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] up:boot 2024-03-11T19:55:40.869 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:40 smithi097 ceph-mon[31898]: fsmap 12 up:standby 2024-03-11T19:55:40.869 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:40 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T19:55:40.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.884+0000 7f4c5a400640 1 -- 172.21.15.97:0/2664434139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c54104f70 msgr2=0x7f4c5410b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:40.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.884+0000 7f4c5a400640 1 --2- 172.21.15.97:0/2664434139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c54104f70 0x7f4c5410b800 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009ca0 tx=0x7f4c4c02c100 comp rx=0 tx=0).stop 2024-03-11T19:55:40.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.884+0000 7f4c5a400640 1 -- 172.21.15.97:0/2664434139 shutdown_connections 2024-03-11T19:55:40.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.884+0000 7f4c5a400640 1 --2- 172.21.15.97:0/2664434139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c54104f70 0x7f4c5410b800 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:40.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.884+0000 7f4c5a400640 1 --2- 172.21.15.97:0/2664434139 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c541045b0 0x7f4c54104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:40.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.884+0000 7f4c5a400640 1 --2- 172.21.15.97:0/2664434139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c541033c0 0x7f4c541037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:40.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.884+0000 7f4c5a400640 1 -- 172.21.15.97:0/2664434139 >> 172.21.15.97:0/2664434139 conn(0x7f4c540fe930 msgr2=0x7f4c54100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:40.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.884+0000 7f4c5a400640 1 -- 172.21.15.97:0/2664434139 shutdown_connections 2024-03-11T19:55:40.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.884+0000 7f4c5a400640 1 -- 172.21.15.97:0/2664434139 wait complete. 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c5a400640 1 Processor -- start 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c5a400640 1 -- start start 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c5a400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c541033c0 0x7f4c541993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c5a400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c541045b0 0x7f4c54199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c5a400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c54104f70 0x7f4c541a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c5a400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541a0ee0 con 0x7f4c54104f70 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c5a400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541a1050 con 0x7f4c541045b0 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c5a400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541a1210 con 0x7f4c541033c0 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c53400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c541033c0 0x7f4c541993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c52a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c541045b0 0x7f4c54199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c52a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c541045b0 0x7f4c54199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:41124/0 (socket says 172.21.15.97:41124) 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c53400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c541033c0 0x7f4c541993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:40912/0 (socket says 172.21.15.97:40912) 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c52a00640 1 -- 172.21.15.97:0/3864879971 learned_addr learned my addr 172.21.15.97:0/3864879971 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:40.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.885+0000 7f4c53e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c54104f70 0x7f4c541a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:40.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.886+0000 7f4c53400640 1 -- 172.21.15.97:0/3864879971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c541045b0 msgr2=0x7f4c54199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:40.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.886+0000 7f4c53400640 1 --2- 172.21.15.97:0/3864879971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c541045b0 0x7f4c54199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:40.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.886+0000 7f4c53400640 1 -- 172.21.15.97:0/3864879971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c54104f70 msgr2=0x7f4c541a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:40.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.886+0000 7f4c53400640 1 --2- 172.21.15.97:0/3864879971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c54104f70 0x7f4c541a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:40.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.886+0000 7f4c53400640 1 -- 172.21.15.97:0/3864879971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c009900 con 0x7f4c541033c0 2024-03-11T19:55:40.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.886+0000 7f4c53400640 1 --2- 172.21.15.97:0/3864879971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c541033c0 0x7f4c541993f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400e9c0 tx=0x7f4c4400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:40.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.886+0000 7f4c4be00640 1 -- 172.21.15.97:0/3864879971 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4400ccc0 con 0x7f4c541033c0 2024-03-11T19:55:40.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.886+0000 7f4c5a400640 1 -- 172.21.15.97:0/3864879971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c541a14f0 con 0x7f4c541033c0 2024-03-11T19:55:40.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.886+0000 7f4c4be00640 1 -- 172.21.15.97:0/3864879971 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c4400ce20 con 0x7f4c541033c0 2024-03-11T19:55:40.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.886+0000 7f4c4be00640 1 -- 172.21.15.97:0/3864879971 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44010430 con 0x7f4c541033c0 2024-03-11T19:55:40.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.887+0000 7f4c5a400640 1 -- 172.21.15.97:0/3864879971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c541a1a00 con 0x7f4c541033c0 2024-03-11T19:55:40.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.888+0000 7f4c5a400640 1 -- 172.21.15.97:0/3864879971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5410a760 con 0x7f4c541033c0 2024-03-11T19:55:40.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.890+0000 7f4c4be00640 1 -- 172.21.15.97:0/3864879971 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4c44004000 con 0x7f4c541033c0 2024-03-11T19:55:40.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.891+0000 7f4c4be00640 1 --2- 172.21.15.97:0/3864879971 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c380a1ab0 0x7f4c380a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:40.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.891+0000 7f4c4be00640 1 -- 172.21.15.97:0/3864879971 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4c44014070 con 0x7f4c541033c0 2024-03-11T19:55:40.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.891+0000 7f4c52a00640 1 --2- 172.21.15.97:0/3864879971 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c380a1ab0 0x7f4c380a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:40.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.892+0000 7f4c52a00640 1 --2- 172.21.15.97:0/3864879971 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c380a1ab0 0x7f4c380a3f70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4c5419a900 tx=0x7f4c4000b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:40.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:40.896+0000 7f4c4be00640 1 -- 172.21.15.97:0/3864879971 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4c440968f0 con 0x7f4c541033c0 2024-03-11T19:55:40.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:40 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] up:boot 2024-03-11T19:55:40.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:40 smithi114 ceph-mon[36376]: fsmap 12 up:standby 2024-03-11T19:55:40.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:40 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T19:55:41.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:40 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] up:boot 2024-03-11T19:55:41.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:40 smithi161 ceph-mon[36332]: fsmap 12 up:standby 2024-03-11T19:55:41.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:40 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T19:55:41.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.133+0000 7f4c5a400640 1 -- 172.21.15.97:0/3864879971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c5410edd0 con 0x7f4c541033c0 2024-03-11T19:55:41.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.134+0000 7f4c4be00640 1 -- 172.21.15.97:0/3864879971 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10046 (secure 0 0 0) 0x7f4c44099fb0 con 0x7f4c541033c0 2024-03-11T19:55:41.138 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:41.138 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24499,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6841/2531110572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2531110572},{"type":"v1","addr":"172.21.15.97:6841","nonce":2531110572}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24599,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/1624532248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":1624532248},{"type":"v1","addr":"172.21.15.114:6839","nonce":1624532248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-11T19:55:41.139 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 13 2024-03-11T19:55:41.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.137+0000 7f4c5a400640 1 -- 172.21.15.97:0/3864879971 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c380a1ab0 msgr2=0x7f4c380a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:41.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.137+0000 7f4c5a400640 1 --2- 172.21.15.97:0/3864879971 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c380a1ab0 0x7f4c380a3f70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4c5419a900 tx=0x7f4c4000b480 comp rx=0 tx=0).stop 2024-03-11T19:55:41.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.137+0000 7f4c5a400640 1 -- 172.21.15.97:0/3864879971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c541033c0 msgr2=0x7f4c541993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:41.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.137+0000 7f4c5a400640 1 --2- 172.21.15.97:0/3864879971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c541033c0 0x7f4c541993f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400e9c0 tx=0x7f4c4400ee80 comp rx=0 tx=0).stop 2024-03-11T19:55:41.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.137+0000 7f4c5a400640 1 -- 172.21.15.97:0/3864879971 shutdown_connections 2024-03-11T19:55:41.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.137+0000 7f4c5a400640 1 --2- 172.21.15.97:0/3864879971 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c380a1ab0 0x7f4c380a3f70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.137+0000 7f4c5a400640 1 --2- 172.21.15.97:0/3864879971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4c54104f70 0x7f4c541a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.137+0000 7f4c5a400640 1 --2- 172.21.15.97:0/3864879971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c541045b0 0x7f4c54199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.137+0000 7f4c5a400640 1 --2- 172.21.15.97:0/3864879971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c541033c0 0x7f4c541993f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.137+0000 7f4c5a400640 1 -- 172.21.15.97:0/3864879971 >> 172.21.15.97:0/3864879971 conn(0x7f4c540fe930 msgr2=0x7f4c54107930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:41.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.138+0000 7f4c5a400640 1 -- 172.21.15.97:0/3864879971 shutdown_connections 2024-03-11T19:55:41.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.138+0000 7f4c5a400640 1 -- 172.21.15.97:0/3864879971 wait complete. 2024-03-11T19:55:41.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:55:41.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.355+0000 7f04c9400640 1 -- 172.21.15.97:0/3724800489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c4104f70 msgr2=0x7f04c410b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:41.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.355+0000 7f04c9400640 1 --2- 172.21.15.97:0/3724800489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c4104f70 0x7f04c410b800 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009be0 tx=0x7f04bc02bef0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.355+0000 7f04c9400640 1 -- 172.21.15.97:0/3724800489 shutdown_connections 2024-03-11T19:55:41.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.355+0000 7f04c9400640 1 --2- 172.21.15.97:0/3724800489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c4104f70 0x7f04c410b800 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.355+0000 7f04c9400640 1 --2- 172.21.15.97:0/3724800489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04c41045b0 0x7f04c4104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.355+0000 7f04c9400640 1 --2- 172.21.15.97:0/3724800489 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04c41033c0 0x7f04c41037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.355+0000 7f04c9400640 1 -- 172.21.15.97:0/3724800489 >> 172.21.15.97:0/3724800489 conn(0x7f04c40fe970 msgr2=0x7f04c4100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:41.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.355+0000 7f04c9400640 1 -- 172.21.15.97:0/3724800489 shutdown_connections 2024-03-11T19:55:41.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.355+0000 7f04c9400640 1 -- 172.21.15.97:0/3724800489 wait complete. 2024-03-11T19:55:41.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.356+0000 7f04c9400640 1 Processor -- start 2024-03-11T19:55:41.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.356+0000 7f04c9400640 1 -- start start 2024-03-11T19:55:41.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c9400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c41033c0 0x7f04c41993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:41.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04c41045b0 0x7f04c4199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04c4104f70 0x7f04c41a0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c9400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c41a0ed0 con 0x7f04c4104f70 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c9400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c41a1040 con 0x7f04c41045b0 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c9400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c41a1200 con 0x7f04c41033c0 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c3200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04c4104f70 0x7f04c41a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c2800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c41033c0 0x7f04c41993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c3200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04c4104f70 0x7f04c41a0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52182/0 (socket says 172.21.15.97:52182) 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c2800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c41033c0 0x7f04c41993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:40928/0 (socket says 172.21.15.97:40928) 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c3200640 1 -- 172.21.15.97:0/1246917124 learned_addr learned my addr 172.21.15.97:0/1246917124 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04bbe00640 1 --2- 172.21.15.97:0/1246917124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04c41045b0 0x7f04c4199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c2800640 1 -- 172.21.15.97:0/1246917124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04c41045b0 msgr2=0x7f04c4199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c2800640 1 --2- 172.21.15.97:0/1246917124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04c41045b0 0x7f04c4199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c2800640 1 -- 172.21.15.97:0/1246917124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04c4104f70 msgr2=0x7f04c41a0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:41.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c2800640 1 --2- 172.21.15.97:0/1246917124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04c4104f70 0x7f04c41a0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.357+0000 7f04c2800640 1 -- 172.21.15.97:0/1246917124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4009560 con 0x7f04c41033c0 2024-03-11T19:55:41.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.358+0000 7f04c2800640 1 --2- 172.21.15.97:0/1246917124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c41033c0 0x7f04c41993e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f04c4104410 tx=0x7f04b400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:41.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.358+0000 7f04bb400640 1 -- 172.21.15.97:0/1246917124 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4004440 con 0x7f04c41033c0 2024-03-11T19:55:41.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.358+0000 7f04bb400640 1 -- 172.21.15.97:0/1246917124 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04b400cda0 con 0x7f04c41033c0 2024-03-11T19:55:41.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.358+0000 7f04c9400640 1 -- 172.21.15.97:0/1246917124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04bc009840 con 0x7f04c41033c0 2024-03-11T19:55:41.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.358+0000 7f04bb400640 1 -- 172.21.15.97:0/1246917124 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4019480 con 0x7f04c41033c0 2024-03-11T19:55:41.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.358+0000 7f04c9400640 1 -- 172.21.15.97:0/1246917124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c41a18b0 con 0x7f04c41033c0 2024-03-11T19:55:41.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.359+0000 7f04c9400640 1 -- 172.21.15.97:0/1246917124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c410a610 con 0x7f04c41033c0 2024-03-11T19:55:41.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.361+0000 7f04bb400640 1 -- 172.21.15.97:0/1246917124 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f04b400c8c0 con 0x7f04c41033c0 2024-03-11T19:55:41.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.362+0000 7f04bb400640 1 --2- 172.21.15.97:0/1246917124 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f04a80a1ab0 0x7f04a80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:41.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.362+0000 7f04bb400640 1 -- 172.21.15.97:0/1246917124 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f04b4014070 con 0x7f04c41033c0 2024-03-11T19:55:41.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.362+0000 7f04bbe00640 1 --2- 172.21.15.97:0/1246917124 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f04a80a1ab0 0x7f04a80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:41.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.363+0000 7f04bbe00640 1 --2- 172.21.15.97:0/1246917124 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f04a80a1ab0 0x7f04a80a3f70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f04c419a8f0 tx=0x7f04ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:41.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.366+0000 7f04bb400640 1 -- 172.21.15.97:0/1246917124 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f04b4096b40 con 0x7f04c41033c0 2024-03-11T19:55:41.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.583+0000 7f04c9400640 1 -- 172.21.15.97:0/1246917124 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04c410a080 con 0x7f04c41033c0 2024-03-11T19:55:41.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.584+0000 7f04bb400640 1 -- 172.21.15.97:0/1246917124 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18480 (secure 0 0 0) 0x7f04b409a200 con 0x7f04c41033c0 2024-03-11T19:55:41.588 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:41.588 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":81,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:54:44.381499+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:55:41.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.587+0000 7f04c9400640 1 -- 172.21.15.97:0/1246917124 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f04a80a1ab0 msgr2=0x7f04a80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:41.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.587+0000 7f04c9400640 1 --2- 172.21.15.97:0/1246917124 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f04a80a1ab0 0x7f04a80a3f70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f04c419a8f0 tx=0x7f04ac005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:41.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.587+0000 7f04c9400640 1 -- 172.21.15.97:0/1246917124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c41033c0 msgr2=0x7f04c41993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:41.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.587+0000 7f04c9400640 1 --2- 172.21.15.97:0/1246917124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c41033c0 0x7f04c41993e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f04c4104410 tx=0x7f04b400ee50 comp rx=0 tx=0).stop 2024-03-11T19:55:41.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.587+0000 7f04c9400640 1 -- 172.21.15.97:0/1246917124 shutdown_connections 2024-03-11T19:55:41.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.587+0000 7f04c9400640 1 --2- 172.21.15.97:0/1246917124 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f04a80a1ab0 0x7f04a80a3f70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.587+0000 7f04c9400640 1 --2- 172.21.15.97:0/1246917124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f04c4104f70 0x7f04c41a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.587+0000 7f04c9400640 1 --2- 172.21.15.97:0/1246917124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04c41045b0 0x7f04c4199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.587+0000 7f04c9400640 1 --2- 172.21.15.97:0/1246917124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f04c41033c0 0x7f04c41993e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.587+0000 7f04c9400640 1 -- 172.21.15.97:0/1246917124 >> 172.21.15.97:0/1246917124 conn(0x7f04c40fe970 msgr2=0x7f04c41077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:41.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.587+0000 7f04c9400640 1 -- 172.21.15.97:0/1246917124 shutdown_connections 2024-03-11T19:55:41.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.587+0000 7f04c9400640 1 -- 172.21.15.97:0/1246917124 wait complete. 2024-03-11T19:55:41.609 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:50:29.051985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T19:55:41.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:55:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.798+0000 7f608e800640 1 -- 172.21.15.97:0/3985814951 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60881045b0 msgr2=0x7f6088104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.798+0000 7f608e800640 1 --2- 172.21.15.97:0/3985814951 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60881045b0 0x7f6088104a30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6070009900 tx=0x7f607002bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.798+0000 7f608e800640 1 -- 172.21.15.97:0/3985814951 shutdown_connections 2024-03-11T19:55:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.798+0000 7f608e800640 1 --2- 172.21.15.97:0/3985814951 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6088104f70 0x7f608810b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.798+0000 7f608e800640 1 --2- 172.21.15.97:0/3985814951 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60881045b0 0x7f6088104a30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.798+0000 7f608e800640 1 --2- 172.21.15.97:0/3985814951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60881033c0 0x7f60881037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.798+0000 7f608e800640 1 -- 172.21.15.97:0/3985814951 >> 172.21.15.97:0/3985814951 conn(0x7f60880fe930 msgr2=0x7f6088100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.798+0000 7f608e800640 1 -- 172.21.15.97:0/3985814951 shutdown_connections 2024-03-11T19:55:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.798+0000 7f608e800640 1 -- 172.21.15.97:0/3985814951 wait complete. 2024-03-11T19:55:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.799+0000 7f608e800640 1 Processor -- start 2024-03-11T19:55:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.800+0000 7f608e800640 1 -- start start 2024-03-11T19:55:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.800+0000 7f608e800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60881033c0 0x7f6088199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.800+0000 7f608e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f60881045b0 0x7f6088199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.800+0000 7f608e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6088104f70 0x7f60881a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:41.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.800+0000 7f608e800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60881a0f00 con 0x7f60881045b0 2024-03-11T19:55:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.800+0000 7f608e800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60881a1070 con 0x7f6088104f70 2024-03-11T19:55:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.800+0000 7f608e800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60881a1230 con 0x7f60881033c0 2024-03-11T19:55:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.800+0000 7f6087400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60881033c0 0x7f6088199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.800+0000 7f607fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f60881045b0 0x7f6088199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.800+0000 7f6087e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6088104f70 0x7f60881a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.800+0000 7f6087400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60881033c0 0x7f6088199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:40944/0 (socket says 172.21.15.97:40944) 2024-03-11T19:55:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.800+0000 7f6087400640 1 -- 172.21.15.97:0/1905212662 learned_addr learned my addr 172.21.15.97:0/1905212662 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.801+0000 7f6087400640 1 -- 172.21.15.97:0/1905212662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6088104f70 msgr2=0x7f60881a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.801+0000 7f6087400640 1 --2- 172.21.15.97:0/1905212662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6088104f70 0x7f60881a09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.801+0000 7f6087400640 1 -- 172.21.15.97:0/1905212662 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f60881045b0 msgr2=0x7f6088199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.801+0000 7f6087400640 1 --2- 172.21.15.97:0/1905212662 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f60881045b0 0x7f6088199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.801+0000 7f6087400640 1 -- 172.21.15.97:0/1905212662 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6070009560 con 0x7f60881033c0 2024-03-11T19:55:41.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.801+0000 7f6087e00640 1 --2- 172.21.15.97:0/1905212662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6088104f70 0x7f60881a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.801+0000 7f6087400640 1 --2- 172.21.15.97:0/1905212662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60881033c0 0x7f6088199410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f607800e9c0 tx=0x7f607800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.801+0000 7f6084c00640 1 -- 172.21.15.97:0/1905212662 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6078009700 con 0x7f60881033c0 2024-03-11T19:55:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.801+0000 7f6084c00640 1 -- 172.21.15.97:0/1905212662 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f607800cd80 con 0x7f60881033c0 2024-03-11T19:55:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.801+0000 7f608e800640 1 -- 172.21.15.97:0/1905212662 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60881a1510 con 0x7f60881033c0 2024-03-11T19:55:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.801+0000 7f6084c00640 1 -- 172.21.15.97:0/1905212662 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6078010430 con 0x7f60881033c0 2024-03-11T19:55:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.802+0000 7f608e800640 1 -- 172.21.15.97:0/1905212662 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60881a19f0 con 0x7f60881033c0 2024-03-11T19:55:41.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.803+0000 7f608e800640 1 -- 172.21.15.97:0/1905212662 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608810a6e0 con 0x7f60881033c0 2024-03-11T19:55:41.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.806+0000 7f6084c00640 1 -- 172.21.15.97:0/1905212662 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6078003680 con 0x7f60881033c0 2024-03-11T19:55:41.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.807+0000 7f6084c00640 1 --2- 172.21.15.97:0/1905212662 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f606c0a1a30 0x7f606c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:41.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.807+0000 7f6084c00640 1 -- 172.21.15.97:0/1905212662 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6078014070 con 0x7f60881033c0 2024-03-11T19:55:41.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.807+0000 7f607fe00640 1 --2- 172.21.15.97:0/1905212662 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f606c0a1a30 0x7f606c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:41.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.808+0000 7f607fe00640 1 --2- 172.21.15.97:0/1905212662 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f606c0a1a30 0x7f606c0a3ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6070000c00 tx=0x7f607002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:41.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:41.811+0000 7f6084c00640 1 -- 172.21.15.97:0/1905212662 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f60780969c0 con 0x7f60881033c0 2024-03-11T19:55:41.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:41 smithi097 ceph-mon[31898]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:41.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:41 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3864879971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:41.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:41 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:41.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:41 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:41.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:41 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:41.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:41 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:41.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:41 smithi114 ceph-mon[36376]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:41.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:41 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3864879971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:41.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:41 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:41.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:41 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:41.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:41 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:41.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:41 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:42.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.016+0000 7f608e800640 1 -- 172.21.15.97:0/1905212662 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f608810ee20 con 0x7f60881033c0 2024-03-11T19:55:42.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.017+0000 7f6084c00640 1 -- 172.21.15.97:0/1905212662 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f607809a080 con 0x7f60881033c0 2024-03-11T19:55:42.020 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-11T19:55:42.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.020+0000 7f608e800640 1 -- 172.21.15.97:0/1905212662 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f606c0a1a30 msgr2=0x7f606c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.020+0000 7f608e800640 1 --2- 172.21.15.97:0/1905212662 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f606c0a1a30 0x7f606c0a3ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6070000c00 tx=0x7f607002c7e0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.020+0000 7f608e800640 1 -- 172.21.15.97:0/1905212662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60881033c0 msgr2=0x7f6088199410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.020+0000 7f608e800640 1 --2- 172.21.15.97:0/1905212662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60881033c0 0x7f6088199410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f607800e9c0 tx=0x7f607800ee80 comp rx=0 tx=0).stop 2024-03-11T19:55:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.020+0000 7f608e800640 1 -- 172.21.15.97:0/1905212662 shutdown_connections 2024-03-11T19:55:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.020+0000 7f608e800640 1 --2- 172.21.15.97:0/1905212662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6088104f70 0x7f60881a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.020+0000 7f608e800640 1 --2- 172.21.15.97:0/1905212662 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f606c0a1a30 0x7f606c0a3ef0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.020+0000 7f608e800640 1 --2- 172.21.15.97:0/1905212662 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f60881045b0 0x7f6088199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.020+0000 7f608e800640 1 --2- 172.21.15.97:0/1905212662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60881033c0 0x7f6088199410 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.020+0000 7f608e800640 1 -- 172.21.15.97:0/1905212662 >> 172.21.15.97:0/1905212662 conn(0x7f60880fe930 msgr2=0x7f60881078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.020+0000 7f608e800640 1 -- 172.21.15.97:0/1905212662 shutdown_connections 2024-03-11T19:55:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.020+0000 7f608e800640 1 -- 172.21.15.97:0/1905212662 wait complete. 2024-03-11T19:55:42.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:41 smithi161 ceph-mon[36332]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:42.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:41 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3864879971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:42.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:41 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:42.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:41 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:42.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:41 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:42.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:41 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:42.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-11T19:55:42.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.246+0000 7f5c99200640 1 -- 172.21.15.97:0/3726910272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94102180 msgr2=0x7f5c94102600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:42.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.246+0000 7f5c99200640 1 --2- 172.21.15.97:0/3726910272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94102180 0x7f5c94102600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5c80009900 tx=0x7f5c8002bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:42.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.246+0000 7f5c99200640 1 -- 172.21.15.97:0/3726910272 shutdown_connections 2024-03-11T19:55:42.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.246+0000 7f5c99200640 1 --2- 172.21.15.97:0/3726910272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5c94102b40 0x7f5c941093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.246+0000 7f5c99200640 1 --2- 172.21.15.97:0/3726910272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94102180 0x7f5c94102600 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.246+0000 7f5c99200640 1 --2- 172.21.15.97:0/3726910272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c94100f90 0x7f5c941013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.246+0000 7f5c99200640 1 -- 172.21.15.97:0/3726910272 >> 172.21.15.97:0/3726910272 conn(0x7f5c940fc4e0 msgr2=0x7f5c940fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:42.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.246+0000 7f5c99200640 1 -- 172.21.15.97:0/3726910272 shutdown_connections 2024-03-11T19:55:42.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.246+0000 7f5c99200640 1 -- 172.21.15.97:0/3726910272 wait complete. 2024-03-11T19:55:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.247+0000 7f5c99200640 1 Processor -- start 2024-03-11T19:55:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.247+0000 7f5c99200640 1 -- start start 2024-03-11T19:55:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c99200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c94100f90 0x7f5c94199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c99200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5c94102180 0x7f5c941997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c99200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94102b40 0x7f5c941a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c99200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c941a0d50 con 0x7f5c94102180 2024-03-11T19:55:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c99200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c941a0ec0 con 0x7f5c94102b40 2024-03-11T19:55:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c99200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c941a1080 con 0x7f5c94100f90 2024-03-11T19:55:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c92a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5c94102180 0x7f5c941997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c93e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94102b40 0x7f5c941a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c93e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94102b40 0x7f5c941a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:41186/0 (socket says 172.21.15.97:41186) 2024-03-11T19:55:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c93e00640 1 -- 172.21.15.97:0/1191596547 learned_addr learned my addr 172.21.15.97:0/1191596547 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c93400640 1 --2- 172.21.15.97:0/1191596547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c94100f90 0x7f5c94199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c93e00640 1 -- 172.21.15.97:0/1191596547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c94100f90 msgr2=0x7f5c94199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c93e00640 1 --2- 172.21.15.97:0/1191596547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c94100f90 0x7f5c94199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c93e00640 1 -- 172.21.15.97:0/1191596547 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5c94102180 msgr2=0x7f5c941997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c93e00640 1 --2- 172.21.15.97:0/1191596547 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5c94102180 0x7f5c941997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.248+0000 7f5c93e00640 1 -- 172.21.15.97:0/1191596547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c80009560 con 0x7f5c94102b40 2024-03-11T19:55:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.249+0000 7f5c93e00640 1 --2- 172.21.15.97:0/1191596547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94102b40 0x7f5c941a0810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00eb10 tx=0x7f5c8c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.249+0000 7f5c8be00640 1 -- 172.21.15.97:0/1191596547 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c004b20 con 0x7f5c94102b40 2024-03-11T19:55:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.249+0000 7f5c8be00640 1 -- 172.21.15.97:0/1191596547 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c8c004c80 con 0x7f5c94102b40 2024-03-11T19:55:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.249+0000 7f5c8be00640 1 -- 172.21.15.97:0/1191596547 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c010650 con 0x7f5c94102b40 2024-03-11T19:55:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.249+0000 7f5c99200640 1 -- 172.21.15.97:0/1191596547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c941a1360 con 0x7f5c94102b40 2024-03-11T19:55:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.249+0000 7f5c99200640 1 -- 172.21.15.97:0/1191596547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c941a1840 con 0x7f5c94102b40 2024-03-11T19:55:42.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.250+0000 7f5c99200640 1 -- 172.21.15.97:0/1191596547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9410c970 con 0x7f5c94102b40 2024-03-11T19:55:42.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.252+0000 7f5c8be00640 1 -- 172.21.15.97:0/1191596547 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5c8c010830 con 0x7f5c94102b40 2024-03-11T19:55:42.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.253+0000 7f5c8be00640 1 --2- 172.21.15.97:0/1191596547 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5c780a1ab0 0x7f5c780a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:42.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.254+0000 7f5c8be00640 1 -- 172.21.15.97:0/1191596547 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5c8c014070 con 0x7f5c94102b40 2024-03-11T19:55:42.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.254+0000 7f5c93400640 1 --2- 172.21.15.97:0/1191596547 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5c780a1ab0 0x7f5c780a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:42.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.255+0000 7f5c93400640 1 --2- 172.21.15.97:0/1191596547 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5c780a1ab0 0x7f5c780a3f70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5c84005fd0 tx=0x7f5c8400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:42.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.256+0000 7f5c8be00640 1 -- 172.21.15.97:0/1191596547 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5c8c096a80 con 0x7f5c94102b40 2024-03-11T19:55:42.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.461+0000 7f5c99200640 1 -- 172.21.15.97:0/1191596547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5c9410cb60 con 0x7f5c94102b40 2024-03-11T19:55:42.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.462+0000 7f5c8be00640 1 -- 172.21.15.97:0/1191596547 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5c8c09a140 con 0x7f5c94102b40 2024-03-11T19:55:42.465 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:42.465 INFO:teuthology.orchestra.run.smithi097.stdout:[ 2024-03-11T19:55:42.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T19:55:42.465 INFO:teuthology.orchestra.run.smithi097.stdout: "poolnum": 1, 2024-03-11T19:55:42.465 INFO:teuthology.orchestra.run.smithi097.stdout: "poolname": ".mgr" 2024-03-11T19:55:42.465 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T19:55:42.465 INFO:teuthology.orchestra.run.smithi097.stdout:] 2024-03-11T19:55:42.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.464+0000 7f5c99200640 1 -- 172.21.15.97:0/1191596547 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5c780a1ab0 msgr2=0x7f5c780a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:42.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.464+0000 7f5c99200640 1 --2- 172.21.15.97:0/1191596547 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5c780a1ab0 0x7f5c780a3f70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5c84005fd0 tx=0x7f5c8400b410 comp rx=0 tx=0).stop 2024-03-11T19:55:42.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.464+0000 7f5c99200640 1 -- 172.21.15.97:0/1191596547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94102b40 msgr2=0x7f5c941a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:42.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.464+0000 7f5c99200640 1 --2- 172.21.15.97:0/1191596547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94102b40 0x7f5c941a0810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00eb10 tx=0x7f5c8c00c2d0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.464+0000 7f5c99200640 1 -- 172.21.15.97:0/1191596547 shutdown_connections 2024-03-11T19:55:42.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.464+0000 7f5c99200640 1 --2- 172.21.15.97:0/1191596547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c94102b40 0x7f5c941a0810 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.464+0000 7f5c99200640 1 --2- 172.21.15.97:0/1191596547 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5c780a1ab0 0x7f5c780a3f70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.464+0000 7f5c99200640 1 --2- 172.21.15.97:0/1191596547 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5c94102180 0x7f5c941997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.464+0000 7f5c99200640 1 --2- 172.21.15.97:0/1191596547 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c94100f90 0x7f5c94199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.464+0000 7f5c99200640 1 -- 172.21.15.97:0/1191596547 >> 172.21.15.97:0/1191596547 conn(0x7f5c940fc4e0 msgr2=0x7f5c940733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:42.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.464+0000 7f5c99200640 1 -- 172.21.15.97:0/1191596547 shutdown_connections 2024-03-11T19:55:42.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.464+0000 7f5c99200640 1 -- 172.21.15.97:0/1191596547 wait complete. 2024-03-11T19:55:42.484 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-11T19:55:42.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-11T19:55:42.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.669+0000 7f7d56600640 1 -- 172.21.15.97:0/2981654950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d50075b40 msgr2=0x7f7d50075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:42.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.669+0000 7f7d56600640 1 --2- 172.21.15.97:0/2981654950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d50075b40 0x7f7d50075f60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009900 tx=0x7f7d4002bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:42.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.669+0000 7f7d56600640 1 -- 172.21.15.97:0/2981654950 shutdown_connections 2024-03-11T19:55:42.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.669+0000 7f7d56600640 1 --2- 172.21.15.97:0/2981654950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d50077720 0x7f7d50077fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.669+0000 7f7d56600640 1 --2- 172.21.15.97:0/2981654950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d50076d80 0x7f7d500771e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.669+0000 7f7d56600640 1 --2- 172.21.15.97:0/2981654950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d50075b40 0x7f7d50075f60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.669+0000 7f7d56600640 1 -- 172.21.15.97:0/2981654950 >> 172.21.15.97:0/2981654950 conn(0x7f7d500fe930 msgr2=0x7f7d50100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:42.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.669+0000 7f7d56600640 1 -- 172.21.15.97:0/2981654950 shutdown_connections 2024-03-11T19:55:42.673 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.669+0000 7f7d56600640 1 -- 172.21.15.97:0/2981654950 wait complete. 2024-03-11T19:55:42.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.670+0000 7f7d56600640 1 Processor -- start 2024-03-11T19:55:42.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.670+0000 7f7d56600640 1 -- start start 2024-03-11T19:55:42.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.670+0000 7f7d56600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d50075b40 0x7f7d5019d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:42.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.670+0000 7f7d56600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d50076d80 0x7f7d5019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:42.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.670+0000 7f7d56600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d50077720 0x7f7d501a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:42.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.670+0000 7f7d56600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d501a5350 con 0x7f7d50076d80 2024-03-11T19:55:42.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.670+0000 7f7d56600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d501a54c0 con 0x7f7d50075b40 2024-03-11T19:55:42.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.670+0000 7f7d56600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d501a5680 con 0x7f7d50077720 2024-03-11T19:55:42.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.671+0000 7f7d4f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d50075b40 0x7f7d5019d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:42.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.671+0000 7f7d4f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d50075b40 0x7f7d5019d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:41202/0 (socket says 172.21.15.97:41202) 2024-03-11T19:55:42.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.671+0000 7f7d4f400640 1 -- 172.21.15.97:0/621228466 learned_addr learned my addr 172.21.15.97:0/621228466 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.671+0000 7f7d4fe00640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d50077720 0x7f7d501a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.671+0000 7f7d4ea00640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d50076d80 0x7f7d5019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.671+0000 7f7d4f400640 1 -- 172.21.15.97:0/621228466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d50077720 msgr2=0x7f7d501a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.671+0000 7f7d4f400640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d50077720 0x7f7d501a4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.671+0000 7f7d4f400640 1 -- 172.21.15.97:0/621228466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d50076d80 msgr2=0x7f7d5019dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.671+0000 7f7d4f400640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d50076d80 0x7f7d5019dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.671+0000 7f7d4f400640 1 -- 172.21.15.97:0/621228466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d48009840 con 0x7f7d50075b40 2024-03-11T19:55:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.671+0000 7f7d4f400640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d50075b40 0x7f7d5019d860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d40006010 tx=0x7f7d40005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.671+0000 7f7d47e00640 1 -- 172.21.15.97:0/621228466 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d400043d0 con 0x7f7d50075b40 2024-03-11T19:55:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.672+0000 7f7d47e00640 1 -- 172.21.15.97:0/621228466 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d400052d0 con 0x7f7d50075b40 2024-03-11T19:55:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.672+0000 7f7d47e00640 1 -- 172.21.15.97:0/621228466 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40003bc0 con 0x7f7d50075b40 2024-03-11T19:55:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.672+0000 7f7d56600640 1 -- 172.21.15.97:0/621228466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d40009560 con 0x7f7d50075b40 2024-03-11T19:55:42.676 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.672+0000 7f7d56600640 1 -- 172.21.15.97:0/621228466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d501a5c30 con 0x7f7d50075b40 2024-03-11T19:55:42.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.673+0000 7f7d56600640 1 -- 172.21.15.97:0/621228466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d5010fd30 con 0x7f7d50075b40 2024-03-11T19:55:42.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.675+0000 7f7d47e00640 1 -- 172.21.15.97:0/621228466 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7d40003d40 con 0x7f7d50075b40 2024-03-11T19:55:42.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.676+0000 7f7d47e00640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7d340a1ab0 0x7f7d340a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:42.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.676+0000 7f7d47e00640 1 -- 172.21.15.97:0/621228466 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7d400ef400 con 0x7f7d50075b40 2024-03-11T19:55:42.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.676+0000 7f7d4ea00640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7d340a1ab0 0x7f7d340a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:42.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.677+0000 7f7d4ea00640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7d340a1ab0 0x7f7d340a3f70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7d5019ed70 tx=0x7f7d3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.678+0000 7f7d47e00640 1 -- 172.21.15.97:0/621228466 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7d400b8a10 con 0x7f7d50075b40 2024-03-11T19:55:42.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:42.881+0000 7f7d56600640 1 -- 172.21.15.97:0/621228466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7d501131c0 con 0x7f7d50075b40 2024-03-11T19:55:42.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:42 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1246917124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:42.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:42 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1905212662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:55:42.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:42.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:42.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:42.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:55:42.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:42.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:42 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:55:42.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:42 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1191596547' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-11T19:55:42.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:42 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1246917124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:42.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:42 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1905212662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:55:42.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:42.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:42.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:42.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:55:42.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:42.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:42 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:55:42.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:42 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1191596547' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-11T19:55:43.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:42 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1246917124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:43.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:42 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1905212662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:55:43.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:43.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:43.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:43.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:55:43.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:43.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:42 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:55:43.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:42 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1191596547' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-11T19:55:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.577+0000 7f7d47e00640 1 -- 172.21.15.97:0/621228466 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7d400bc0d0 con 0x7f7d50075b40 2024-03-11T19:55:43.585 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' created 2024-03-11T19:55:43.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.583+0000 7f7d56600640 1 -- 172.21.15.97:0/621228466 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7d340a1ab0 msgr2=0x7f7d340a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:43.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.583+0000 7f7d56600640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7d340a1ab0 0x7f7d340a3f70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7d5019ed70 tx=0x7f7d3c005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:43.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.584+0000 7f7d56600640 1 -- 172.21.15.97:0/621228466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d50075b40 msgr2=0x7f7d5019d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:43.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.584+0000 7f7d56600640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d50075b40 0x7f7d5019d860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d40006010 tx=0x7f7d40005e80 comp rx=0 tx=0).stop 2024-03-11T19:55:43.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.584+0000 7f7d56600640 1 -- 172.21.15.97:0/621228466 shutdown_connections 2024-03-11T19:55:43.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.584+0000 7f7d56600640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d50077720 0x7f7d501a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:43.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.584+0000 7f7d56600640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7d340a1ab0 0x7f7d340a3f70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:43.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.584+0000 7f7d56600640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7d50076d80 0x7f7d5019dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:43.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.584+0000 7f7d56600640 1 --2- 172.21.15.97:0/621228466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d50075b40 0x7f7d5019d860 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:43.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.584+0000 7f7d56600640 1 -- 172.21.15.97:0/621228466 >> 172.21.15.97:0/621228466 conn(0x7f7d500fe930 msgr2=0x7f7d50101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:43.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.584+0000 7f7d56600640 1 -- 172.21.15.97:0/621228466 shutdown_connections 2024-03-11T19:55:43.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.585+0000 7f7d56600640 1 -- 172.21.15.97:0/621228466 wait complete. 2024-03-11T19:55:43.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-11T19:55:43.788 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:43 smithi161 ceph-mon[36332]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:43.788 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:43 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/621228466' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-11T19:55:43.788 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:43 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-11T19:55:43.802 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.799+0000 7fc0ae800640 1 -- 172.21.15.97:0/3574301095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a81077a0 msgr2=0x7fc0a8107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:43.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.799+0000 7fc0ae800640 1 --2- 172.21.15.97:0/3574301095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a81077a0 0x7fc0a8107bc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc098009900 tx=0x7fc09802bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:43.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.799+0000 7fc0ae800640 1 -- 172.21.15.97:0/3574301095 shutdown_connections 2024-03-11T19:55:43.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.799+0000 7fc0ae800640 1 --2- 172.21.15.97:0/3574301095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a8109350 0x7fc0a810fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:43.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.799+0000 7fc0ae800640 1 --2- 172.21.15.97:0/3574301095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0a8108990 0x7fc0a8108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:43.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.799+0000 7fc0ae800640 1 --2- 172.21.15.97:0/3574301095 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a81077a0 0x7fc0a8107bc0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:43.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.799+0000 7fc0ae800640 1 -- 172.21.15.97:0/3574301095 >> 172.21.15.97:0/3574301095 conn(0x7fc0a8076740 msgr2=0x7fc0a8078b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:43.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.799+0000 7fc0ae800640 1 -- 172.21.15.97:0/3574301095 shutdown_connections 2024-03-11T19:55:43.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.799+0000 7fc0ae800640 1 -- 172.21.15.97:0/3574301095 wait complete. 2024-03-11T19:55:43.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.800+0000 7fc0ae800640 1 Processor -- start 2024-03-11T19:55:43.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.800+0000 7fc0ae800640 1 -- start start 2024-03-11T19:55:43.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.800+0000 7fc0ae800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0a81077a0 0x7fc0a819d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:43.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.800+0000 7fc0ae800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a8108990 0x7fc0a819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:43.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.800+0000 7fc0ae800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a8109350 0x7fc0a81a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:43.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.800+0000 7fc0ae800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a81a5360 con 0x7fc0a81077a0 2024-03-11T19:55:43.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.800+0000 7fc0ae800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a81a54d0 con 0x7fc0a8108990 2024-03-11T19:55:43.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.800+0000 7fc0ae800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a81a5690 con 0x7fc0a8109350 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.801+0000 7fc0a7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0a81077a0 0x7fc0a819d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.801+0000 7fc0a7e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a8109350 0x7fc0a81a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.801+0000 7fc0a7e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a8109350 0x7fc0a81a4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:40974/0 (socket says 172.21.15.97:40974) 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.801+0000 7fc0a7e00640 1 -- 172.21.15.97:0/2620569603 learned_addr learned my addr 172.21.15.97:0/2620569603 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.801+0000 7fc0a6a00640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a8108990 0x7fc0a819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.801+0000 7fc0a7e00640 1 -- 172.21.15.97:0/2620569603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a8108990 msgr2=0x7fc0a819ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.801+0000 7fc0a7e00640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a8108990 0x7fc0a819ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.801+0000 7fc0a7e00640 1 -- 172.21.15.97:0/2620569603 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0a81077a0 msgr2=0x7fc0a819d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.801+0000 7fc0a7e00640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0a81077a0 0x7fc0a819d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.801+0000 7fc0a7e00640 1 -- 172.21.15.97:0/2620569603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc098009560 con 0x7fc0a8109350 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.801+0000 7fc0a7400640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0a81077a0 0x7fc0a819d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.801+0000 7fc0a6a00640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a8108990 0x7fc0a819ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.801+0000 7fc0a7e00640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a8109350 0x7fc0a81a4e20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc0a000eb10 tx=0x7fc0a000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:43.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.802+0000 7fc09fe00640 1 -- 172.21.15.97:0/2620569603 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a0004b20 con 0x7fc0a8109350 2024-03-11T19:55:43.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.802+0000 7fc09fe00640 1 -- 172.21.15.97:0/2620569603 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0a0004c80 con 0x7fc0a8109350 2024-03-11T19:55:43.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.802+0000 7fc0ae800640 1 -- 172.21.15.97:0/2620569603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a81a5970 con 0x7fc0a8109350 2024-03-11T19:55:43.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.802+0000 7fc09fe00640 1 -- 172.21.15.97:0/2620569603 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a0010650 con 0x7fc0a8109350 2024-03-11T19:55:43.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.802+0000 7fc0ae800640 1 -- 172.21.15.97:0/2620569603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a81a5e30 con 0x7fc0a8109350 2024-03-11T19:55:43.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.803+0000 7fc0ae800640 1 -- 172.21.15.97:0/2620569603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a810eb90 con 0x7fc0a8109350 2024-03-11T19:55:43.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.805+0000 7fc09fe00640 1 -- 172.21.15.97:0/2620569603 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc0a0005020 con 0x7fc0a8109350 2024-03-11T19:55:43.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.806+0000 7fc09fe00640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc08c0a1a30 0x7fc08c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:43.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.806+0000 7fc09fe00640 1 -- 172.21.15.97:0/2620569603 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fc0a0014070 con 0x7fc0a8109350 2024-03-11T19:55:43.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.806+0000 7fc0a7400640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc08c0a1a30 0x7fc08c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:43.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.807+0000 7fc0a7400640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc08c0a1a30 0x7fc08c0a3ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc098005ab0 tx=0x7fc098005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:43.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:43.810+0000 7fc09fe00640 1 -- 172.21.15.97:0/2620569603 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc0a0096bd0 con 0x7fc0a8109350 2024-03-11T19:55:43.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:43 smithi097 ceph-mon[31898]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:43.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:43 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/621228466' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-11T19:55:43.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:43 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-11T19:55:43.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:43 smithi114 ceph-mon[36376]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:43.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:43 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/621228466' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-11T19:55:43.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:43 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-11T19:55:44.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.008+0000 7fc0ae800640 1 -- 172.21.15.97:0/2620569603 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7fc0a802ce40 con 0x7fc0a8109350 2024-03-11T19:55:44.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.659+0000 7fc09fe00640 1 -- 172.21.15.97:0/2620569603 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) v1 ==== 171+0+0 (secure 0 0 0) 0x7fc0a009a290 con 0x7fc0a8109350 2024-03-11T19:55:44.662 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' created 2024-03-11T19:55:44.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.664+0000 7fc0ae800640 1 -- 172.21.15.97:0/2620569603 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc08c0a1a30 msgr2=0x7fc08c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:44.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.664+0000 7fc0ae800640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc08c0a1a30 0x7fc08c0a3ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc098005ab0 tx=0x7fc098005fd0 comp rx=0 tx=0).stop 2024-03-11T19:55:44.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.664+0000 7fc0ae800640 1 -- 172.21.15.97:0/2620569603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a8109350 msgr2=0x7fc0a81a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:44.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.664+0000 7fc0ae800640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a8109350 0x7fc0a81a4e20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc0a000eb10 tx=0x7fc0a000c2d0 comp rx=0 tx=0).stop 2024-03-11T19:55:44.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.665+0000 7fc0ae800640 1 -- 172.21.15.97:0/2620569603 shutdown_connections 2024-03-11T19:55:44.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.665+0000 7fc0ae800640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a8109350 0x7fc0a81a4e20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:44.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.665+0000 7fc0ae800640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc0a8108990 0x7fc0a819ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:44.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.665+0000 7fc0ae800640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc08c0a1a30 0x7fc08c0a3ef0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:44.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.665+0000 7fc0ae800640 1 --2- 172.21.15.97:0/2620569603 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc0a81077a0 0x7fc0a819d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:44.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.665+0000 7fc0ae800640 1 -- 172.21.15.97:0/2620569603 >> 172.21.15.97:0/2620569603 conn(0x7fc0a8076740 msgr2=0x7fc0a810bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:44.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.665+0000 7fc0ae800640 1 -- 172.21.15.97:0/2620569603 shutdown_connections 2024-03-11T19:55:44.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.665+0000 7fc0ae800640 1 -- 172.21.15.97:0/2620569603 wait complete. 2024-03-11T19:55:44.696 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-11T19:55:44.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.872+0000 7f22fda00640 1 -- 172.21.15.97:0/469058732 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22f81045b0 msgr2=0x7f22f8104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:44.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.872+0000 7f22fda00640 1 --2- 172.21.15.97:0/469058732 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22f81045b0 0x7f22f8104a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f22e4009860 tx=0x7f22e402bb70 comp rx=0 tx=0).stop 2024-03-11T19:55:44.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.873+0000 7f22fda00640 1 -- 172.21.15.97:0/469058732 shutdown_connections 2024-03-11T19:55:44.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.873+0000 7f22fda00640 1 --2- 172.21.15.97:0/469058732 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22f8104f70 0x7f22f810b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:44.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.873+0000 7f22fda00640 1 --2- 172.21.15.97:0/469058732 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22f81045b0 0x7f22f8104a30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:44.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.873+0000 7f22fda00640 1 --2- 172.21.15.97:0/469058732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f81033c0 0x7f22f81037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:44.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.873+0000 7f22fda00640 1 -- 172.21.15.97:0/469058732 >> 172.21.15.97:0/469058732 conn(0x7f22f80fe930 msgr2=0x7f22f8100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:44.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.873+0000 7f22fda00640 1 -- 172.21.15.97:0/469058732 shutdown_connections 2024-03-11T19:55:44.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.873+0000 7f22fda00640 1 -- 172.21.15.97:0/469058732 wait complete. 2024-03-11T19:55:44.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22fda00640 1 Processor -- start 2024-03-11T19:55:44.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22fda00640 1 -- start start 2024-03-11T19:55:44.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22fda00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f81033c0 0x7f22f8199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:44.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22fda00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22f81045b0 0x7f22f8199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:44.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22fda00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22f8104f70 0x7f22f81a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:44.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22fda00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f81a0f10 con 0x7f22f8104f70 2024-03-11T19:55:44.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22fda00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f81a1080 con 0x7f22f81033c0 2024-03-11T19:55:44.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22fda00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f81a1240 con 0x7f22f81045b0 2024-03-11T19:55:44.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22f7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22f8104f70 0x7f22f81a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:44.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22f6000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22f81045b0 0x7f22f8199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:44.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22f7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22f8104f70 0x7f22f81a09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40562/0 (socket says 172.21.15.97:40562) 2024-03-11T19:55:44.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22f7400640 1 -- 172.21.15.97:0/3314055032 learned_addr learned my addr 172.21.15.97:0/3314055032 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:44.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22f6000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22f81045b0 0x7f22f8199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:41510/0 (socket says 172.21.15.97:41510) 2024-03-11T19:55:44.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.874+0000 7f22f6a00640 1 --2- 172.21.15.97:0/3314055032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f81033c0 0x7f22f8199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:44.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.875+0000 7f22f6000640 1 -- 172.21.15.97:0/3314055032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f81033c0 msgr2=0x7f22f8199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:44.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.875+0000 7f22f6000640 1 --2- 172.21.15.97:0/3314055032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f81033c0 0x7f22f8199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:44.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.875+0000 7f22f6000640 1 -- 172.21.15.97:0/3314055032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22f8104f70 msgr2=0x7f22f81a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:44.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.875+0000 7f22f6000640 1 --2- 172.21.15.97:0/3314055032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22f8104f70 0x7f22f81a09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:44.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.875+0000 7f22f6000640 1 -- 172.21.15.97:0/3314055032 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e4009490 con 0x7f22f81045b0 2024-03-11T19:55:44.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.875+0000 7f22f6000640 1 --2- 172.21.15.97:0/3314055032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22f81045b0 0x7f22f8199960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f22e402c0b0 tx=0x7f22e40045a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:44.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.875+0000 7f22ef400640 1 -- 172.21.15.97:0/3314055032 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e402c510 con 0x7f22f81045b0 2024-03-11T19:55:44.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.875+0000 7f22fda00640 1 -- 172.21.15.97:0/3314055032 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f81a14c0 con 0x7f22f81045b0 2024-03-11T19:55:44.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.875+0000 7f22ef400640 1 -- 172.21.15.97:0/3314055032 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22e402cb50 con 0x7f22f81045b0 2024-03-11T19:55:44.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.875+0000 7f22ef400640 1 -- 172.21.15.97:0/3314055032 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e4032600 con 0x7f22f81045b0 2024-03-11T19:55:44.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.876+0000 7f22fda00640 1 -- 172.21.15.97:0/3314055032 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f81a1880 con 0x7f22f81045b0 2024-03-11T19:55:44.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.877+0000 7f22fda00640 1 -- 172.21.15.97:0/3314055032 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f810a680 con 0x7f22f81045b0 2024-03-11T19:55:44.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.879+0000 7f22ef400640 1 -- 172.21.15.97:0/3314055032 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f22e403b040 con 0x7f22f81045b0 2024-03-11T19:55:44.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.880+0000 7f22ef400640 1 --2- 172.21.15.97:0/3314055032 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22dc0a1ab0 0x7f22dc0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:44.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.880+0000 7f22ef400640 1 -- 172.21.15.97:0/3314055032 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f22e40ef140 con 0x7f22f81045b0 2024-03-11T19:55:44.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.880+0000 7f22f6a00640 1 --2- 172.21.15.97:0/3314055032 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22dc0a1ab0 0x7f22dc0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:44.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.881+0000 7f22f6a00640 1 --2- 172.21.15.97:0/3314055032 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22dc0a1ab0 0x7f22dc0a3f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f22f8104410 tx=0x7f22e8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:44.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:44.884+0000 7f22ef400640 1 -- 172.21.15.97:0/3314055032 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f22e40b8460 con 0x7f22f81045b0 2024-03-11T19:55:44.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:44 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-11T19:55:44.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:44 smithi114 ceph-mon[36376]: osdmap e82: 12 total, 12 up, 12 in 2024-03-11T19:55:45.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:44 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-11T19:55:45.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:44 smithi161 ceph-mon[36332]: osdmap e82: 12 total, 12 up, 12 in 2024-03-11T19:55:45.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:45.145+0000 7f22fda00640 1 -- 172.21.15.97:0/3314055032 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f22f802ce10 con 0x7f22f81045b0 2024-03-11T19:55:45.169 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:44 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-11T19:55:45.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:44 smithi097 ceph-mon[31898]: osdmap e82: 12 total, 12 up, 12 in 2024-03-11T19:55:45.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2620569603' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-11T19:55:45.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-11T19:55:45.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: pgmap v312: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:45.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-11T19:55:45.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: osdmap e83: 12 total, 12 up, 12 in 2024-03-11T19:55:45.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:45.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:45.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:45.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:45.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T19:55:45.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:45.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:55:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3314055032' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T19:55:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T19:55:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:45 smithi114 ceph-mon[36376]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-11T19:55:46.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2620569603' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-11T19:55:46.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-11T19:55:46.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: pgmap v312: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:46.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-11T19:55:46.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: osdmap e83: 12 total, 12 up, 12 in 2024-03-11T19:55:46.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T19:55:46.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:46.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:55:46.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3314055032' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T19:55:46.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T19:55:46.039 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:45 smithi161 ceph-mon[36332]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-11T19:55:46.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.041+0000 7f22ef400640 1 -- 172.21.15.97:0/3314055032 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f22e40bbb20 con 0x7f22f81045b0 2024-03-11T19:55:46.045 INFO:teuthology.orchestra.run.smithi097.stderr:new fs with metadata pool 2 and data pool 3 2024-03-11T19:55:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.048+0000 7f22fda00640 1 -- 172.21.15.97:0/3314055032 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22dc0a1ab0 msgr2=0x7f22dc0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.048+0000 7f22fda00640 1 --2- 172.21.15.97:0/3314055032 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22dc0a1ab0 0x7f22dc0a3f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f22f8104410 tx=0x7f22e8009260 comp rx=0 tx=0).stop 2024-03-11T19:55:46.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.048+0000 7f22fda00640 1 -- 172.21.15.97:0/3314055032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22f81045b0 msgr2=0x7f22f8199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.048+0000 7f22fda00640 1 --2- 172.21.15.97:0/3314055032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22f81045b0 0x7f22f8199960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f22e402c0b0 tx=0x7f22e40045a0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.049+0000 7f22fda00640 1 -- 172.21.15.97:0/3314055032 shutdown_connections 2024-03-11T19:55:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.049+0000 7f22fda00640 1 --2- 172.21.15.97:0/3314055032 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22dc0a1ab0 0x7f22dc0a3f70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.049+0000 7f22fda00640 1 --2- 172.21.15.97:0/3314055032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f22f8104f70 0x7f22f81a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.049+0000 7f22fda00640 1 --2- 172.21.15.97:0/3314055032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22f81045b0 0x7f22f8199960 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.049+0000 7f22fda00640 1 --2- 172.21.15.97:0/3314055032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f22f81033c0 0x7f22f8199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.049+0000 7f22fda00640 1 -- 172.21.15.97:0/3314055032 >> 172.21.15.97:0/3314055032 conn(0x7f22f80fe930 msgr2=0x7f22f8107850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.049+0000 7f22fda00640 1 -- 172.21.15.97:0/3314055032 shutdown_connections 2024-03-11T19:55:46.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.050+0000 7f22fda00640 1 -- 172.21.15.97:0/3314055032 wait complete. 2024-03-11T19:55:46.072 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2620569603' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-11T19:55:46.072 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-11T19:55:46.072 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: pgmap v312: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:55:46.072 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-11T19:55:46.072 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: osdmap e83: 12 total, 12 up, 12 in 2024-03-11T19:55:46.072 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.072 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.073 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.073 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.073 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T19:55:46.073 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.073 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.073 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:55:46.073 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:55:46.073 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:55:46.073 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3314055032' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T19:55:46.074 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T19:55:46.074 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:45 smithi097 ceph-mon[31898]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-11T19:55:46.075 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-11T19:55:46.075 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-11T19:55:46.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.268+0000 7f4ca5a00640 1 -- 172.21.15.97:0/4144586880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ca0102d40 msgr2=0x7f4ca01095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:46.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.268+0000 7f4ca5a00640 1 --2- 172.21.15.97:0/4144586880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ca0102d40 0x7f4ca01095d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4c98009ca0 tx=0x7f4c9802c100 comp rx=0 tx=0).stop 2024-03-11T19:55:46.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.268+0000 7f4ca5a00640 1 -- 172.21.15.97:0/4144586880 shutdown_connections 2024-03-11T19:55:46.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.268+0000 7f4ca5a00640 1 --2- 172.21.15.97:0/4144586880 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ca0102d40 0x7f4ca01095d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.268+0000 7f4ca5a00640 1 --2- 172.21.15.97:0/4144586880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ca0102380 0x7f4ca0102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.268+0000 7f4ca5a00640 1 --2- 172.21.15.97:0/4144586880 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ca0101190 0x7f4ca01015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.268+0000 7f4ca5a00640 1 -- 172.21.15.97:0/4144586880 >> 172.21.15.97:0/4144586880 conn(0x7f4ca00fc720 msgr2=0x7f4ca00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:46.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.268+0000 7f4ca5a00640 1 -- 172.21.15.97:0/4144586880 shutdown_connections 2024-03-11T19:55:46.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.268+0000 7f4ca5a00640 1 -- 172.21.15.97:0/4144586880 wait complete. 2024-03-11T19:55:46.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.269+0000 7f4ca5a00640 1 Processor -- start 2024-03-11T19:55:46.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.269+0000 7f4ca5a00640 1 -- start start 2024-03-11T19:55:46.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4ca5a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ca0101190 0x7f4ca0199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4ca5a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ca0102380 0x7f4ca0199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4ca5a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ca0102d40 0x7f4ca01a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4ca5a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca01a0f00 con 0x7f4ca0102380 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4ca5a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca01a1070 con 0x7f4ca0101190 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4ca5a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca01a1230 con 0x7f4ca0102d40 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4c97e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ca0102380 0x7f4ca0199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4c97e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ca0102380 0x7f4ca0199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40572/0 (socket says 172.21.15.97:40572) 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4c97e00640 1 -- 172.21.15.97:0/1633040309 learned_addr learned my addr 172.21.15.97:0/1633040309 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4c9ea00640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ca0101190 0x7f4ca0199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4c9f400640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ca0102d40 0x7f4ca01a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4c97e00640 1 -- 172.21.15.97:0/1633040309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ca0102d40 msgr2=0x7f4ca01a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4c97e00640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ca0102d40 0x7f4ca01a09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4c97e00640 1 -- 172.21.15.97:0/1633040309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ca0101190 msgr2=0x7f4ca0199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4c97e00640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ca0101190 0x7f4ca0199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4c97e00640 1 -- 172.21.15.97:0/1633040309 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c98009900 con 0x7f4ca0102380 2024-03-11T19:55:46.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.270+0000 7f4c9ea00640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ca0101190 0x7f4ca0199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:46.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.271+0000 7f4c97e00640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ca0102380 0x7f4ca0199950 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800e9f0 tx=0x7f4c8800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:46.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.271+0000 7f4c97400640 1 -- 172.21.15.97:0/1633040309 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88009700 con 0x7f4ca0102380 2024-03-11T19:55:46.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.271+0000 7f4ca5a00640 1 -- 172.21.15.97:0/1633040309 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca01a1510 con 0x7f4ca0102380 2024-03-11T19:55:46.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.271+0000 7f4c97400640 1 -- 172.21.15.97:0/1633040309 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c8800cd80 con 0x7f4ca0102380 2024-03-11T19:55:46.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.271+0000 7f4c97400640 1 -- 172.21.15.97:0/1633040309 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88018920 con 0x7f4ca0102380 2024-03-11T19:55:46.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.271+0000 7f4ca5a00640 1 -- 172.21.15.97:0/1633040309 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca01a1a20 con 0x7f4ca0102380 2024-03-11T19:55:46.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.272+0000 7f4ca5a00640 1 -- 172.21.15.97:0/1633040309 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca010cba0 con 0x7f4ca0102380 2024-03-11T19:55:46.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.276+0000 7f4c97400640 1 -- 172.21.15.97:0/1633040309 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4c88018a80 con 0x7f4ca0102380 2024-03-11T19:55:46.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.277+0000 7f4c97400640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c840a1a30 0x7f4c840a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:46.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.278+0000 7f4c97400640 1 -- 172.21.15.97:0/1633040309 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4c88014070 con 0x7f4ca0102380 2024-03-11T19:55:46.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.278+0000 7f4c9ea00640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c840a1a30 0x7f4c840a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:46.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.278+0000 7f4c9ea00640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c840a1a30 0x7f4c840a3ef0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4ca01021e0 tx=0x7f4c90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:46.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.279+0000 7f4c97400640 1 -- 172.21.15.97:0/1633040309 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4c88096ed0 con 0x7f4ca0102380 2024-03-11T19:55:46.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.486+0000 7f4ca5a00640 1 -- 172.21.15.97:0/1633040309 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f4ca010cd30 con 0x7f4ca0102380 2024-03-11T19:55:46.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.679+0000 7f4c97400640 1 -- 172.21.15.97:0/1633040309 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) v1 ==== 157+0+0 (secure 0 0 0) 0x7f4c8809a590 con 0x7f4ca0102380 2024-03-11T19:55:46.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.681+0000 7f4ca5a00640 1 -- 172.21.15.97:0/1633040309 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c840a1a30 msgr2=0x7f4c840a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:46.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.681+0000 7f4ca5a00640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c840a1a30 0x7f4c840a3ef0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4ca01021e0 tx=0x7f4c90005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:46.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.681+0000 7f4ca5a00640 1 -- 172.21.15.97:0/1633040309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ca0102380 msgr2=0x7f4ca0199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:46.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.681+0000 7f4ca5a00640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ca0102380 0x7f4ca0199950 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800e9f0 tx=0x7f4c8800eeb0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.682+0000 7f4ca5a00640 1 -- 172.21.15.97:0/1633040309 shutdown_connections 2024-03-11T19:55:46.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.682+0000 7f4ca5a00640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ca0102d40 0x7f4ca01a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.682+0000 7f4ca5a00640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c840a1a30 0x7f4c840a3ef0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.682+0000 7f4ca5a00640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ca0102380 0x7f4ca0199950 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.682+0000 7f4ca5a00640 1 --2- 172.21.15.97:0/1633040309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4ca0101190 0x7f4ca0199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.682+0000 7f4ca5a00640 1 -- 172.21.15.97:0/1633040309 >> 172.21.15.97:0/1633040309 conn(0x7f4ca00fc720 msgr2=0x7f4ca00733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:46.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.682+0000 7f4ca5a00640 1 -- 172.21.15.97:0/1633040309 shutdown_connections 2024-03-11T19:55:46.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.682+0000 7f4ca5a00640 1 -- 172.21.15.97:0/1633040309 wait complete. 2024-03-11T19:55:46.705 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-11T19:55:46.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.898+0000 7f44c8c00640 1 -- 172.21.15.97:0/1014400616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44c4100bc0 msgr2=0x7f44c4101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:46.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.898+0000 7f44c8c00640 1 --2- 172.21.15.97:0/1014400616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44c4100bc0 0x7f44c4101040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f44bc009ca0 tx=0x7f44bc02c180 comp rx=0 tx=0).stop 2024-03-11T19:55:46.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.899+0000 7f44c8c00640 1 -- 172.21.15.97:0/1014400616 shutdown_connections 2024-03-11T19:55:46.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.899+0000 7f44c8c00640 1 --2- 172.21.15.97:0/1014400616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44c4100bc0 0x7f44c4101040 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.899+0000 7f44c8c00640 1 --2- 172.21.15.97:0/1014400616 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44c4100220 0x7f44c4100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.899+0000 7f44c8c00640 1 --2- 172.21.15.97:0/1014400616 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44c40ff8c0 0x7f44c40ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.899+0000 7f44c8c00640 1 -- 172.21.15.97:0/1014400616 >> 172.21.15.97:0/1014400616 conn(0x7f44c40fb460 msgr2=0x7f44c40fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:46.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.899+0000 7f44c8c00640 1 -- 172.21.15.97:0/1014400616 shutdown_connections 2024-03-11T19:55:46.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.899+0000 7f44c8c00640 1 -- 172.21.15.97:0/1014400616 wait complete. 2024-03-11T19:55:46.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.899+0000 7f44c8c00640 1 Processor -- start 2024-03-11T19:55:46.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.900+0000 7f44c8c00640 1 -- start start 2024-03-11T19:55:46.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.900+0000 7f44c8c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44c40ff8c0 0x7f44c4199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.900+0000 7f44c8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44c4100220 0x7f44c4199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.900+0000 7f44c8c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44c4100bc0 0x7f44c41a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.900+0000 7f44c8c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c41a0d00 con 0x7f44c4100220 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.900+0000 7f44c8c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c41a0e70 con 0x7f44c40ff8c0 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.900+0000 7f44c8c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c41a1030 con 0x7f44c4100bc0 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.900+0000 7f44c2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44c4100220 0x7f44c4199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.900+0000 7f44c2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44c4100220 0x7f44c4199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40592/0 (socket says 172.21.15.97:40592) 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.900+0000 7f44c2a00640 1 -- 172.21.15.97:0/3766446331 learned_addr learned my addr 172.21.15.97:0/3766446331 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.900+0000 7f44c3e00640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44c4100bc0 0x7f44c41a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.900+0000 7f44c3400640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44c40ff8c0 0x7f44c4199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.901+0000 7f44c2a00640 1 -- 172.21.15.97:0/3766446331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44c4100bc0 msgr2=0x7f44c41a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.901+0000 7f44c2a00640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44c4100bc0 0x7f44c41a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.901+0000 7f44c2a00640 1 -- 172.21.15.97:0/3766446331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44c40ff8c0 msgr2=0x7f44c4199210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.901+0000 7f44c2a00640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44c40ff8c0 0x7f44c4199210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:46.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.901+0000 7f44c2a00640 1 -- 172.21.15.97:0/3766446331 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b0009490 con 0x7f44c4100220 2024-03-11T19:55:46.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.901+0000 7f44c2a00640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44c4100220 0x7f44c4199750 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f44b000ea60 tx=0x7f44b000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:46.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.901+0000 7f44bbe00640 1 -- 172.21.15.97:0/3766446331 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b000cde0 con 0x7f44c4100220 2024-03-11T19:55:46.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.901+0000 7f44c8c00640 1 -- 172.21.15.97:0/3766446331 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44bc009900 con 0x7f44c4100220 2024-03-11T19:55:46.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.901+0000 7f44bbe00640 1 -- 172.21.15.97:0/3766446331 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44b00035f0 con 0x7f44c4100220 2024-03-11T19:55:46.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.901+0000 7f44bbe00640 1 -- 172.21.15.97:0/3766446331 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0010690 con 0x7f44c4100220 2024-03-11T19:55:46.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.901+0000 7f44c8c00640 1 -- 172.21.15.97:0/3766446331 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c41a1690 con 0x7f44c4100220 2024-03-11T19:55:46.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.902+0000 7f44c8c00640 1 -- 172.21.15.97:0/3766446331 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c410a610 con 0x7f44c4100220 2024-03-11T19:55:46.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.904+0000 7f44bbe00640 1 -- 172.21.15.97:0/3766446331 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f44b0003af0 con 0x7f44c4100220 2024-03-11T19:55:46.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.906+0000 7f44bbe00640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f44a80a1a30 0x7f44a80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:46.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.906+0000 7f44bbe00640 1 -- 172.21.15.97:0/3766446331 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f44b0014070 con 0x7f44c4100220 2024-03-11T19:55:46.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.906+0000 7f44c3400640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f44a80a1a30 0x7f44a80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:46.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.907+0000 7f44c3400640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f44a80a1a30 0x7f44a80a3ef0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009bd0 tx=0x7f44b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:46.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:46.910+0000 7f44bbe00640 1 -- 172.21.15.97:0/3766446331 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f44b0096a70 con 0x7f44c4100220 2024-03-11T19:55:46.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:46 smithi114 ceph-mon[36376]: osdmap e84: 12 total, 12 up, 12 in 2024-03-11T19:55:46.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:46 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-11T19:55:46.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:46 smithi114 ceph-mon[36376]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-11T19:55:46.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:46 smithi114 ceph-mon[36376]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-11T19:55:46.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:46 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1633040309' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T19:55:47.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:46 smithi161 ceph-mon[36332]: osdmap e84: 12 total, 12 up, 12 in 2024-03-11T19:55:47.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:46 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-11T19:55:47.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:46 smithi161 ceph-mon[36332]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-11T19:55:47.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:46 smithi161 ceph-mon[36332]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-11T19:55:47.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:46 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1633040309' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T19:55:47.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:47.107+0000 7f44c8c00640 1 -- 172.21.15.97:0/3766446331 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f44c402ce40 con 0x7f44c4100220 2024-03-11T19:55:47.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:46 smithi097 ceph-mon[31898]: osdmap e84: 12 total, 12 up, 12 in 2024-03-11T19:55:47.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:46 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-11T19:55:47.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:46 smithi097 ceph-mon[31898]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-11T19:55:47.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:46 smithi097 ceph-mon[31898]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-11T19:55:47.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:46 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1633040309' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T19:55:47.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:47 smithi097 ceph-mon[31898]: pgmap v315: 97 pgs: 11 creating+peering, 20 active+clean, 66 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-03-11T19:55:47.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:47 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1633040309' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-11T19:55:47.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:47 smithi097 ceph-mon[31898]: osdmap e85: 12 total, 12 up, 12 in 2024-03-11T19:55:47.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:47 smithi097 ceph-mon[31898]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T19:55:47.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:47 smithi097 ceph-mon[31898]: Cluster is now healthy 2024-03-11T19:55:47.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:47 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] up:active 2024-03-11T19:55:47.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:47 smithi097 ceph-mon[31898]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:55:47.920 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:47 smithi097 ceph-mon[31898]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:55:47.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:47 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3766446331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-11T19:55:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:47 smithi114 ceph-mon[36376]: pgmap v315: 97 pgs: 11 creating+peering, 20 active+clean, 66 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-03-11T19:55:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:47 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1633040309' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-11T19:55:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:47 smithi114 ceph-mon[36376]: osdmap e85: 12 total, 12 up, 12 in 2024-03-11T19:55:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:47 smithi114 ceph-mon[36376]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T19:55:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:47 smithi114 ceph-mon[36376]: Cluster is now healthy 2024-03-11T19:55:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:47 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] up:active 2024-03-11T19:55:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:47 smithi114 ceph-mon[36376]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:55:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:47 smithi114 ceph-mon[36376]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:55:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:47 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3766446331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-11T19:55:48.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:47 smithi161 ceph-mon[36332]: pgmap v315: 97 pgs: 11 creating+peering, 20 active+clean, 66 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-03-11T19:55:48.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:47 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1633040309' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-11T19:55:48.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:47 smithi161 ceph-mon[36332]: osdmap e85: 12 total, 12 up, 12 in 2024-03-11T19:55:48.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:47 smithi161 ceph-mon[36332]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T19:55:48.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:47 smithi161 ceph-mon[36332]: Cluster is now healthy 2024-03-11T19:55:48.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:47 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] up:active 2024-03-11T19:55:48.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:47 smithi161 ceph-mon[36332]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:55:48.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:47 smithi161 ceph-mon[36332]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:55:48.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:47 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3766446331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-11T19:55:49.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.066+0000 7f44bbe00640 1 -- 172.21.15.97:0/3766446331 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) v1 ==== 228+0+0 (secure 0 0 0) 0x7f44b009a130 con 0x7f44c4100220 2024-03-11T19:55:49.069 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data_ec' created 2024-03-11T19:55:49.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.071+0000 7f44c8c00640 1 -- 172.21.15.97:0/3766446331 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f44a80a1a30 msgr2=0x7f44a80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:49.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.071+0000 7f44c8c00640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f44a80a1a30 0x7f44a80a3ef0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009bd0 tx=0x7f44b4005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:49.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.071+0000 7f44c8c00640 1 -- 172.21.15.97:0/3766446331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44c4100220 msgr2=0x7f44c4199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:49.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.071+0000 7f44c8c00640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44c4100220 0x7f44c4199750 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f44b000ea60 tx=0x7f44b000ef20 comp rx=0 tx=0).stop 2024-03-11T19:55:49.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.071+0000 7f44c8c00640 1 -- 172.21.15.97:0/3766446331 shutdown_connections 2024-03-11T19:55:49.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.071+0000 7f44c8c00640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44c4100bc0 0x7f44c41a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:49.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.071+0000 7f44c8c00640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f44a80a1a30 0x7f44a80a3ef0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:49.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.071+0000 7f44c8c00640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44c4100220 0x7f44c4199750 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:49.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.071+0000 7f44c8c00640 1 --2- 172.21.15.97:0/3766446331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44c40ff8c0 0x7f44c4199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:49.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.071+0000 7f44c8c00640 1 -- 172.21.15.97:0/3766446331 >> 172.21.15.97:0/3766446331 conn(0x7f44c40fb460 msgr2=0x7f44c40fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:49.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.072+0000 7f44c8c00640 1 -- 172.21.15.97:0/3766446331 shutdown_connections 2024-03-11T19:55:49.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.072+0000 7f44c8c00640 1 -- 172.21.15.97:0/3766446331 wait complete. 2024-03-11T19:55:49.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-11T19:55:49.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.292+0000 7fed9f600640 1 -- 172.21.15.97:0/1335018798 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed981089b0 msgr2=0x7fed98108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:49.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.292+0000 7fed9f600640 1 --2- 172.21.15.97:0/1335018798 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed981089b0 0x7fed98108e30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fed80009900 tx=0x7fed8002bc50 comp rx=0 tx=0).stop 2024-03-11T19:55:49.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.292+0000 7fed9f600640 1 -- 172.21.15.97:0/1335018798 shutdown_connections 2024-03-11T19:55:49.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.292+0000 7fed9f600640 1 --2- 172.21.15.97:0/1335018798 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed98109370 0x7fed9810fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:49.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.292+0000 7fed9f600640 1 --2- 172.21.15.97:0/1335018798 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed981089b0 0x7fed98108e30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:49.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.292+0000 7fed9f600640 1 --2- 172.21.15.97:0/1335018798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed981077c0 0x7fed98107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:49.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.292+0000 7fed9f600640 1 -- 172.21.15.97:0/1335018798 >> 172.21.15.97:0/1335018798 conn(0x7fed98076770 msgr2=0x7fed98078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:49.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.292+0000 7fed9f600640 1 -- 172.21.15.97:0/1335018798 shutdown_connections 2024-03-11T19:55:49.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.292+0000 7fed9f600640 1 -- 172.21.15.97:0/1335018798 wait complete. 2024-03-11T19:55:49.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.293+0000 7fed9f600640 1 Processor -- start 2024-03-11T19:55:49.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.293+0000 7fed9f600640 1 -- start start 2024-03-11T19:55:49.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9f600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed981077c0 0x7fed9819d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:49.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9f600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed981089b0 0x7fed9819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:49.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9f600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed98109370 0x7fed981a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:49.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9f600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed981a5300 con 0x7fed98109370 2024-03-11T19:55:49.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9f600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed981a5470 con 0x7fed981077c0 2024-03-11T19:55:49.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9f600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed981a5630 con 0x7fed981089b0 2024-03-11T19:55:49.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed98109370 0x7fed981a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:49.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed98109370 0x7fed981a4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40614/0 (socket says 172.21.15.97:40614) 2024-03-11T19:55:49.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9d600640 1 -- 172.21.15.97:0/3720300711 learned_addr learned my addr 172.21.15.97:0/3720300711 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:49.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9cc00640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed981077c0 0x7fed9819d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:49.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed8fe00640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed981089b0 0x7fed9819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:49.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9cc00640 1 -- 172.21.15.97:0/3720300711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed981089b0 msgr2=0x7fed9819dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:49.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9cc00640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed981089b0 0x7fed9819dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:49.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9cc00640 1 -- 172.21.15.97:0/3720300711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed98109370 msgr2=0x7fed981a4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:49.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9cc00640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed98109370 0x7fed981a4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:49.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.294+0000 7fed9cc00640 1 -- 172.21.15.97:0/3720300711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed80009560 con 0x7fed981077c0 2024-03-11T19:55:49.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.295+0000 7fed9cc00640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed981077c0 0x7fed9819d860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fed8800e9c0 tx=0x7fed8800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:49.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.295+0000 7fed96000640 1 -- 172.21.15.97:0/3720300711 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8800ccc0 con 0x7fed981077c0 2024-03-11T19:55:49.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.295+0000 7fed9f600640 1 -- 172.21.15.97:0/3720300711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed981a5910 con 0x7fed981077c0 2024-03-11T19:55:49.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.295+0000 7fed96000640 1 -- 172.21.15.97:0/3720300711 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed8800ce20 con 0x7fed981077c0 2024-03-11T19:55:49.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.295+0000 7fed96000640 1 -- 172.21.15.97:0/3720300711 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed88010430 con 0x7fed981077c0 2024-03-11T19:55:49.300 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.295+0000 7fed9f600640 1 -- 172.21.15.97:0/3720300711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed981a5df0 con 0x7fed981077c0 2024-03-11T19:55:49.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.296+0000 7fed9f600640 1 -- 172.21.15.97:0/3720300711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed9810eb60 con 0x7fed981077c0 2024-03-11T19:55:49.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.299+0000 7fed96000640 1 -- 172.21.15.97:0/3720300711 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fed88003f60 con 0x7fed981077c0 2024-03-11T19:55:49.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.299+0000 7fed96000640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fed7c0a1a30 0x7fed7c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:49.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.299+0000 7fed96000640 1 -- 172.21.15.97:0/3720300711 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fed88014070 con 0x7fed981077c0 2024-03-11T19:55:49.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.299+0000 7fed8fe00640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fed7c0a1a30 0x7fed7c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:49.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.300+0000 7fed8fe00640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fed7c0a1a30 0x7fed7c0a3ef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed80009230 tx=0x7fed80005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:49.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.303+0000 7fed96000640 1 -- 172.21.15.97:0/3720300711 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fed88096a90 con 0x7fed981077c0 2024-03-11T19:55:49.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:49 smithi097 ceph-mon[31898]: pgmap v318: 97 pgs: 11 creating+peering, 72 active+clean, 14 unknown; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-03-11T19:55:49.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:49 smithi097 ceph-mon[31898]: osdmap e86: 12 total, 12 up, 12 in 2024-03-11T19:55:49.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:49 smithi114 ceph-mon[36376]: pgmap v318: 97 pgs: 11 creating+peering, 72 active+clean, 14 unknown; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-03-11T19:55:49.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:49 smithi114 ceph-mon[36376]: osdmap e86: 12 total, 12 up, 12 in 2024-03-11T19:55:49.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:49.497+0000 7fed9f600640 1 -- 172.21.15.97:0/3720300711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fed981131a0 con 0x7fed981077c0 2024-03-11T19:55:49.537 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:49 smithi161 ceph-mon[36332]: pgmap v318: 97 pgs: 11 creating+peering, 72 active+clean, 14 unknown; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-03-11T19:55:49.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:49 smithi161 ceph-mon[36332]: osdmap e86: 12 total, 12 up, 12 in 2024-03-11T19:55:49.992 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 19:55:49 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b[66693]: 2024-03-11T19:55:49.563+0000 7f7bc72f8700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-11T19:55:50.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.075+0000 7fed96000640 1 -- 172.21.15.97:0/3720300711 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7fed8809a150 con 0x7fed981077c0 2024-03-11T19:55:50.078 INFO:teuthology.orchestra.run.smithi097.stderr:set pool 4 allow_ec_overwrites to true 2024-03-11T19:55:50.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.081+0000 7fed9f600640 1 -- 172.21.15.97:0/3720300711 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fed7c0a1a30 msgr2=0x7fed7c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:50.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.081+0000 7fed9f600640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fed7c0a1a30 0x7fed7c0a3ef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed80009230 tx=0x7fed80005fd0 comp rx=0 tx=0).stop 2024-03-11T19:55:50.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.081+0000 7fed9f600640 1 -- 172.21.15.97:0/3720300711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed981077c0 msgr2=0x7fed9819d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:50.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.081+0000 7fed9f600640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed981077c0 0x7fed9819d860 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fed8800e9c0 tx=0x7fed8800ee80 comp rx=0 tx=0).stop 2024-03-11T19:55:50.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.083+0000 7fed9f600640 1 -- 172.21.15.97:0/3720300711 shutdown_connections 2024-03-11T19:55:50.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.083+0000 7fed9f600640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fed7c0a1a30 0x7fed7c0a3ef0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:50.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.083+0000 7fed9f600640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fed98109370 0x7fed981a4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:50.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.083+0000 7fed9f600640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed981089b0 0x7fed9819dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:50.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.083+0000 7fed9f600640 1 --2- 172.21.15.97:0/3720300711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed981077c0 0x7fed9819d860 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:50.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.083+0000 7fed9f600640 1 -- 172.21.15.97:0/3720300711 >> 172.21.15.97:0/3720300711 conn(0x7fed98076770 msgr2=0x7fed9810bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:50.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.083+0000 7fed9f600640 1 -- 172.21.15.97:0/3720300711 shutdown_connections 2024-03-11T19:55:50.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.083+0000 7fed9f600640 1 -- 172.21.15.97:0/3720300711 wait complete. 2024-03-11T19:55:50.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-11T19:55:50.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.305+0000 7fa6d2000640 1 -- 172.21.15.97:0/1104488562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6cc1077a0 msgr2=0x7fa6cc107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:50.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.305+0000 7fa6d2000640 1 --2- 172.21.15.97:0/1104488562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6cc1077a0 0x7fa6cc107bc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009900 tx=0x7fa6bc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:50.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.305+0000 7fa6d2000640 1 -- 172.21.15.97:0/1104488562 shutdown_connections 2024-03-11T19:55:50.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.305+0000 7fa6d2000640 1 --2- 172.21.15.97:0/1104488562 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6cc109350 0x7fa6cc10fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:50.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.305+0000 7fa6d2000640 1 --2- 172.21.15.97:0/1104488562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6cc108990 0x7fa6cc108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:50.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.305+0000 7fa6d2000640 1 --2- 172.21.15.97:0/1104488562 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6cc1077a0 0x7fa6cc107bc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:50.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.305+0000 7fa6d2000640 1 -- 172.21.15.97:0/1104488562 >> 172.21.15.97:0/1104488562 conn(0x7fa6cc076740 msgr2=0x7fa6cc078b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:50.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.305+0000 7fa6d2000640 1 -- 172.21.15.97:0/1104488562 shutdown_connections 2024-03-11T19:55:50.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.305+0000 7fa6d2000640 1 -- 172.21.15.97:0/1104488562 wait complete. 2024-03-11T19:55:50.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.306+0000 7fa6d2000640 1 Processor -- start 2024-03-11T19:55:50.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.306+0000 7fa6d2000640 1 -- start start 2024-03-11T19:55:50.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6d2000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6cc1077a0 0x7fa6cc19d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:50.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6d2000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6cc108990 0x7fa6cc19dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:50.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6d2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6cc109350 0x7fa6cc1a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6d2000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc1a5330 con 0x7fa6cc1077a0 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6d2000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc1a54a0 con 0x7fa6cc109350 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6d2000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc1a5660 con 0x7fa6cc108990 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6cb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6cc1077a0 0x7fa6cc19d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6cb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6cc1077a0 0x7fa6cc19d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40636/0 (socket says 172.21.15.97:40636) 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6cb400640 1 -- 172.21.15.97:0/369935034 learned_addr learned my addr 172.21.15.97:0/369935034 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6c2a00640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6cc108990 0x7fa6cc19dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6cbe00640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6cc109350 0x7fa6cc1a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6cb400640 1 -- 172.21.15.97:0/369935034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6cc108990 msgr2=0x7fa6cc19dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6cb400640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6cc108990 0x7fa6cc19dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6cb400640 1 -- 172.21.15.97:0/369935034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6cc109350 msgr2=0x7fa6cc1a4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6cb400640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6cc109350 0x7fa6cc1a4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6cb400640 1 -- 172.21.15.97:0/369935034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b4009490 con 0x7fa6cc1077a0 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.307+0000 7fa6c2a00640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6cc108990 0x7fa6cc19dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:50.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.308+0000 7fa6cb400640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6cc1077a0 0x7fa6cc19d840 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc005ab0 tx=0x7fa6bc005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:50.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.308+0000 7fa6c8c00640 1 -- 172.21.15.97:0/369935034 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc0043d0 con 0x7fa6cc1077a0 2024-03-11T19:55:50.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.308+0000 7fa6d2000640 1 -- 172.21.15.97:0/369935034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6bc009560 con 0x7fa6cc1077a0 2024-03-11T19:55:50.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.308+0000 7fa6c8c00640 1 -- 172.21.15.97:0/369935034 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6bc0052d0 con 0x7fa6cc1077a0 2024-03-11T19:55:50.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.308+0000 7fa6c8c00640 1 -- 172.21.15.97:0/369935034 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc003d40 con 0x7fa6cc1077a0 2024-03-11T19:55:50.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.308+0000 7fa6d2000640 1 -- 172.21.15.97:0/369935034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6cc1a5c60 con 0x7fa6cc1077a0 2024-03-11T19:55:50.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.309+0000 7fa6d2000640 1 -- 172.21.15.97:0/369935034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6cc10eac0 con 0x7fa6cc1077a0 2024-03-11T19:55:50.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.311+0000 7fa6c8c00640 1 -- 172.21.15.97:0/369935034 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa6bc005440 con 0x7fa6cc1077a0 2024-03-11T19:55:50.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.312+0000 7fa6c8c00640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa6b00a1a30 0x7fa6b00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:50.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.313+0000 7fa6c8c00640 1 -- 172.21.15.97:0/369935034 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fa6bc0efd50 con 0x7fa6cc1077a0 2024-03-11T19:55:50.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.313+0000 7fa6c2a00640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa6b00a1a30 0x7fa6b00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:50.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.313+0000 7fa6c2a00640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa6b00a1a30 0x7fa6b00a3ef0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc19ed50 tx=0x7fa6b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:50.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.315+0000 7fa6c8c00640 1 -- 172.21.15.97:0/369935034 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa6bc0b8ce0 con 0x7fa6cc1077a0 2024-03-11T19:55:50.419 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:50 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3766446331' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-11T19:55:50.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:50 smithi097 ceph-mon[31898]: osdmap e87: 12 total, 12 up, 12 in 2024-03-11T19:55:50.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:50 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3720300711' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-11T19:55:50.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:50 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-11T19:55:50.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:50 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3766446331' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-11T19:55:50.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:50 smithi114 ceph-mon[36376]: osdmap e87: 12 total, 12 up, 12 in 2024-03-11T19:55:50.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:50 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3720300711' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-11T19:55:50.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:50 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-11T19:55:50.537 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:50 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3766446331' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-11T19:55:50.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:50 smithi161 ceph-mon[36332]: osdmap e87: 12 total, 12 up, 12 in 2024-03-11T19:55:50.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:50 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3720300711' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-11T19:55:50.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:50 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-11T19:55:50.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:50.539+0000 7fa6d2000640 1 -- 172.21.15.97:0/369935034 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fa6cc1131a0 con 0x7fa6cc1077a0 2024-03-11T19:55:51.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.107+0000 7fa6c8c00640 1 -- 172.21.15.97:0/369935034 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7fa6bc0430b0 con 0x7fa6cc1077a0 2024-03-11T19:55:51.110 INFO:teuthology.orchestra.run.smithi097.stderr:added data pool 4 to fsmap 2024-03-11T19:55:51.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.111+0000 7fa6d2000640 1 -- 172.21.15.97:0/369935034 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa6b00a1a30 msgr2=0x7fa6b00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:51.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.111+0000 7fa6d2000640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa6b00a1a30 0x7fa6b00a3ef0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc19ed50 tx=0x7fa6b4005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:51.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.111+0000 7fa6d2000640 1 -- 172.21.15.97:0/369935034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6cc1077a0 msgr2=0x7fa6cc19d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:51.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.111+0000 7fa6d2000640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6cc1077a0 0x7fa6cc19d840 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc005ab0 tx=0x7fa6bc005e50 comp rx=0 tx=0).stop 2024-03-11T19:55:51.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.112+0000 7fa6d2000640 1 -- 172.21.15.97:0/369935034 shutdown_connections 2024-03-11T19:55:51.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.112+0000 7fa6d2000640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6cc109350 0x7fa6cc1a4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.112+0000 7fa6d2000640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6cc108990 0x7fa6cc19dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.112+0000 7fa6d2000640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa6b00a1a30 0x7fa6b00a3ef0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.112+0000 7fa6d2000640 1 --2- 172.21.15.97:0/369935034 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa6cc1077a0 0x7fa6cc19d840 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.112+0000 7fa6d2000640 1 -- 172.21.15.97:0/369935034 >> 172.21.15.97:0/369935034 conn(0x7fa6cc076740 msgr2=0x7fa6cc10bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:51.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.112+0000 7fa6d2000640 1 -- 172.21.15.97:0/369935034 shutdown_connections 2024-03-11T19:55:51.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.112+0000 7fa6d2000640 1 -- 172.21.15.97:0/369935034 wait complete. 2024-03-11T19:55:51.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:51.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.324+0000 7f0bbc600640 1 -- 172.21.15.97:0/3944658988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb4109370 msgr2=0x7f0bb410fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:51.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.324+0000 7f0bbc600640 1 --2- 172.21.15.97:0/3944658988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb4109370 0x7f0bb410fc00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0bac009ca0 tx=0x7f0bac02c0f0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.324+0000 7f0bbc600640 1 -- 172.21.15.97:0/3944658988 shutdown_connections 2024-03-11T19:55:51.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.324+0000 7f0bbc600640 1 --2- 172.21.15.97:0/3944658988 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb4109370 0x7f0bb410fc00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.324+0000 7f0bbc600640 1 --2- 172.21.15.97:0/3944658988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb41089b0 0x7f0bb4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.324+0000 7f0bbc600640 1 --2- 172.21.15.97:0/3944658988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bb41077c0 0x7f0bb4107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.324+0000 7f0bbc600640 1 -- 172.21.15.97:0/3944658988 >> 172.21.15.97:0/3944658988 conn(0x7f0bb4076790 msgr2=0x7f0bb4078bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.325+0000 7f0bbc600640 1 -- 172.21.15.97:0/3944658988 shutdown_connections 2024-03-11T19:55:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.325+0000 7f0bbc600640 1 -- 172.21.15.97:0/3944658988 wait complete. 2024-03-11T19:55:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.325+0000 7f0bbc600640 1 Processor -- start 2024-03-11T19:55:51.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bbc600640 1 -- start start 2024-03-11T19:55:51.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bbc600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bb41077c0 0x7f0bb419d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:51.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bbc600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb41089b0 0x7f0bb419dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:51.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bbc600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb4109370 0x7f0bb41a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:51.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bbc600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41a52d0 con 0x7f0bb41089b0 2024-03-11T19:55:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bbc600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41a5440 con 0x7f0bb4109370 2024-03-11T19:55:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bbc600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41a5600 con 0x7f0bb41077c0 2024-03-11T19:55:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bb9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb41089b0 0x7f0bb419dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bb9200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb41089b0 0x7f0bb419dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40662/0 (socket says 172.21.15.97:40662) 2024-03-11T19:55:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bb9200640 1 -- 172.21.15.97:0/522254885 learned_addr learned my addr 172.21.15.97:0/522254885 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bb9c00640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bb41077c0 0x7f0bb419d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bba600640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb4109370 0x7f0bb41a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bb9200640 1 -- 172.21.15.97:0/522254885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bb41077c0 msgr2=0x7f0bb419d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bb9200640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bb41077c0 0x7f0bb419d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bb9200640 1 -- 172.21.15.97:0/522254885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb4109370 msgr2=0x7f0bb41a4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bb9200640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb4109370 0x7f0bb41a4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.326+0000 7f0bb9200640 1 -- 172.21.15.97:0/522254885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8009490 con 0x7f0bb41089b0 2024-03-11T19:55:51.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.327+0000 7f0bb9200640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb41089b0 0x7f0bb419dd20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0ba000d950 tx=0x7f0ba000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:51.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.327+0000 7f0bb2000640 1 -- 172.21.15.97:0/522254885 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba0009a30 con 0x7f0bb41089b0 2024-03-11T19:55:51.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.327+0000 7f0bbc600640 1 -- 172.21.15.97:0/522254885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bac009900 con 0x7f0bb41089b0 2024-03-11T19:55:51.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.327+0000 7f0bb2000640 1 -- 172.21.15.97:0/522254885 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ba0009b90 con 0x7f0bb41089b0 2024-03-11T19:55:51.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.327+0000 7f0bb2000640 1 -- 172.21.15.97:0/522254885 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba0004a00 con 0x7f0bb41089b0 2024-03-11T19:55:51.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.327+0000 7f0bbc600640 1 -- 172.21.15.97:0/522254885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb41a5c60 con 0x7f0bb41089b0 2024-03-11T19:55:51.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.328+0000 7f0bbc600640 1 -- 172.21.15.97:0/522254885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb410eae0 con 0x7f0bb41089b0 2024-03-11T19:55:51.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.331+0000 7f0bb2000640 1 -- 172.21.15.97:0/522254885 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0ba0009d00 con 0x7f0bb41089b0 2024-03-11T19:55:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.332+0000 7f0bb2000640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0b9c0a1a30 0x7f0b9c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.332+0000 7f0bb2000640 1 -- 172.21.15.97:0/522254885 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0ba00cd0d0 con 0x7f0bb41089b0 2024-03-11T19:55:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.332+0000 7f0bb9c00640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0b9c0a1a30 0x7f0b9c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:51.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.333+0000 7f0bb9c00640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0b9c0a1a30 0x7f0b9c0a3ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4108810 tx=0x7f0ba8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:51.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.336+0000 7f0bb2000640 1 -- 172.21.15.97:0/522254885 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0ba0096040 con 0x7f0bb41089b0 2024-03-11T19:55:51.419 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:51 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-11T19:55:51.419 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:51 smithi097 ceph-mon[31898]: osdmap e88: 12 total, 12 up, 12 in 2024-03-11T19:55:51.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:51 smithi097 ceph-mon[31898]: pgmap v321: 129 pgs: 4 creating+peering, 97 active+clean, 28 unknown; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2024-03-11T19:55:51.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:51 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/369935034' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-11T19:55:51.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:51 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-11T19:55:51.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:51 smithi114 ceph-mon[36376]: osdmap e88: 12 total, 12 up, 12 in 2024-03-11T19:55:51.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:51 smithi114 ceph-mon[36376]: pgmap v321: 129 pgs: 4 creating+peering, 97 active+clean, 28 unknown; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2024-03-11T19:55:51.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:51 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/369935034' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-11T19:55:51.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:51 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-11T19:55:51.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:51 smithi161 ceph-mon[36332]: osdmap e88: 12 total, 12 up, 12 in 2024-03-11T19:55:51.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:51 smithi161 ceph-mon[36332]: pgmap v321: 129 pgs: 4 creating+peering, 97 active+clean, 28 unknown; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2024-03-11T19:55:51.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:51 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/369935034' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-11T19:55:51.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.556+0000 7f0bbc600640 1 -- 172.21.15.97:0/522254885 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bb41131a0 con 0x7f0bb41089b0 2024-03-11T19:55:51.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.558+0000 7f0bb2000640 1 -- 172.21.15.97:0/522254885 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11289 (secure 0 0 0) 0x7f0ba0099700 con 0x7f0bb41089b0 2024-03-11T19:55:51.560 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:51.560 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24499,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6841/2531110572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2531110572},{"type":"v1","addr":"172.21.15.97:6841","nonce":2531110572}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-11T19:55:45.149335+0000","modified":"2024-03-11T19:55:50.541218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6839/1624532248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":1624532248},{"type":"v1","addr":"172.21.15.114:6839","nonce":1624532248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-11T19:55:51.560 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 17 2024-03-11T19:55:51.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.561+0000 7f0bbc600640 1 -- 172.21.15.97:0/522254885 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0b9c0a1a30 msgr2=0x7f0b9c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:51.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.561+0000 7f0bbc600640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0b9c0a1a30 0x7f0b9c0a3ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4108810 tx=0x7f0ba8005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:51.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.561+0000 7f0bbc600640 1 -- 172.21.15.97:0/522254885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb41089b0 msgr2=0x7f0bb419dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:51.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.561+0000 7f0bbc600640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb41089b0 0x7f0bb419dd20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0ba000d950 tx=0x7f0ba000de10 comp rx=0 tx=0).stop 2024-03-11T19:55:51.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.561+0000 7f0bbc600640 1 -- 172.21.15.97:0/522254885 shutdown_connections 2024-03-11T19:55:51.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.561+0000 7f0bbc600640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bb4109370 0x7f0bb41a4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.561+0000 7f0bbc600640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0b9c0a1a30 0x7f0b9c0a3ef0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.561+0000 7f0bbc600640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0bb41089b0 0x7f0bb419dd20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.561+0000 7f0bbc600640 1 --2- 172.21.15.97:0/522254885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0bb41077c0 0x7f0bb419d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.561+0000 7f0bbc600640 1 -- 172.21.15.97:0/522254885 >> 172.21.15.97:0/522254885 conn(0x7f0bb4076790 msgr2=0x7f0bb410bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:51.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.561+0000 7f0bbc600640 1 -- 172.21.15.97:0/522254885 shutdown_connections 2024-03-11T19:55:51.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.561+0000 7f0bbc600640 1 -- 172.21.15.97:0/522254885 wait complete. 2024-03-11T19:55:51.580 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:55:51.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.771+0000 7f59cce00640 1 -- 172.21.15.97:0/1746137314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c81089b0 msgr2=0x7f59c8108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:51.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.771+0000 7f59cce00640 1 --2- 172.21.15.97:0/1746137314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c81089b0 0x7f59c8108e30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f59b0009900 tx=0x7f59b002bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:51.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.771+0000 7f59cce00640 1 -- 172.21.15.97:0/1746137314 shutdown_connections 2024-03-11T19:55:51.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.771+0000 7f59cce00640 1 --2- 172.21.15.97:0/1746137314 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59c8109370 0x7f59c810fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.771+0000 7f59cce00640 1 --2- 172.21.15.97:0/1746137314 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c81089b0 0x7f59c8108e30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.771+0000 7f59cce00640 1 --2- 172.21.15.97:0/1746137314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59c81077c0 0x7f59c8107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.771+0000 7f59cce00640 1 -- 172.21.15.97:0/1746137314 >> 172.21.15.97:0/1746137314 conn(0x7f59c8076790 msgr2=0x7f59c8078bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.771+0000 7f59cce00640 1 -- 172.21.15.97:0/1746137314 shutdown_connections 2024-03-11T19:55:51.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.771+0000 7f59cce00640 1 -- 172.21.15.97:0/1746137314 wait complete. 2024-03-11T19:55:51.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.772+0000 7f59cce00640 1 Processor -- start 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.772+0000 7f59cce00640 1 -- start start 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.772+0000 7f59cce00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59c81077c0 0x7f59c819d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.772+0000 7f59cce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59c81089b0 0x7f59c819dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.772+0000 7f59cce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8109370 0x7f59c81a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.772+0000 7f59cce00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c81a5300 con 0x7f59c8109370 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.772+0000 7f59cce00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c81a5470 con 0x7f59c81089b0 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.772+0000 7f59cce00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c81a5630 con 0x7f59c81077c0 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.773+0000 7f59c6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8109370 0x7f59c81a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.773+0000 7f59c6800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8109370 0x7f59c81a4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40678/0 (socket says 172.21.15.97:40678) 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.773+0000 7f59bd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59c81089b0 0x7f59c819dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.773+0000 7f59c5e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59c81077c0 0x7f59c819d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.773+0000 7f59c6800640 1 -- 172.21.15.97:0/923592692 learned_addr learned my addr 172.21.15.97:0/923592692 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.773+0000 7f59c6800640 1 -- 172.21.15.97:0/923592692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59c81077c0 msgr2=0x7f59c819d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:51.779 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.773+0000 7f59c6800640 1 --2- 172.21.15.97:0/923592692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59c81077c0 0x7f59c819d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.773+0000 7f59c6800640 1 -- 172.21.15.97:0/923592692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59c81089b0 msgr2=0x7f59c819dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.773+0000 7f59c6800640 1 --2- 172.21.15.97:0/923592692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59c81089b0 0x7f59c819dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.773+0000 7f59c6800640 1 -- 172.21.15.97:0/923592692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b0009560 con 0x7f59c8109370 2024-03-11T19:55:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.773+0000 7f59c6800640 1 --2- 172.21.15.97:0/923592692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8109370 0x7f59c81a4dc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f59c000eb10 tx=0x7f59c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.774+0000 7f59bf400640 1 -- 172.21.15.97:0/923592692 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0004b20 con 0x7f59c8109370 2024-03-11T19:55:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.774+0000 7f59bf400640 1 -- 172.21.15.97:0/923592692 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59c0004c80 con 0x7f59c8109370 2024-03-11T19:55:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.774+0000 7f59cce00640 1 -- 172.21.15.97:0/923592692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c81a5910 con 0x7f59c8109370 2024-03-11T19:55:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.774+0000 7f59bf400640 1 -- 172.21.15.97:0/923592692 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0010650 con 0x7f59c8109370 2024-03-11T19:55:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.774+0000 7f59cce00640 1 -- 172.21.15.97:0/923592692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c81a5e40 con 0x7f59c8109370 2024-03-11T19:55:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.775+0000 7f59cce00640 1 -- 172.21.15.97:0/923592692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c810eae0 con 0x7f59c8109370 2024-03-11T19:55:51.780 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.777+0000 7f59bf400640 1 -- 172.21.15.97:0/923592692 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f59c0005020 con 0x7f59c8109370 2024-03-11T19:55:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.779+0000 7f59bf400640 1 --2- 172.21.15.97:0/923592692 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f59ac0a1ab0 0x7f59ac0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.779+0000 7f59bf400640 1 -- 172.21.15.97:0/923592692 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f59c0014070 con 0x7f59c8109370 2024-03-11T19:55:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.779+0000 7f59c5e00640 1 --2- 172.21.15.97:0/923592692 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f59ac0a1ab0 0x7f59ac0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:51.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.780+0000 7f59c5e00640 1 --2- 172.21.15.97:0/923592692 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f59ac0a1ab0 0x7f59ac0a3f70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f59c8108810 tx=0x7f59b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.783+0000 7f59bf400640 1 -- 172.21.15.97:0/923592692 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f59c0096af0 con 0x7f59c8109370 2024-03-11T19:55:51.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.981+0000 7f59cce00640 1 -- 172.21.15.97:0/923592692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59c81131c0 con 0x7f59c8109370 2024-03-11T19:55:51.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.985+0000 7f59bf400640 1 -- 172.21.15.97:0/923592692 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24152 (secure 0 0 0) 0x7f59c009a1b0 con 0x7f59c8109370 2024-03-11T19:55:51.988 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:51.988 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":89,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:55:50.540865+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:55:51.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.988+0000 7f59cce00640 1 -- 172.21.15.97:0/923592692 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f59ac0a1ab0 msgr2=0x7f59ac0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:51.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.988+0000 7f59cce00640 1 --2- 172.21.15.97:0/923592692 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f59ac0a1ab0 0x7f59ac0a3f70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f59c8108810 tx=0x7f59b8005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:51.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.988+0000 7f59cce00640 1 -- 172.21.15.97:0/923592692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8109370 msgr2=0x7f59c81a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:51.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.988+0000 7f59cce00640 1 --2- 172.21.15.97:0/923592692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8109370 0x7f59c81a4dc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f59c000eb10 tx=0x7f59c000c2d0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.988+0000 7f59cce00640 1 -- 172.21.15.97:0/923592692 shutdown_connections 2024-03-11T19:55:51.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.988+0000 7f59cce00640 1 --2- 172.21.15.97:0/923592692 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f59ac0a1ab0 0x7f59ac0a3f70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.988+0000 7f59cce00640 1 --2- 172.21.15.97:0/923592692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f59c8109370 0x7f59c81a4dc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.988+0000 7f59cce00640 1 --2- 172.21.15.97:0/923592692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59c81089b0 0x7f59c819dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.988+0000 7f59cce00640 1 --2- 172.21.15.97:0/923592692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f59c81077c0 0x7f59c819d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:51.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.988+0000 7f59cce00640 1 -- 172.21.15.97:0/923592692 >> 172.21.15.97:0/923592692 conn(0x7f59c8076790 msgr2=0x7f59c810bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:51.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.988+0000 7f59cce00640 1 -- 172.21.15.97:0/923592692 shutdown_connections 2024-03-11T19:55:51.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:51.988+0000 7f59cce00640 1 -- 172.21.15.97:0/923592692 wait complete. 2024-03-11T19:55:52.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:55:52.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.218+0000 7f4061e00640 1 -- 172.21.15.97:0/2580835991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f405c1077c0 msgr2=0x7f405c107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:52.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.218+0000 7f4061e00640 1 --2- 172.21.15.97:0/2580835991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f405c1077c0 0x7f405c107be0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f404c009860 tx=0x7f404c02bb70 comp rx=0 tx=0).stop 2024-03-11T19:55:52.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.219+0000 7f4061e00640 1 -- 172.21.15.97:0/2580835991 shutdown_connections 2024-03-11T19:55:52.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.219+0000 7f4061e00640 1 --2- 172.21.15.97:0/2580835991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f405c109370 0x7f405c10fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.219+0000 7f4061e00640 1 --2- 172.21.15.97:0/2580835991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c1089b0 0x7f405c108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.219+0000 7f4061e00640 1 --2- 172.21.15.97:0/2580835991 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f405c1077c0 0x7f405c107be0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.219+0000 7f4061e00640 1 -- 172.21.15.97:0/2580835991 >> 172.21.15.97:0/2580835991 conn(0x7f405c076790 msgr2=0x7f405c078bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:52.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.219+0000 7f4061e00640 1 -- 172.21.15.97:0/2580835991 shutdown_connections 2024-03-11T19:55:52.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.219+0000 7f4061e00640 1 -- 172.21.15.97:0/2580835991 wait complete. 2024-03-11T19:55:52.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.220+0000 7f4061e00640 1 Processor -- start 2024-03-11T19:55:52.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.220+0000 7f4061e00640 1 -- start start 2024-03-11T19:55:52.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.220+0000 7f4061e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c1077c0 0x7f405c19d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.220+0000 7f4061e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f405c1089b0 0x7f405c19ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.220+0000 7f4061e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f405c109370 0x7f405c1a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.220+0000 7f4061e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c1a5360 con 0x7f405c1089b0 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.220+0000 7f4061e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c1a54d0 con 0x7f405c109370 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.220+0000 7f4061e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c1a5690 con 0x7f405c1077c0 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.220+0000 7f405a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f405c1089b0 0x7f405c19ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.220+0000 7f405a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f405c1089b0 0x7f405c19ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40698/0 (socket says 172.21.15.97:40698) 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.220+0000 7f405a000640 1 -- 172.21.15.97:0/2626062455 learned_addr learned my addr 172.21.15.97:0/2626062455 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.220+0000 7f405aa00640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c1077c0 0x7f405c19d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.220+0000 7f405b400640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f405c109370 0x7f405c1a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.221+0000 7f405a000640 1 -- 172.21.15.97:0/2626062455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c1077c0 msgr2=0x7f405c19d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.221+0000 7f405a000640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c1077c0 0x7f405c19d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.221+0000 7f405a000640 1 -- 172.21.15.97:0/2626062455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f405c109370 msgr2=0x7f405c1a4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.221+0000 7f405a000640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f405c109370 0x7f405c1a4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.221+0000 7f405a000640 1 -- 172.21.15.97:0/2626062455 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4048009560 con 0x7f405c1089b0 2024-03-11T19:55:52.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.221+0000 7f405b400640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f405c109370 0x7f405c1a4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:55:52.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.221+0000 7f405aa00640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c1077c0 0x7f405c19d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:52.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.221+0000 7f405a000640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f405c1089b0 0x7f405c19ddb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f404800e9c0 tx=0x7f404800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:52.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.221+0000 7f4060a00640 1 -- 172.21.15.97:0/2626062455 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404800cde0 con 0x7f405c1089b0 2024-03-11T19:55:52.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.221+0000 7f4060a00640 1 -- 172.21.15.97:0/2626062455 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40480035f0 con 0x7f405c1089b0 2024-03-11T19:55:52.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.221+0000 7f4061e00640 1 -- 172.21.15.97:0/2626062455 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404c009490 con 0x7f405c1089b0 2024-03-11T19:55:52.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.221+0000 7f4060a00640 1 -- 172.21.15.97:0/2626062455 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4048010670 con 0x7f405c1089b0 2024-03-11T19:55:52.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.221+0000 7f4061e00640 1 -- 172.21.15.97:0/2626062455 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405c1a5cf0 con 0x7f405c1089b0 2024-03-11T19:55:52.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.222+0000 7f4061e00640 1 -- 172.21.15.97:0/2626062455 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405c10eae0 con 0x7f405c1089b0 2024-03-11T19:55:52.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.225+0000 7f4060a00640 1 -- 172.21.15.97:0/2626062455 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4048003ad0 con 0x7f405c1089b0 2024-03-11T19:55:52.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.226+0000 7f4060a00640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f40400a1a30 0x7f40400a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:52.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.227+0000 7f4060a00640 1 -- 172.21.15.97:0/2626062455 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4048014070 con 0x7f405c1089b0 2024-03-11T19:55:52.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.227+0000 7f405aa00640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f40400a1a30 0x7f40400a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:52.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.227+0000 7f405aa00640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f40400a1a30 0x7f40400a3ef0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f404c009830 tx=0x7f404c034c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:52.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.231+0000 7f4060a00640 1 -- 172.21.15.97:0/2626062455 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4048096af0 con 0x7f405c1089b0 2024-03-11T19:55:52.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.413+0000 7f4061e00640 1 -- 172.21.15.97:0/2626062455 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f405c113190 con 0x7f405c1089b0 2024-03-11T19:55:52.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.417+0000 7f4060a00640 1 -- 172.21.15.97:0/2626062455 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24152 (secure 0 0 0) 0x7f4048003760 con 0x7f405c1089b0 2024-03-11T19:55:52.419 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:52 smithi097 ceph-mon[31898]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T19:55:52.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:52 smithi097 ceph-mon[31898]: osdmap e89: 12 total, 12 up, 12 in 2024-03-11T19:55:52.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:52 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/369935034' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-11T19:55:52.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:52 smithi097 ceph-mon[31898]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:55:52.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:52 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/522254885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:52.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:52 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/923592692' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:52.420 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:52.421 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":89,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:55:50.540865+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:55:52.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.420+0000 7f4061e00640 1 -- 172.21.15.97:0/2626062455 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f40400a1a30 msgr2=0x7f40400a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:52.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.420+0000 7f4061e00640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f40400a1a30 0x7f40400a3ef0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f404c009830 tx=0x7f404c034c00 comp rx=0 tx=0).stop 2024-03-11T19:55:52.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.420+0000 7f4061e00640 1 -- 172.21.15.97:0/2626062455 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f405c1089b0 msgr2=0x7f405c19ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:52.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.420+0000 7f4061e00640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f405c1089b0 0x7f405c19ddb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f404800e9c0 tx=0x7f404800ee80 comp rx=0 tx=0).stop 2024-03-11T19:55:52.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.420+0000 7f4061e00640 1 -- 172.21.15.97:0/2626062455 shutdown_connections 2024-03-11T19:55:52.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.420+0000 7f4061e00640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f405c109370 0x7f405c1a4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.420+0000 7f4061e00640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f40400a1a30 0x7f40400a3ef0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.420+0000 7f4061e00640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f405c1089b0 0x7f405c19ddb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.420+0000 7f4061e00640 1 --2- 172.21.15.97:0/2626062455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f405c1077c0 0x7f405c19d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.420+0000 7f4061e00640 1 -- 172.21.15.97:0/2626062455 >> 172.21.15.97:0/2626062455 conn(0x7f405c076790 msgr2=0x7f405c10bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:52.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.420+0000 7f4061e00640 1 -- 172.21.15.97:0/2626062455 shutdown_connections 2024-03-11T19:55:52.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.420+0000 7f4061e00640 1 -- 172.21.15.97:0/2626062455 wait complete. 2024-03-11T19:55:52.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:55:52.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:52 smithi114 ceph-mon[36376]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T19:55:52.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:52 smithi114 ceph-mon[36376]: osdmap e89: 12 total, 12 up, 12 in 2024-03-11T19:55:52.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:52 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/369935034' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-11T19:55:52.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:52 smithi114 ceph-mon[36376]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:55:52.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:52 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/522254885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:52.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:52 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/923592692' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:52.537 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:52 smithi161 ceph-mon[36332]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T19:55:52.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:52 smithi161 ceph-mon[36332]: osdmap e89: 12 total, 12 up, 12 in 2024-03-11T19:55:52.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:52 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/369935034' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-11T19:55:52.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:52 smithi161 ceph-mon[36332]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:55:52.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:52 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/522254885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:52.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:52 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/923592692' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:52.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.652+0000 7f0afec00640 1 -- 172.21.15.97:0/2737025387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0af810d590 msgr2=0x7f0af810f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:52.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.652+0000 7f0afec00640 1 --2- 172.21.15.97:0/2737025387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0af810d590 0x7f0af810f980 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0af0009cd0 tx=0x7f0af002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.652+0000 7f0afec00640 1 -- 172.21.15.97:0/2737025387 shutdown_connections 2024-03-11T19:55:52.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.652+0000 7f0afec00640 1 --2- 172.21.15.97:0/2737025387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0af810d590 0x7f0af810f980 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.652+0000 7f0afec00640 1 --2- 172.21.15.97:0/2737025387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0af8073be0 0x7f0af8074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.652+0000 7f0afec00640 1 --2- 172.21.15.97:0/2737025387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0af80751c0 0x7f0af8073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.652+0000 7f0afec00640 1 -- 172.21.15.97:0/2737025387 >> 172.21.15.97:0/2737025387 conn(0x7f0af80fc510 msgr2=0x7f0af80fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:52.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.652+0000 7f0afec00640 1 -- 172.21.15.97:0/2737025387 shutdown_connections 2024-03-11T19:55:52.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.653+0000 7f0afec00640 1 -- 172.21.15.97:0/2737025387 wait complete. 2024-03-11T19:55:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.653+0000 7f0afec00640 1 Processor -- start 2024-03-11T19:55:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.653+0000 7f0afec00640 1 -- start start 2024-03-11T19:55:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0af8073be0 0x7f0af819d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0af80751c0 0x7f0af819dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afec00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0af810d590 0x7f0af81a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af81a5150 con 0x7f0af8073be0 2024-03-11T19:55:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af81a52c0 con 0x7f0af80751c0 2024-03-11T19:55:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afec00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af81a5480 con 0x7f0af810d590 2024-03-11T19:55:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afd800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0af8073be0 0x7f0af819d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afd800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0af8073be0 0x7f0af819d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40718/0 (socket says 172.21.15.97:40718) 2024-03-11T19:55:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afd800640 1 -- 172.21.15.97:0/666654849 learned_addr learned my addr 172.21.15.97:0/666654849 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afe200640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0af810d590 0x7f0af81a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afce00640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0af80751c0 0x7f0af819dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:52.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afd800640 1 -- 172.21.15.97:0/666654849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0af810d590 msgr2=0x7f0af81a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:52.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afd800640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0af810d590 0x7f0af81a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afd800640 1 -- 172.21.15.97:0/666654849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0af80751c0 msgr2=0x7f0af819dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:52.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afd800640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0af80751c0 0x7f0af819dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.654+0000 7f0afd800640 1 -- 172.21.15.97:0/666654849 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af0009900 con 0x7f0af8073be0 2024-03-11T19:55:52.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.655+0000 7f0afd800640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0af8073be0 0x7f0af819d660 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0aec00e9f0 tx=0x7f0aec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:52.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.655+0000 7f0af6000640 1 -- 172.21.15.97:0/666654849 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec009700 con 0x7f0af8073be0 2024-03-11T19:55:52.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.655+0000 7f0af6000640 1 -- 172.21.15.97:0/666654849 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0aec00cd80 con 0x7f0af8073be0 2024-03-11T19:55:52.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.655+0000 7f0afec00640 1 -- 172.21.15.97:0/666654849 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af81a5760 con 0x7f0af8073be0 2024-03-11T19:55:52.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.655+0000 7f0af6000640 1 -- 172.21.15.97:0/666654849 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aec018920 con 0x7f0af8073be0 2024-03-11T19:55:52.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.655+0000 7f0afec00640 1 -- 172.21.15.97:0/666654849 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af81a5cc0 con 0x7f0af8073be0 2024-03-11T19:55:52.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.656+0000 7f0afec00640 1 -- 172.21.15.97:0/666654849 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af8103650 con 0x7f0af8073be0 2024-03-11T19:55:52.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.658+0000 7f0af6000640 1 -- 172.21.15.97:0/666654849 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0aec018a80 con 0x7f0af8073be0 2024-03-11T19:55:52.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.660+0000 7f0af6000640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0ae00a1ab0 0x7f0ae00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:52.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.660+0000 7f0af6000640 1 -- 172.21.15.97:0/666654849 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0aec014070 con 0x7f0af8073be0 2024-03-11T19:55:52.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.660+0000 7f0afce00640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0ae00a1ab0 0x7f0ae00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:52.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.661+0000 7f0afce00640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0ae00a1ab0 0x7f0ae00a3f70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0af819eb70 tx=0x7f0ae4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:52.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.664+0000 7f0af6000640 1 -- 172.21.15.97:0/666654849 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0aec052d20 con 0x7f0af8073be0 2024-03-11T19:55:52.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.903+0000 7f0afec00640 1 -- 172.21.15.97:0/666654849 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0af8112fd0 con 0x7f0af8073be0 2024-03-11T19:55:52.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.905+0000 7f0af6000640 1 -- 172.21.15.97:0/666654849 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11289 (secure 0 0 0) 0x7f0af8112fd0 con 0x7f0af8073be0 2024-03-11T19:55:52.907 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:52.907 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24499,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6841/2531110572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2531110572},{"type":"v1","addr":"172.21.15.97:6841","nonce":2531110572}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-11T19:55:45.149335+0000","modified":"2024-03-11T19:55:50.541218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6839/1624532248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":1624532248},{"type":"v1","addr":"172.21.15.114:6839","nonce":1624532248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-11T19:55:52.908 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 17 2024-03-11T19:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.908+0000 7f0afec00640 1 -- 172.21.15.97:0/666654849 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0ae00a1ab0 msgr2=0x7f0ae00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.908+0000 7f0afec00640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0ae00a1ab0 0x7f0ae00a3f70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0af819eb70 tx=0x7f0ae4005f90 comp rx=0 tx=0).stop 2024-03-11T19:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.908+0000 7f0afec00640 1 -- 172.21.15.97:0/666654849 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0af8073be0 msgr2=0x7f0af819d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.908+0000 7f0afec00640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0af8073be0 0x7f0af819d660 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0aec00e9f0 tx=0x7f0aec00eeb0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.908+0000 7f0afec00640 1 -- 172.21.15.97:0/666654849 shutdown_connections 2024-03-11T19:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.908+0000 7f0afec00640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0af810d590 0x7f0af81a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.908+0000 7f0afec00640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0af80751c0 0x7f0af819dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.908+0000 7f0afec00640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0ae00a1ab0 0x7f0ae00a3f70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.908+0000 7f0afec00640 1 --2- 172.21.15.97:0/666654849 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0af8073be0 0x7f0af819d660 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.908+0000 7f0afec00640 1 -- 172.21.15.97:0/666654849 >> 172.21.15.97:0/666654849 conn(0x7f0af80fc510 msgr2=0x7f0af8100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.908+0000 7f0afec00640 1 -- 172.21.15.97:0/666654849 shutdown_connections 2024-03-11T19:55:52.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:52.908+0000 7f0afec00640 1 -- 172.21.15.97:0/666654849 wait complete. 2024-03-11T19:55:52.928 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:55:53.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.140+0000 7f4412e00640 1 -- 172.21.15.97:0/516496105 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f440c1033c0 msgr2=0x7f440c1037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.140+0000 7f4412e00640 1 --2- 172.21.15.97:0/516496105 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f440c1033c0 0x7f440c1037e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009900 tx=0x7f43fc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:53.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.141+0000 7f4412e00640 1 -- 172.21.15.97:0/516496105 shutdown_connections 2024-03-11T19:55:53.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.141+0000 7f4412e00640 1 --2- 172.21.15.97:0/516496105 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f440c104f70 0x7f440c10b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.141+0000 7f4412e00640 1 --2- 172.21.15.97:0/516496105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c1045b0 0x7f440c104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.141+0000 7f4412e00640 1 --2- 172.21.15.97:0/516496105 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f440c1033c0 0x7f440c1037e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.141+0000 7f4412e00640 1 -- 172.21.15.97:0/516496105 >> 172.21.15.97:0/516496105 conn(0x7f440c0fe970 msgr2=0x7f440c100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:53.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.141+0000 7f4412e00640 1 -- 172.21.15.97:0/516496105 shutdown_connections 2024-03-11T19:55:53.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.141+0000 7f4412e00640 1 -- 172.21.15.97:0/516496105 wait complete. 2024-03-11T19:55:53.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.142+0000 7f4412e00640 1 Processor -- start 2024-03-11T19:55:53.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.142+0000 7f4412e00640 1 -- start start 2024-03-11T19:55:53.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.142+0000 7f4412e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c1033c0 0x7f440c199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.142+0000 7f4412e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f440c1045b0 0x7f440c199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.142+0000 7f4412e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f440c104f70 0x7f440c1a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.142+0000 7f4412e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c1a0f20 con 0x7f440c1045b0 2024-03-11T19:55:53.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.142+0000 7f4412e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c1a1090 con 0x7f440c104f70 2024-03-11T19:55:53.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.142+0000 7f4412e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c1a1250 con 0x7f440c1033c0 2024-03-11T19:55:53.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.142+0000 7f4403400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f440c1045b0 0x7f440c199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.142+0000 7f4403400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f440c1045b0 0x7f440c199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40742/0 (socket says 172.21.15.97:40742) 2024-03-11T19:55:53.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.142+0000 7f4403400640 1 -- 172.21.15.97:0/590453830 learned_addr learned my addr 172.21.15.97:0/590453830 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:53.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.142+0000 7f440be00640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c1033c0 0x7f440c199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.142+0000 7f4410e00640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f440c104f70 0x7f440c1a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.143+0000 7f4403400640 1 -- 172.21.15.97:0/590453830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c1033c0 msgr2=0x7f440c199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.143+0000 7f4403400640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c1033c0 0x7f440c199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.143+0000 7f4403400640 1 -- 172.21.15.97:0/590453830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f440c104f70 msgr2=0x7f440c1a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.143+0000 7f4403400640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f440c104f70 0x7f440c1a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.143+0000 7f4403400640 1 -- 172.21.15.97:0/590453830 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc009560 con 0x7f440c1045b0 2024-03-11T19:55:53.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.143+0000 7f4403400640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f440c1045b0 0x7f440c199970 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f43f400e9f0 tx=0x7f43f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:53.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.143+0000 7f4409600640 1 -- 172.21.15.97:0/590453830 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f4009700 con 0x7f440c1045b0 2024-03-11T19:55:53.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.143+0000 7f4409600640 1 -- 172.21.15.97:0/590453830 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43f400cd80 con 0x7f440c1045b0 2024-03-11T19:55:53.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.143+0000 7f4412e00640 1 -- 172.21.15.97:0/590453830 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440c1a1530 con 0x7f440c1045b0 2024-03-11T19:55:53.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.143+0000 7f4409600640 1 -- 172.21.15.97:0/590453830 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f4018920 con 0x7f440c1045b0 2024-03-11T19:55:53.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.143+0000 7f4412e00640 1 -- 172.21.15.97:0/590453830 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440c1a1a40 con 0x7f440c1045b0 2024-03-11T19:55:53.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.145+0000 7f4412e00640 1 -- 172.21.15.97:0/590453830 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f440c10a6e0 con 0x7f440c1045b0 2024-03-11T19:55:53.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.147+0000 7f4409600640 1 -- 172.21.15.97:0/590453830 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f43f4018a80 con 0x7f440c1045b0 2024-03-11T19:55:53.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.148+0000 7f4409600640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f43ec0a1a30 0x7f43ec0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.148+0000 7f4409600640 1 -- 172.21.15.97:0/590453830 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f43f4014070 con 0x7f440c1045b0 2024-03-11T19:55:53.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.148+0000 7f440be00640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f43ec0a1a30 0x7f43ec0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.149+0000 7f440be00640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f43ec0a1a30 0x7f43ec0a3ef0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f43fc005ab0 tx=0x7f43fc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:53.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.152+0000 7f4409600640 1 -- 172.21.15.97:0/590453830 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f43f4052d20 con 0x7f440c1045b0 2024-03-11T19:55:53.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.348+0000 7f4412e00640 1 -- 172.21.15.97:0/590453830 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f440c02ce40 con 0x7f440c1045b0 2024-03-11T19:55:53.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.352+0000 7f4409600640 1 -- 172.21.15.97:0/590453830 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24152 (secure 0 0 0) 0x7f43f4096a30 con 0x7f440c1045b0 2024-03-11T19:55:53.354 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:55:53.354 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":89,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:55:50.540865+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:55:53.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.354+0000 7f4412e00640 1 -- 172.21.15.97:0/590453830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f43ec0a1a30 msgr2=0x7f43ec0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.354+0000 7f4412e00640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f43ec0a1a30 0x7f43ec0a3ef0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f43fc005ab0 tx=0x7f43fc005fd0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.355+0000 7f4412e00640 1 -- 172.21.15.97:0/590453830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f440c1045b0 msgr2=0x7f440c199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.355+0000 7f4412e00640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f440c1045b0 0x7f440c199970 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f43f400e9f0 tx=0x7f43f400eeb0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.355+0000 7f4412e00640 1 -- 172.21.15.97:0/590453830 shutdown_connections 2024-03-11T19:55:53.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.355+0000 7f4412e00640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f440c104f70 0x7f440c1a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.355+0000 7f4412e00640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f43ec0a1a30 0x7f43ec0a3ef0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.355+0000 7f4412e00640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f440c1045b0 0x7f440c199970 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.355+0000 7f4412e00640 1 --2- 172.21.15.97:0/590453830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f440c1033c0 0x7f440c199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.355+0000 7f4412e00640 1 -- 172.21.15.97:0/590453830 >> 172.21.15.97:0/590453830 conn(0x7f440c0fe970 msgr2=0x7f440c1078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:53.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.355+0000 7f4412e00640 1 -- 172.21.15.97:0/590453830 shutdown_connections 2024-03-11T19:55:53.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.355+0000 7f4412e00640 1 -- 172.21.15.97:0/590453830 wait complete. 2024-03-11T19:55:53.372 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:53 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2626062455' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:53.373 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:53 smithi097 ceph-mon[31898]: pgmap v323: 129 pgs: 4 creating+peering, 108 active+clean, 17 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:55:53.373 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:53 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/666654849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:53.377 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-11T19:55:45.149335+0000', 'modified': '2024-03-11T19:55:50.541218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24599}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24599': {'gid': 24599, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6839/1624532248', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6838', 'nonce': 1624532248}, {'type': 'v1', 'addr': '172.21.15.114:6839', 'nonce': 1624532248}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-03-11T19:55:53.377 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-11T19:55:53.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-11T19:55:53.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:53 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2626062455' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:53.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:53 smithi114 ceph-mon[36376]: pgmap v323: 129 pgs: 4 creating+peering, 108 active+clean, 17 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:55:53.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:53 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/666654849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:53.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:53 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2626062455' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:53.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:53 smithi161 ceph-mon[36332]: pgmap v323: 129 pgs: 4 creating+peering, 108 active+clean, 17 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:55:53.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:53 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/666654849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:55:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.580+0000 7f8e67e00640 1 -- 172.21.15.97:0/1817911695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e601033c0 msgr2=0x7f8e601037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.580+0000 7f8e67e00640 1 --2- 172.21.15.97:0/1817911695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e601033c0 0x7f8e601037e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8e54009900 tx=0x7f8e5402bc30 comp rx=0 tx=0).stop 2024-03-11T19:55:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.580+0000 7f8e67e00640 1 -- 172.21.15.97:0/1817911695 shutdown_connections 2024-03-11T19:55:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.580+0000 7f8e67e00640 1 --2- 172.21.15.97:0/1817911695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e60104f70 0x7f8e6010b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.580+0000 7f8e67e00640 1 --2- 172.21.15.97:0/1817911695 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e601045b0 0x7f8e60104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.580+0000 7f8e67e00640 1 --2- 172.21.15.97:0/1817911695 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e601033c0 0x7f8e601037e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.580+0000 7f8e67e00640 1 -- 172.21.15.97:0/1817911695 >> 172.21.15.97:0/1817911695 conn(0x7f8e600fe930 msgr2=0x7f8e60100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:53.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.580+0000 7f8e67e00640 1 -- 172.21.15.97:0/1817911695 shutdown_connections 2024-03-11T19:55:53.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.580+0000 7f8e67e00640 1 -- 172.21.15.97:0/1817911695 wait complete. 2024-03-11T19:55:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.581+0000 7f8e67e00640 1 Processor -- start 2024-03-11T19:55:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.581+0000 7f8e67e00640 1 -- start start 2024-03-11T19:55:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e67e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e601033c0 0x7f8e601994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e67e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e601045b0 0x7f8e601999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e67e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e60104f70 0x7f8e601a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e67e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e601a0fa0 con 0x7f8e601045b0 2024-03-11T19:55:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e67e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e601a1110 con 0x7f8e60104f70 2024-03-11T19:55:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e67e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e601a12d0 con 0x7f8e601033c0 2024-03-11T19:55:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e64a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e601045b0 0x7f8e601999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e65400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e601033c0 0x7f8e601994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e64a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e601045b0 0x7f8e601999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40762/0 (socket says 172.21.15.97:40762) 2024-03-11T19:55:53.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e64a00640 1 -- 172.21.15.97:0/1050037998 learned_addr learned my addr 172.21.15.97:0/1050037998 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e65e00640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e60104f70 0x7f8e601a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e65400640 1 -- 172.21.15.97:0/1050037998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e60104f70 msgr2=0x7f8e601a0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e65400640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e60104f70 0x7f8e601a0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e65400640 1 -- 172.21.15.97:0/1050037998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e601045b0 msgr2=0x7f8e601999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e65400640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e601045b0 0x7f8e601999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e65400640 1 -- 172.21.15.97:0/1050037998 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e58009840 con 0x7f8e601033c0 2024-03-11T19:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.582+0000 7f8e64a00640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e601045b0 0x7f8e601999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.583+0000 7f8e65400640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e601033c0 0x7f8e601994b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8e54036040 tx=0x7f8e54005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.583+0000 7f8e5e000640 1 -- 172.21.15.97:0/1050037998 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e540043d0 con 0x7f8e601033c0 2024-03-11T19:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.583+0000 7f8e5e000640 1 -- 172.21.15.97:0/1050037998 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e540052d0 con 0x7f8e601033c0 2024-03-11T19:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.583+0000 7f8e67e00640 1 -- 172.21.15.97:0/1050037998 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e54009560 con 0x7f8e601033c0 2024-03-11T19:55:53.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.583+0000 7f8e5e000640 1 -- 172.21.15.97:0/1050037998 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e54003c60 con 0x7f8e601033c0 2024-03-11T19:55:53.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.583+0000 7f8e67e00640 1 -- 172.21.15.97:0/1050037998 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e601a18a0 con 0x7f8e601033c0 2024-03-11T19:55:53.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.586+0000 7f8e6ab77740 1 Processor -- start 2024-03-11T19:55:53.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.586+0000 7f8e5e000640 1 -- 172.21.15.97:0/1050037998 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8e54005440 con 0x7f8e601033c0 2024-03-11T19:55:53.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.586+0000 7f8e6ab77740 1 -- start start 2024-03-11T19:55:53.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.586+0000 7f8e6ab77740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77580 0x55aa1db779a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.586+0000 7f8e6ab77740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77f70 0x55aa1db7ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e6ab77740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db7d0e0 0x55aa1db7f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e6ab77740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa1dae6b30 con 0x55aa1db77580 2024-03-11T19:55:53.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e6ab77740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa1dae4060 con 0x55aa1db77f70 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e6ab77740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa1dadf840 con 0x55aa1db7d0e0 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e65e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77580 0x55aa1db779a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e65e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77580 0x55aa1db779a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40774/0 (socket says 172.21.15.97:40774) 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e65e00640 1 -- 172.21.15.97:0/3875851859 learned_addr learned my addr 172.21.15.97:0/3875851859 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e65400640 1 --2- 172.21.15.97:0/3875851859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77f70 0x55aa1db7ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e64a00640 1 --2- 172.21.15.97:0/3875851859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db7d0e0 0x55aa1db7f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e65e00640 1 -- 172.21.15.97:0/3875851859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db7d0e0 msgr2=0x55aa1db7f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e65e00640 1 --2- 172.21.15.97:0/3875851859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db7d0e0 0x55aa1db7f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e65e00640 1 -- 172.21.15.97:0/3875851859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77f70 msgr2=0x55aa1db7ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e65e00640 1 --2- 172.21.15.97:0/3875851859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77f70 0x55aa1db7ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e65e00640 1 -- 172.21.15.97:0/3875851859 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa1daa74c0 con 0x55aa1db77580 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e64a00640 1 --2- 172.21.15.97:0/3875851859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db7d0e0 0x55aa1db7f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.587+0000 7f8e65e00640 1 --2- 172.21.15.97:0/3875851859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77580 0x55aa1db779a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8e580115b0 tx=0x7f8e58011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbd45f10982e692 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:53.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e52600640 1 -- 172.21.15.97:0/3875851859 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58004d20 con 0x55aa1db77580 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e52600640 1 -- 172.21.15.97:0/3875851859 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e58004e80 con 0x55aa1db77580 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e52600640 1 -- 172.21.15.97:0/3875851859 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5800f8c0 con 0x55aa1db77580 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e5e000640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8e480a1a30 0x7f8e480a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e5e000640 1 -- 172.21.15.97:0/1050037998 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8e540ef6a0 con 0x7f8e601033c0 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e65400640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8e480a1a30 0x7f8e480a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e6ab77740 1 -- 172.21.15.97:0/3875851859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77580 msgr2=0x55aa1db779a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/3875851859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77580 0x55aa1db779a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8e580115b0 tx=0x7f8e58011a70 comp rx=0 tx=0).stop 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e6ab77740 1 -- 172.21.15.97:0/3875851859 shutdown_connections 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/3875851859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db7d0e0 0x55aa1db7f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/3875851859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77f70 0x55aa1db7ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/3875851859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77580 0x55aa1db779a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e6ab77740 1 -- 172.21.15.97:0/3875851859 >> 172.21.15.97:0/3875851859 conn(0x55aa1db72af0 msgr2=0x55aa1db72ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.588+0000 7f8e6ab77740 1 -- 172.21.15.97:0/3875851859 shutdown_connections 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.589+0000 7f8e6ab77740 1 -- 172.21.15.97:0/3875851859 wait complete. 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.589+0000 7f8e65400640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8e480a1a30 0x7f8e480a3ef0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8e540ed2d0 tx=0x7f8e540b8040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:53.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.589+0000 7f8e6ab77740 1 -- start start 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.589+0000 7f8e6ab77740 1 Processor -- start 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.589+0000 7f8e6ab77740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77580 0x55aa1db813b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.589+0000 7f8e6ab77740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 0x55aa1db818f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e6ab77740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db7d0e0 0x55aa1db88930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e6ab77740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa1da84ea0 con 0x55aa1db77f70 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e6ab77740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa1da84b40 con 0x55aa1db77580 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e6ab77740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa1d827f20 con 0x55aa1db7d0e0 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e6ab77740 10 client.0 _ll_register_callbacks cb 0x55aa1da98950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e64a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 0x55aa1db818f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e64a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 0x55aa1db818f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40782/0 (socket says 172.21.15.97:40782) 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e64a00640 1 -- 172.21.15.97:0/3457939029 learned_addr learned my addr 172.21.15.97:0/3457939029 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e65e00640 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77580 0x55aa1db813b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e65e00640 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db7d0e0 0x55aa1db88930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e64a00640 1 -- 172.21.15.97:0/3457939029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db7d0e0 msgr2=0x55aa1db88930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e64a00640 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db7d0e0 0x55aa1db88930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e64a00640 1 -- 172.21.15.97:0/3457939029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77580 msgr2=0x55aa1db813b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e64a00640 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77580 0x55aa1db813b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e64a00640 1 -- 172.21.15.97:0/3457939029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e58009e20 con 0x55aa1db77f70 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e65e00640 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77580 0x55aa1db813b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e64a00640 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 0x55aa1db818f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c012d20 tx=0x7f8e4c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112651ea90ff0087 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.590+0000 7f8e46a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.591+0000 7f8e46a00640 1 -- 172.21.15.97:0/3457939029 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c017070 con 0x55aa1db77f70 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.591+0000 7f8e46a00640 1 -- 172.21.15.97:0/3457939029 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8e4c0120d0 con 0x55aa1db77f70 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.591+0000 7f8e6ab77740 1 -- 172.21.15.97:0/3457939029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aa1d7df6c0 con 0x55aa1db77f70 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.591+0000 7f8e46a00640 1 -- 172.21.15.97:0/3457939029 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e4c012290 con 0x55aa1db77f70 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.591+0000 7f8e6ab77740 10 client.14586 fetch_fsmap learned FSMap version 17 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.591+0000 7f8e6ab77740 1 -- 172.21.15.97:0/3457939029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55aa1daddec0 con 0x55aa1db77f70 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.591+0000 7f8e46a00640 1 -- 172.21.15.97:0/3457939029 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55aa1d7df6c0 con 0x55aa1db77f70 2024-03-11T19:55:53.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.591+0000 7f8e46a00640 1 -- 172.21.15.97:0/3457939029 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 17) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8e4c0106d0 con 0x55aa1db77f70 2024-03-11T19:55:53.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.592+0000 7f8e6ab77740 10 client.14586 fetch_fsmap finished waiting for FSMap version 17 2024-03-11T19:55:53.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.592+0000 7f8e6ab77740 10 client.14586 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T19:55:53.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.592+0000 7f8e6ab77740 20 client.14586 populate_metadata read hostname 'smithi097' 2024-03-11T19:55:53.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.592+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55aa1db8cd20 0x55aa1db91d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.592+0000 7f8e6ab77740 4 client.14586 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:55:53.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.592+0000 7f8e6ab77740 1 -- 172.21.15.97:0/3457939029 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aa1d830270 con 0x55aa1db8cd20 2024-03-11T19:55:53.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.592+0000 7f8e65e00640 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55aa1db8cd20 0x55aa1db91d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.593+0000 7f8e65e00640 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55aa1db8cd20 0x55aa1db91d40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a18281aef8187acc server_cookie=1c73991760f6fe in_seq=0 out_seq=0 2024-03-11T19:55:53.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.593+0000 7f8e46a00640 10 client.14586 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.593+0000 7f8e46a00640 1 -- 172.21.15.97:0/3457939029 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55aa1d830270 con 0x55aa1db8cd20 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.593+0000 7f8e46a00640 10 client.14586 handle_command_reply: tid=0 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.593+0000 7f8e6ab77740 1 client.14586 shutdown 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.594+0000 7f8e6ab77740 1 -- 172.21.15.97:0/3457939029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 msgr2=0x55aa1db818f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.594+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 0x55aa1db818f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c012d20 tx=0x7f8e4c00f790 comp rx=0 tx=0).stop 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.594+0000 7f8e6ab77740 1 -- 172.21.15.97:0/3457939029 shutdown_connections 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.594+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db7d0e0 0x55aa1db88930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.594+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55aa1db8cd20 0x55aa1db91d40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.594+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 0x55aa1db818f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.594+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/3457939029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77580 0x55aa1db813b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.594+0000 7f8e46a00640 0 client.14586 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.594+0000 7f8e6ab77740 1 -- 172.21.15.97:0/3457939029 >> 172.21.15.97:0/3457939029 conn(0x55aa1db72af0 msgr2=0x55aa1db71e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.594+0000 7f8e6ab77740 1 -- 172.21.15.97:0/3457939029 shutdown_connections 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.594+0000 7f8e6ab77740 1 -- 172.21.15.97:0/3457939029 wait complete. 2024-03-11T19:55:53.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.594+0000 7f8e6ab77740 20 client.14586 trim_cache size 0 max 16384 2024-03-11T19:55:53.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.610+0000 7f8e6ab77740 1 Processor -- start 2024-03-11T19:55:53.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.610+0000 7f8e6ab77740 1 -- start start 2024-03-11T19:55:53.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.611+0000 7f8e6ab77740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db77580 0x55aa1db816d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.611+0000 7f8e6ab77740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 0x55aa1db888f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.611+0000 7f8e6ab77740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db7d0e0 0x55aa1db95d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.611+0000 7f8e6ab77740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa1db8ee10 con 0x55aa1db77f70 2024-03-11T19:55:53.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.611+0000 7f8e6ab77740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa1db80c00 con 0x55aa1db7d0e0 2024-03-11T19:55:53.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.611+0000 7f8e6ab77740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa1db7fc30 con 0x55aa1db77580 2024-03-11T19:55:53.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.611+0000 7f8e64a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 0x55aa1db888f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.611+0000 7f8e65e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db77580 0x55aa1db816d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.611+0000 7f8e65e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db77580 0x55aa1db816d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:41742/0 (socket says 172.21.15.97:41742) 2024-03-11T19:55:53.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.611+0000 7f8e64a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 0x55aa1db888f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:40792/0 (socket says 172.21.15.97:40792) 2024-03-11T19:55:53.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.611+0000 7f8e65e00640 1 -- 172.21.15.97:0/2944766939 learned_addr learned my addr 172.21.15.97:0/2944766939 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:53.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.611+0000 7f8e65e00640 1 --2- 172.21.15.97:0/2944766939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db7d0e0 0x55aa1db95d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.612+0000 7f8e64a00640 1 -- 172.21.15.97:0/2944766939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db77580 msgr2=0x55aa1db816d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:55:53.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.612+0000 7f8e64a00640 1 --2- 172.21.15.97:0/2944766939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db77580 0x55aa1db816d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.612+0000 7f8e64a00640 1 -- 172.21.15.97:0/2944766939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db7d0e0 msgr2=0x55aa1db95d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:55:53.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.612+0000 7f8e64a00640 1 --2- 172.21.15.97:0/2944766939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db7d0e0 0x55aa1db95d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.612+0000 7f8e64a00640 1 -- 172.21.15.97:0/2944766939 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa1da7a8b0 con 0x55aa1db77f70 2024-03-11T19:55:53.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.612+0000 7f8e64a00640 1 --2- 172.21.15.97:0/2944766939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 0x55aa1db888f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00a010 tx=0x7f8e4c012910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c1f9160489eee1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.612+0000 7f8e51c00640 1 -- 172.21.15.97:0/2944766939 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c017040 con 0x55aa1db77f70 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.612+0000 7f8e51c00640 1 -- 172.21.15.97:0/2944766939 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e4c012070 con 0x55aa1db77f70 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.612+0000 7f8e51c00640 1 -- 172.21.15.97:0/2944766939 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0048e0 con 0x55aa1db77f70 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.613+0000 7f8e6ab77740 1 -- 172.21.15.97:0/2944766939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 msgr2=0x55aa1db888f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.613+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/2944766939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 0x55aa1db888f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00a010 tx=0x7f8e4c012910 comp rx=0 tx=0).stop 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.613+0000 7f8e6ab77740 1 -- 172.21.15.97:0/2944766939 shutdown_connections 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.613+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/2944766939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db7d0e0 0x55aa1db95d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.613+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/2944766939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db77f70 0x55aa1db888f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.613+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/2944766939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db77580 0x55aa1db816d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.613+0000 7f8e6ab77740 1 -- 172.21.15.97:0/2944766939 >> 172.21.15.97:0/2944766939 conn(0x55aa1db72af0 msgr2=0x55aa1db80fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.613+0000 7f8e6ab77740 1 -- 172.21.15.97:0/2944766939 shutdown_connections 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.613+0000 7f8e6ab77740 1 -- 172.21.15.97:0/2944766939 wait complete. 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.613+0000 7f8e6ab77740 1 -- start start 2024-03-11T19:55:53.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.613+0000 7f8e6ab77740 1 Processor -- start 2024-03-11T19:55:53.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e6ab77740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db77580 0x55aa1db853f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e6ab77740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77f70 0x55aa1db85930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e6ab77740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db7d0e0 0x55aa1dbe3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e6ab77740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa1db88900 con 0x55aa1db7d0e0 2024-03-11T19:55:53.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e6ab77740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa1db88a70 con 0x55aa1db77f70 2024-03-11T19:55:53.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e6ab77740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa1db88c30 con 0x55aa1db77580 2024-03-11T19:55:53.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e6ab77740 10 client.0 _ll_register_callbacks cb 0x7f8e4c009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:55:53.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e64a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77f70 0x55aa1db85930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e64a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77f70 0x55aa1db85930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:43388/0 (socket says 172.21.15.97:43388) 2024-03-11T19:55:53.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e64a00640 1 -- 172.21.15.97:0/2574284922 learned_addr learned my addr 172.21.15.97:0/2574284922 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:53.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e65e00640 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db77580 0x55aa1db853f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e65e00640 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db7d0e0 0x55aa1dbe3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e64a00640 1 -- 172.21.15.97:0/2574284922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db77580 msgr2=0x55aa1db853f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e64a00640 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db77580 0x55aa1db853f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e64a00640 1 -- 172.21.15.97:0/2574284922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db7d0e0 msgr2=0x55aa1dbe3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e64a00640 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db7d0e0 0x55aa1dbe3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e64a00640 1 -- 172.21.15.97:0/2574284922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa1daddec0 con 0x55aa1db77f70 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.614+0000 7f8e65e00640 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db7d0e0 0x55aa1dbe3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.615+0000 7f8e64a00640 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77f70 0x55aa1db85930 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c010810 tx=0x7f8e4c010840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7a422c8f342a8d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.615+0000 7f8e46a00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.615+0000 7f8e46a00640 1 -- 172.21.15.97:0/2574284922 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c017040 con 0x55aa1db77f70 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.615+0000 7f8e6ab77740 1 -- 172.21.15.97:0/2574284922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aa1db8ef50 con 0x55aa1db77f70 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.615+0000 7f8e46a00640 1 -- 172.21.15.97:0/2574284922 <== mon.2 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8e4c000e40 con 0x55aa1db77f70 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.615+0000 7f8e46a00640 1 -- 172.21.15.97:0/2574284922 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e4c01c670 con 0x55aa1db77f70 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.615+0000 7f8e6ab77740 10 client.24713 fetch_fsmap learned FSMap version 17 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.615+0000 7f8e6ab77740 1 -- 172.21.15.97:0/2574284922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55aa1daa0510 con 0x55aa1db77f70 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.615+0000 7f8e46a00640 1 -- 172.21.15.97:0/2574284922 <== mon.2 v2:172.21.15.114:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55aa1db8ef50 con 0x55aa1db77f70 2024-03-11T19:55:53.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.616+0000 7f8e46a00640 1 -- 172.21.15.97:0/2574284922 <== mon.2 v2:172.21.15.114:3300/0 5 ==== fsmap(e 17) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f8e4c0229d0 con 0x55aa1db77f70 2024-03-11T19:55:53.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.616+0000 7f8e6ab77740 10 client.24713 fetch_fsmap finished waiting for FSMap version 17 2024-03-11T19:55:53.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.616+0000 7f8e6ab77740 10 client.24713 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T19:55:53.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.616+0000 7f8e6ab77740 20 client.24713 populate_metadata read hostname 'smithi097' 2024-03-11T19:55:53.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.616+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55aa1dbe8c10 0x55aa1dbe50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:53.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.616+0000 7f8e6ab77740 4 client.24713 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:55:53.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.616+0000 7f8e6ab77740 1 -- 172.21.15.97:0/2574284922 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55aa1db03c40 con 0x55aa1dbe8c10 2024-03-11T19:55:53.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.616+0000 7f8e65e00640 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55aa1dbe8c10 0x55aa1dbe50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:53.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.616+0000 7f8e65e00640 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55aa1dbe8c10 0x55aa1dbe50d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3046769a83100cfa server_cookie=28991ea316f0f10f in_seq=0 out_seq=0 2024-03-11T19:55:53.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.616+0000 7f8e46a00640 10 client.24713 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T19:55:53.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e46a00640 1 -- 172.21.15.97:0/2574284922 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55aa1db03c40 con 0x55aa1dbe8c10 2024-03-11T19:55:53.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e46a00640 10 client.24713 handle_command_reply: tid=0 2024-03-11T19:55:53.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e6ab77740 1 client.24713 shutdown 2024-03-11T19:55:53.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e6ab77740 1 -- 172.21.15.97:0/2574284922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77f70 msgr2=0x55aa1db85930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77f70 0x55aa1db85930 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c010810 tx=0x7f8e4c010840 comp rx=0 tx=0).stop 2024-03-11T19:55:53.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e6ab77740 1 -- 172.21.15.97:0/2574284922 shutdown_connections 2024-03-11T19:55:53.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55aa1dbe8c10 0x55aa1dbe50d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa1db7d0e0 0x55aa1dbe3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e46a00640 0 client.24713 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T19:55:53.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa1db77f70 0x55aa1db85930 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e6ab77740 1 --2- 172.21.15.97:0/2574284922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa1db77580 0x55aa1db853f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e6ab77740 1 -- 172.21.15.97:0/2574284922 >> 172.21.15.97:0/2574284922 conn(0x55aa1db72af0 msgr2=0x55aa1db95d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:53.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e6ab77740 1 -- 172.21.15.97:0/2574284922 shutdown_connections 2024-03-11T19:55:53.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e6ab77740 1 -- 172.21.15.97:0/2574284922 wait complete. 2024-03-11T19:55:53.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.617+0000 7f8e6ab77740 20 client.24713 trim_cache size 0 max 16384 2024-03-11T19:55:53.626 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T19:55:53.626 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:55:53.626 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-11T19:55:53.626 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24599, 2024-03-11T19:55:53.626 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-11T19:55:53.626 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-11T19:55:53.627 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-11T19:55:53.627 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 7.5774595839999996, 2024-03-11T19:55:53.627 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 17, 2024-03-11T19:55:53.627 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 89, 2024-03-11T19:55:53.627 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 89, 2024-03-11T19:55:53.627 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 14.590712691 2024-03-11T19:55:53.627 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T19:55:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.618+0000 7f8e67e00640 1 -- 172.21.15.97:0/1050037998 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8e480a1a30 msgr2=0x7f8e480a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.618+0000 7f8e67e00640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8e480a1a30 0x7f8e480a3ef0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8e540ed2d0 tx=0x7f8e540b8040 comp rx=0 tx=0).stop 2024-03-11T19:55:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.618+0000 7f8e67e00640 1 -- 172.21.15.97:0/1050037998 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e601033c0 msgr2=0x7f8e601994b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.618+0000 7f8e67e00640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e601033c0 0x7f8e601994b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8e54036040 tx=0x7f8e54005e50 comp rx=0 tx=0).stop 2024-03-11T19:55:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.618+0000 7f8e67e00640 1 -- 172.21.15.97:0/1050037998 shutdown_connections 2024-03-11T19:55:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.618+0000 7f8e67e00640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e60104f70 0x7f8e601a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.618+0000 7f8e67e00640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8e480a1a30 0x7f8e480a3ef0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.618+0000 7f8e67e00640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8e601045b0 0x7f8e601999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.618+0000 7f8e67e00640 1 --2- 172.21.15.97:0/1050037998 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8e601033c0 0x7f8e601994b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.618+0000 7f8e67e00640 1 -- 172.21.15.97:0/1050037998 >> 172.21.15.97:0/1050037998 conn(0x7f8e600fe930 msgr2=0x7f8e60107920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.618+0000 7f8e67e00640 1 -- 172.21.15.97:0/1050037998 shutdown_connections 2024-03-11T19:55:53.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:53.618+0000 7f8e67e00640 1 -- 172.21.15.97:0/1050037998 wait complete. 2024-03-11T19:55:53.637 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-11T19:55:53.637 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-11T19:55:53.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph fs ls 2024-03-11T19:55:54.069 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:55:54.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:54 smithi097 ceph-mon[31898]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T19:55:54.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:54 smithi097 ceph-mon[31898]: Cluster is now healthy 2024-03-11T19:55:54.420 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:54 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/590453830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:54.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:54 smithi114 ceph-mon[36376]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T19:55:54.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:54 smithi114 ceph-mon[36376]: Cluster is now healthy 2024-03-11T19:55:54.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:54 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/590453830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:54.537 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:54 smithi161 ceph-mon[36332]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T19:55:54.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:54 smithi161 ceph-mon[36332]: Cluster is now healthy 2024-03-11T19:55:54.538 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:54 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/590453830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:55:55.419 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:55 smithi097 ceph-mon[31898]: pgmap v324: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:55:55.491 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:55 smithi114 ceph-mon[36376]: pgmap v324: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:55:55.537 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:55 smithi161 ceph-mon[36332]: pgmap v324: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:55:56.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.624+0000 7fb908600700 1 -- 172.21.15.97:0/1954095143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb90010e670 msgr2=0x7fb900110a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:56.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.624+0000 7fb908600700 1 --2- 172.21.15.97:0/1954095143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb90010e670 0x7fb900110a60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb8f800b0a0 tx=0x7fb8f800b3b0 comp rx=0 tx=0).stop 2024-03-11T19:55:56.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.624+0000 7fb908600700 1 -- 172.21.15.97:0/1954095143 shutdown_connections 2024-03-11T19:55:56.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.624+0000 7fb908600700 1 --2- 172.21.15.97:0/1954095143 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb90010e670 0x7fb900110a60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:56.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.624+0000 7fb908600700 1 --2- 172.21.15.97:0/1954095143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb900073ea0 0x7fb900074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:56.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.624+0000 7fb908600700 1 --2- 172.21.15.97:0/1954095143 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9000747c0 0x7fb90010e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:56.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.624+0000 7fb908600700 1 -- 172.21.15.97:0/1954095143 >> 172.21.15.97:0/1954095143 conn(0x7fb900078dd0 msgr2=0x7fb9000791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:56.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.624+0000 7fb908600700 1 -- 172.21.15.97:0/1954095143 shutdown_connections 2024-03-11T19:55:56.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.624+0000 7fb908600700 1 -- 172.21.15.97:0/1954095143 wait complete. 2024-03-11T19:55:56.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.625+0000 7fb908600700 1 Processor -- start 2024-03-11T19:55:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.625+0000 7fb908600700 1 -- start start 2024-03-11T19:55:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.625+0000 7fb908600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb900073ea0 0x7fb90019a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.625+0000 7fb908600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9000747c0 0x7fb90019aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.625+0000 7fb908600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb90010e670 0x7fb90019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.625+0000 7fb908600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90019f490 con 0x7fb9000747c0 2024-03-11T19:55:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.626+0000 7fb908600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90019f600 con 0x7fb900073ea0 2024-03-11T19:55:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.626+0000 7fb908600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90019f7c0 con 0x7fb90010e670 2024-03-11T19:55:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.626+0000 7fb905c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb900073ea0 0x7fb90019a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.626+0000 7fb905c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb900073ea0 0x7fb90019a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:53156/0 (socket says 172.21.15.97:53156) 2024-03-11T19:55:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.626+0000 7fb905c00700 1 -- 172.21.15.97:0/1320818170 learned_addr learned my addr 172.21.15.97:0/1320818170 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:55:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.626+0000 7fb905200700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9000747c0 0x7fb90019aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.626+0000 7fb906600700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb90010e670 0x7fb90019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.626+0000 7fb905c00700 1 -- 172.21.15.97:0/1320818170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb90010e670 msgr2=0x7fb90019ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.626+0000 7fb905c00700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb90010e670 0x7fb90019ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.626+0000 7fb905c00700 1 -- 172.21.15.97:0/1320818170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9000747c0 msgr2=0x7fb90019aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.626+0000 7fb905c00700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9000747c0 0x7fb90019aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.626+0000 7fb905c00700 1 -- 172.21.15.97:0/1320818170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec009520 con 0x7fb900073ea0 2024-03-11T19:55:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.626+0000 7fb906600700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb90010e670 0x7fb90019ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:55:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.627+0000 7fb905c00700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb900073ea0 0x7fb90019a5b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb8f400e9a0 tx=0x7fb8f400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.627+0000 7fb8fe000700 1 -- 172.21.15.97:0/1320818170 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4009790 con 0x7fb900073ea0 2024-03-11T19:55:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.627+0000 7fb8fe000700 1 -- 172.21.15.97:0/1320818170 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8f400cbd0 con 0x7fb900073ea0 2024-03-11T19:55:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.627+0000 7fb908600700 1 -- 172.21.15.97:0/1320818170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f8009d00 con 0x7fb900073ea0 2024-03-11T19:55:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.627+0000 7fb8fe000700 1 -- 172.21.15.97:0/1320818170 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4018680 con 0x7fb900073ea0 2024-03-11T19:55:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.627+0000 7fb908600700 1 -- 172.21.15.97:0/1320818170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9001a7130 con 0x7fb900073ea0 2024-03-11T19:55:56.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.628+0000 7fb908600700 1 -- 172.21.15.97:0/1320818170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb900068930 con 0x7fb900073ea0 2024-03-11T19:55:56.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.630+0000 7fb8fe000700 1 -- 172.21.15.97:0/1320818170 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb8f400cd40 con 0x7fb900073ea0 2024-03-11T19:55:56.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.631+0000 7fb8fe000700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb8e80a1b40 0x7fb8e80a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:55:56.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.632+0000 7fb8fe000700 1 -- 172.21.15.97:0/1320818170 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb8f4014070 con 0x7fb900073ea0 2024-03-11T19:55:56.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.632+0000 7fb905200700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb8e80a1b40 0x7fb8e80a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:55:56.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.632+0000 7fb905200700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb8e80a1b40 0x7fb8e80a4000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb900110370 tx=0x7fb8ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:55:56.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.636+0000 7fb8fe000700 1 -- 172.21.15.97:0/1320818170 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb8f4052d20 con 0x7fb900073ea0 2024-03-11T19:55:56.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.902+0000 7fb908600700 1 -- 172.21.15.97:0/1320818170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb90019b910 con 0x7fb900073ea0 2024-03-11T19:55:56.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.903+0000 7fb8fe000700 1 -- 172.21.15.97:0/1320818170 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7fb8f4096840 con 0x7fb900073ea0 2024-03-11T19:55:56.905 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-11T19:55:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.906+0000 7fb908600700 1 -- 172.21.15.97:0/1320818170 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb8e80a1b40 msgr2=0x7fb8e80a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.906+0000 7fb908600700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb8e80a1b40 0x7fb8e80a4000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb900110370 tx=0x7fb8ec009260 comp rx=0 tx=0).stop 2024-03-11T19:55:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.906+0000 7fb908600700 1 -- 172.21.15.97:0/1320818170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb900073ea0 msgr2=0x7fb90019a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:55:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.906+0000 7fb908600700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb900073ea0 0x7fb90019a5b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb8f400e9a0 tx=0x7fb8f400ecb0 comp rx=0 tx=0).stop 2024-03-11T19:55:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.906+0000 7fb908600700 1 -- 172.21.15.97:0/1320818170 shutdown_connections 2024-03-11T19:55:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.906+0000 7fb908600700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb8e80a1b40 0x7fb8e80a4000 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.906+0000 7fb908600700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb90010e670 0x7fb90019ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.906+0000 7fb908600700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb900073ea0 0x7fb90019a5b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.906+0000 7fb908600700 1 --2- 172.21.15.97:0/1320818170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb9000747c0 0x7fb90019aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:55:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.906+0000 7fb908600700 1 -- 172.21.15.97:0/1320818170 >> 172.21.15.97:0/1320818170 conn(0x7fb900078dd0 msgr2=0x7fb900100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:55:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.906+0000 7fb908600700 1 -- 172.21.15.97:0/1320818170 shutdown_connections 2024-03-11T19:55:56.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:55:56.906+0000 7fb908600700 1 -- 172.21.15.97:0/1320818170 wait complete. 2024-03-11T19:55:57.921 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:57 smithi097 ceph-mon[31898]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:55:57.922 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:57 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1320818170' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T19:55:57.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:57 smithi114 ceph-mon[36376]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:55:57.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:57 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1320818170' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T19:55:57.999 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-11T19:55:57.999 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-11T19:55:57.999 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-11T19:55:58.000 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-11T19:55:58.000 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-11T19:55:58.000 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-11T19:55:58.000 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-11T19:55:58.000 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-11T19:55:58.000 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-11T19:55:58.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:55:58.000 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-11T19:55:58.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:57 smithi161 ceph-mon[36332]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:55:58.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:57 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1320818170' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T19:55:58.056 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-11T19:55:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-11T19:55:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-11T19:55:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-11T19:55:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-11T19:55:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-11T19:55:58.056 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T19:55:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-11T19:55:58.056 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-11T19:55:58.057 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T19:55:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-11T19:55:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-11T19:55:58.057 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T19:55:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-11T19:55:58.057 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-11T19:55:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-11T19:55:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-11T19:55:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 41985sec preferred_lft 41985sec 2024-03-11T19:55:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link 2024-03-11T19:55:58.057 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-11T19:55:58.059 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-11T19:55:58.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:55:58.059 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-11T19:55:58.059 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-11T19:55:58.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link add name ceph-brx type bridge 2024-03-11T19:55:58.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip addr flush dev ceph-brx 2024-03-11T19:55:58.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set ceph-brx up 2024-03-11T19:55:58.059 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-11T19:55:58.059 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-11T19:55:58.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:55:58.521 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-11T19:55:58.601 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-11T19:55:58.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:55:58.603 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-11T19:55:58.674 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-11T19:55:58.674 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-11T19:55:58.674 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-11T19:55:58.674 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-11T19:55:58.674 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-11T19:55:58.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:55:58.675 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-11T19:55:58.675 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-11T19:55:58.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-11T19:55:58.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-11T19:55:58.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-11T19:55:58.675 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-11T19:55:59.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:55:59.371 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-11T19:55:59.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:55:59.424 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list-id 2024-03-11T19:55:59.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:55:59.475 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-11T19:55:59.475 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-11T19:55:59.475 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T19:55:59.475 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-11T19:55:59.475 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-11T19:55:59.615 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-11T19:55:59.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:55:59.615 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-11T19:55:59.615 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-11T19:55:59.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-11T19:55:59.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-11T19:55:59.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-11T19:55:59.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-11T19:55:59.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-11T19:55:59.615 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-11T19:55:59.669 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:55:59 smithi097 ceph-mon[31898]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:55:59.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:55:59.887 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-11T19:55:59.887 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-11T19:55:59.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-11T19:55:59.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set dev brx.0 master ceph-brx 2024-03-11T19:55:59.887 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-11T19:55:59.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:55:59 smithi114 ceph-mon[36376]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:56:00.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:55:59 smithi161 ceph-mon[36332]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:56:00.052 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-11T19:56:00.052 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T19:56:00.052 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-11T19:56:00.095 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-11T19:56:00.096 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T19:56:00.096 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-11T19:56:00.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo modprobe fuse 2024-03-11T19:56:00.283 DEBUG:teuthology.orchestra.run.smithi097:> cat /proc/self/mounts | awk '{print $2}' 2024-03-11T19:56:00.336 INFO:teuthology.orchestra.run.smithi097.stdout:/proc 2024-03-11T19:56:00.336 INFO:teuthology.orchestra.run.smithi097.stdout:/sys 2024-03-11T19:56:00.336 INFO:teuthology.orchestra.run.smithi097.stdout:/dev 2024-03-11T19:56:00.337 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/security 2024-03-11T19:56:00.337 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/shm 2024-03-11T19:56:00.337 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/pts 2024-03-11T19:56:00.337 INFO:teuthology.orchestra.run.smithi097.stdout:/run 2024-03-11T19:56:00.337 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/cgroup 2024-03-11T19:56:00.337 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/pstore 2024-03-11T19:56:00.338 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/bpf 2024-03-11T19:56:00.338 INFO:teuthology.orchestra.run.smithi097.stdout:/ 2024-03-11T19:56:00.338 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/selinux 2024-03-11T19:56:00.338 INFO:teuthology.orchestra.run.smithi097.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T19:56:00.338 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/mqueue 2024-03-11T19:56:00.338 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/hugepages 2024-03-11T19:56:00.339 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/debug 2024-03-11T19:56:00.339 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/tracing 2024-03-11T19:56:00.339 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/config 2024-03-11T19:56:00.339 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/fuse/connections 2024-03-11T19:56:00.339 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-sysctl.service 2024-03-11T19:56:00.339 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-11T19:56:00.340 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-11T19:56:00.340 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/nfs/rpc_pipefs 2024-03-11T19:56:00.340 INFO:teuthology.orchestra.run.smithi097.stdout:/run/user/1000 2024-03-11T19:56:00.340 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/ceph 2024-03-11T19:56:00.340 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay 2024-03-11T19:56:00.341 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/fa8621052974e23c9d8a56ff33761b9bd3dcad3a9e4b3bb3ed04e2c2649c9558/merged 2024-03-11T19:56:00.341 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/db10f39827646906b4622ef908fdf616d6b8e5658e008631616b52cb62f8a56d/merged 2024-03-11T19:56:00.341 INFO:teuthology.orchestra.run.smithi097.stdout:/run/user/0 2024-03-11T19:56:00.341 INFO:teuthology.orchestra.run.smithi097.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T19:56:00.341 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/f51d1f776cbb0e99d3850027c762f5e4cbaa3cb498e936b04c471a84fbd7fcbe/merged 2024-03-11T19:56:00.341 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/ec6b946d4c60bbe62468c69b4d50ed152000e7db099441e0970ee121e4fda4c3/merged 2024-03-11T19:56:00.341 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/267036aea224cf6f0771e4018b35f3c7d3c255b17526536c4ce4e7d07bbd5f18/merged 2024-03-11T19:56:00.342 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/6f2750daaa6b7ad719826904c69fe43dd873c083ae19a4ab00ef467697ba4ca1/merged 2024-03-11T19:56:00.342 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/2cc05717523036ea4bed32b1d08d3dd9102f2dc8588a30fef9aae040470e3a25/merged 2024-03-11T19:56:00.342 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/323711713fbc1b75913406a7a8faef5099ca51845c2fe828c65c818b405a116b/merged 2024-03-11T19:56:00.342 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/00a927362f891155cf8a8d7417fbd5aafe7090e0f54a27fac0706cccdbdce4e1/merged 2024-03-11T19:56:00.343 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/59f8364e95708ab6dc8efd5f3643dbbb449329bf274aaec909a0db8874c68def/merged 2024-03-11T19:56:00.343 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns 2024-03-11T19:56:00.343 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T19:56:00.343 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T19:56:00.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:56:00.345 DEBUG:teuthology.orchestra.run.smithi097:> ls /sys/fs/fuse/connections 2024-03-11T19:56:00.389 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-11T19:56:00.389 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-11T19:56:00.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo modprobe fuse 2024-03-11T19:56:00.518 DEBUG:teuthology.orchestra.run.smithi097:> cat /proc/self/mounts | awk '{print $2}' 2024-03-11T19:56:00.564 INFO:teuthology.orchestra.run.smithi097.stdout:/proc 2024-03-11T19:56:00.564 INFO:teuthology.orchestra.run.smithi097.stdout:/sys 2024-03-11T19:56:00.564 INFO:teuthology.orchestra.run.smithi097.stdout:/dev 2024-03-11T19:56:00.564 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/security 2024-03-11T19:56:00.564 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/shm 2024-03-11T19:56:00.564 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/pts 2024-03-11T19:56:00.564 INFO:teuthology.orchestra.run.smithi097.stdout:/run 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/cgroup 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/pstore 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/bpf 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/ 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/selinux 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/mqueue 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/hugepages 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/debug 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/tracing 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/kernel/config 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/sys/fs/fuse/connections 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-sysctl.service 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-11T19:56:00.565 INFO:teuthology.orchestra.run.smithi097.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/nfs/rpc_pipefs 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/run/user/1000 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/ceph 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/fa8621052974e23c9d8a56ff33761b9bd3dcad3a9e4b3bb3ed04e2c2649c9558/merged 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/db10f39827646906b4622ef908fdf616d6b8e5658e008631616b52cb62f8a56d/merged 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/run/user/0 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/f51d1f776cbb0e99d3850027c762f5e4cbaa3cb498e936b04c471a84fbd7fcbe/merged 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/ec6b946d4c60bbe62468c69b4d50ed152000e7db099441e0970ee121e4fda4c3/merged 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/267036aea224cf6f0771e4018b35f3c7d3c255b17526536c4ce4e7d07bbd5f18/merged 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/6f2750daaa6b7ad719826904c69fe43dd873c083ae19a4ab00ef467697ba4ca1/merged 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/2cc05717523036ea4bed32b1d08d3dd9102f2dc8588a30fef9aae040470e3a25/merged 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/323711713fbc1b75913406a7a8faef5099ca51845c2fe828c65c818b405a116b/merged 2024-03-11T19:56:00.566 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/00a927362f891155cf8a8d7417fbd5aafe7090e0f54a27fac0706cccdbdce4e1/merged 2024-03-11T19:56:00.567 INFO:teuthology.orchestra.run.smithi097.stdout:/var/lib/containers/storage/overlay/59f8364e95708ab6dc8efd5f3643dbbb449329bf274aaec909a0db8874c68def/merged 2024-03-11T19:56:00.567 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns 2024-03-11T19:56:00.567 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T19:56:00.567 INFO:teuthology.orchestra.run.smithi097.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T19:56:00.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:56:00.568 DEBUG:teuthology.orchestra.run.smithi097:> ls /sys/fs/fuse/connections 2024-03-11T19:56:00.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi097.stderr:2024-03-11T19:56:00.635+0000 7efc77836480 -1 init, newargv = 0x560b1ca71df0 newargc=15 2024-03-11T19:56:00.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi097.stderr:ceph-fuse[75909]: starting ceph client 2024-03-11T19:56:00.647 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi097.stderr:ceph-fuse[75909]: starting fuse 2024-03-11T19:56:00.657 INFO:teuthology.orchestra.run.smithi097.stdout:118 2024-03-11T19:56:00.657 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [118] 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> sudo stdin-killer -- python3 -c ' 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> import glob 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> import re 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> import os 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> import subprocess 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> def _find_admin_socket(client_name): 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> files = glob.glob(asok_path) 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> # Given a non-glob path, it better be there 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> if "*" not in asok_path: 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> assert(len(files) == 1) 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> return files[0] 2024-03-11T19:56:00.658 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-11T19:56:00.659 DEBUG:teuthology.orchestra.run.smithi097:> for f in files: 2024-03-11T19:56:00.659 DEBUG:teuthology.orchestra.run.smithi097:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-11T19:56:00.659 DEBUG:teuthology.orchestra.run.smithi097:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-11T19:56:00.659 DEBUG:teuthology.orchestra.run.smithi097:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-11T19:56:00.659 DEBUG:teuthology.orchestra.run.smithi097:> contents = proc_f.read() 2024-03-11T19:56:00.659 DEBUG:teuthology.orchestra.run.smithi097:> if mountpoint in contents: 2024-03-11T19:56:00.659 DEBUG:teuthology.orchestra.run.smithi097:> return f 2024-03-11T19:56:00.659 DEBUG:teuthology.orchestra.run.smithi097:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-11T19:56:00.659 DEBUG:teuthology.orchestra.run.smithi097:> 2024-03-11T19:56:00.659 DEBUG:teuthology.orchestra.run.smithi097:> print(_find_admin_socket("client.admin")) 2024-03-11T19:56:00.659 DEBUG:teuthology.orchestra.run.smithi097:> ' 2024-03-11T19:56:00.873 INFO:teuthology.orchestra.run.smithi097.stdout:/var/run/ceph/ceph-client.admin.75909.asok 2024-03-11T19:56:00.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-11T19:56:00.890 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75909.asok 2024-03-11T19:56:00.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:56:00.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75909.asok status 2024-03-11T19:56:01.127 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T19:56:01.127 INFO:teuthology.orchestra.run.smithi097.stdout: "metadata": { 2024-03-11T19:56:01.127 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_sha1": "94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T19:56:01.127 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "entity_id": "admin", 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "hostname": "smithi097", 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "pid": "75909", 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "root": "/" 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "dentry_count": 0, 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "dentry_pinned_count": 0, 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 14604, 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "inst": { 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "name": { 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "client", 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "num": 14604 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": { 2024-03-11T19:56:01.128 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "192.168.0.1:0", 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3976809282 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": { 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "192.168.0.1:0", 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3976809282 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: "inst_str": "client.14604 192.168.0.1:0/3976809282", 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: "addr_str": "192.168.0.1:0/3976809282", 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: "inode_count": 1, 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_epoch": 17, 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_epoch": 89, 2024-03-11T19:56:01.129 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_epoch_barrier": 0, 2024-03-11T19:56:01.130 INFO:teuthology.orchestra.run.smithi097.stdout: "blocklisted": false, 2024-03-11T19:56:01.130 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs" 2024-03-11T19:56:01.130 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T19:56:01.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:56:01.143 DEBUG:teuthology.orchestra.run.smithi097:> stat --file-system '--printf=%T 2024-03-11T19:56:01.143 DEBUG:teuthology.orchestra.run.smithi097:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-11T19:56:01.194 INFO:teuthology.orchestra.run.smithi097.stdout:fuseblk 2024-03-11T19:56:01.194 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-11T19:56:01.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:56:01.194 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-11T19:56:01.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:56:01.283 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-11T19:56:01.373 INFO:teuthology.orchestra.run.smithi097.stdout:# file: . 2024-03-11T19:56:01.373 INFO:teuthology.orchestra.run.smithi097.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-11T19:56:01.373 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:56:01.374 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:56:01.374 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T19:56:01.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:56:01.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.651+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1810151257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c281021e0 msgr2=0x7f8c28102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:01.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.651+0000 7f8c2ce00640 1 --2- 172.21.15.97:0/1810151257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c281021e0 0x7f8c28102660 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009860 tx=0x7f8c1002bb70 comp rx=0 tx=0).stop 2024-03-11T19:56:01.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.651+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1810151257 shutdown_connections 2024-03-11T19:56:01.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.651+0000 7f8c2ce00640 1 --2- 172.21.15.97:0/1810151257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c28102ba0 0x7f8c28109430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:01.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.651+0000 7f8c2ce00640 1 --2- 172.21.15.97:0/1810151257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c281021e0 0x7f8c28102660 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:01.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.651+0000 7f8c2ce00640 1 --2- 172.21.15.97:0/1810151257 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c28100ff0 0x7f8c28101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:01.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.651+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1810151257 >> 172.21.15.97:0/1810151257 conn(0x7f8c280fc560 msgr2=0x7f8c280fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:01.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.651+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1810151257 shutdown_connections 2024-03-11T19:56:01.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.651+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1810151257 wait complete. 2024-03-11T19:56:01.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.652+0000 7f8c2ce00640 1 Processor -- start 2024-03-11T19:56:01.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.653+0000 7f8c2ce00640 1 -- start start 2024-03-11T19:56:01.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.653+0000 7f8c2ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c28100ff0 0x7f8c28199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:01.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.653+0000 7f8c2ce00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c281021e0 0x7f8c28199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.653+0000 7f8c2ce00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c28102ba0 0x7f8c281a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.653+0000 7f8c2ce00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c281a0d30 con 0x7f8c28102ba0 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.653+0000 7f8c2ce00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c281a0ea0 con 0x7f8c28100ff0 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.653+0000 7f8c2ce00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c281a1060 con 0x7f8c281021e0 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.653+0000 7f8c27e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c28102ba0 0x7f8c281a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.653+0000 7f8c27400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c28100ff0 0x7f8c28199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.653+0000 7f8c26a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c281021e0 0x7f8c28199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.653+0000 7f8c27e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c28102ba0 0x7f8c281a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46112/0 (socket says 172.21.15.97:46112) 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.653+0000 7f8c27e00640 1 -- 172.21.15.97:0/1074629657 learned_addr learned my addr 172.21.15.97:0/1074629657 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.654+0000 7f8c27400640 1 -- 172.21.15.97:0/1074629657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c281021e0 msgr2=0x7f8c28199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.654+0000 7f8c27400640 1 --2- 172.21.15.97:0/1074629657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c281021e0 0x7f8c28199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.654+0000 7f8c27400640 1 -- 172.21.15.97:0/1074629657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c28102ba0 msgr2=0x7f8c281a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.654+0000 7f8c27400640 1 --2- 172.21.15.97:0/1074629657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c28102ba0 0x7f8c281a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:01.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.654+0000 7f8c27400640 1 -- 172.21.15.97:0/1074629657 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c18009560 con 0x7f8c28100ff0 2024-03-11T19:56:01.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.654+0000 7f8c27400640 1 --2- 172.21.15.97:0/1074629657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c28100ff0 0x7f8c28199240 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8c28102040 tx=0x7f8c1800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:01.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.654+0000 7f8c1fe00640 1 -- 172.21.15.97:0/1074629657 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18004440 con 0x7f8c28100ff0 2024-03-11T19:56:01.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.654+0000 7f8c1fe00640 1 -- 172.21.15.97:0/1074629657 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c1800cda0 con 0x7f8c28100ff0 2024-03-11T19:56:01.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.654+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1074629657 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c10009490 con 0x7f8c28100ff0 2024-03-11T19:56:01.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.654+0000 7f8c1fe00640 1 -- 172.21.15.97:0/1074629657 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18019480 con 0x7f8c28100ff0 2024-03-11T19:56:01.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.654+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1074629657 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c281a16e0 con 0x7f8c28100ff0 2024-03-11T19:56:01.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.655+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1074629657 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c2810c9f0 con 0x7f8c28100ff0 2024-03-11T19:56:01.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.657+0000 7f8c1fe00640 1 -- 172.21.15.97:0/1074629657 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8c18003680 con 0x7f8c28100ff0 2024-03-11T19:56:01.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.658+0000 7f8c1fe00640 1 --2- 172.21.15.97:0/1074629657 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8c0c0a1ab0 0x7f8c0c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:01.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.659+0000 7f8c1fe00640 1 -- 172.21.15.97:0/1074629657 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8c18014070 con 0x7f8c28100ff0 2024-03-11T19:56:01.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.659+0000 7f8c26a00640 1 --2- 172.21.15.97:0/1074629657 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8c0c0a1ab0 0x7f8c0c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:01.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.659+0000 7f8c26a00640 1 --2- 172.21.15.97:0/1074629657 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8c0c0a1ab0 0x7f8c0c0a3f70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009830 tx=0x7f8c10034be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:01.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.663+0000 7f8c1fe00640 1 -- 172.21.15.97:0/1074629657 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8c18096b60 con 0x7f8c28100ff0 2024-03-11T19:56:01.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.869+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1074629657 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c2810cbd0 con 0x7f8c28100ff0 2024-03-11T19:56:01.872 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.871+0000 7f8c1fe00640 1 -- 172.21.15.97:0/1074629657 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24152 (secure 0 0 0) 0x7f8c1809a220 con 0x7f8c28100ff0 2024-03-11T19:56:01.873 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:56:01.873 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":89,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:55:50.540865+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:56:01.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.874+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1074629657 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8c0c0a1ab0 msgr2=0x7f8c0c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:01.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.874+0000 7f8c2ce00640 1 --2- 172.21.15.97:0/1074629657 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8c0c0a1ab0 0x7f8c0c0a3f70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009830 tx=0x7f8c10034be0 comp rx=0 tx=0).stop 2024-03-11T19:56:01.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.874+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1074629657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c28100ff0 msgr2=0x7f8c28199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:01.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.874+0000 7f8c2ce00640 1 --2- 172.21.15.97:0/1074629657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c28100ff0 0x7f8c28199240 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8c28102040 tx=0x7f8c1800ee50 comp rx=0 tx=0).stop 2024-03-11T19:56:01.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.874+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1074629657 shutdown_connections 2024-03-11T19:56:01.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.874+0000 7f8c2ce00640 1 --2- 172.21.15.97:0/1074629657 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8c0c0a1ab0 0x7f8c0c0a3f70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:01.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.874+0000 7f8c2ce00640 1 --2- 172.21.15.97:0/1074629657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8c28102ba0 0x7f8c281a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:01.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.874+0000 7f8c2ce00640 1 --2- 172.21.15.97:0/1074629657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c281021e0 0x7f8c28199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:01.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.874+0000 7f8c2ce00640 1 --2- 172.21.15.97:0/1074629657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c28100ff0 0x7f8c28199240 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:01.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.874+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1074629657 >> 172.21.15.97:0/1074629657 conn(0x7f8c280fc560 msgr2=0x7f8c280733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:01.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.874+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1074629657 shutdown_connections 2024-03-11T19:56:01.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:01.874+0000 7f8c2ce00640 1 -- 172.21.15.97:0/1074629657 wait complete. 2024-03-11T19:56:01.891 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:01 smithi097 ceph-mon[31898]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:56:01.895 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:50:29.051985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:55:43.424076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:55:44.652160+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T19:55:48.598077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T19:56:01.896 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:56:01.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:01 smithi114 ceph-mon[36376]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:56:02.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:01 smithi161 ceph-mon[36332]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-11T19:56:02.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.110+0000 7f4cb3200640 1 -- 172.21.15.97:0/2990645238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4102240 msgr2=0x7f4cb41046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:02.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.110+0000 7f4cb3200640 1 --2- 172.21.15.97:0/2990645238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4102240 0x7f4cb41046b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009900 tx=0x7f4c9c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:56:02.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.110+0000 7f4cb3200640 1 -- 172.21.15.97:0/2990645238 shutdown_connections 2024-03-11T19:56:02.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.110+0000 7f4cb3200640 1 --2- 172.21.15.97:0/2990645238 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cb4104bf0 0x7f4cb4105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.110+0000 7f4cb3200640 1 --2- 172.21.15.97:0/2990645238 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4102240 0x7f4cb41046b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.110+0000 7f4cb3200640 1 --2- 172.21.15.97:0/2990645238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb40ff8d0 0x7f4cb4101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.110+0000 7f4cb3200640 1 -- 172.21.15.97:0/2990645238 >> 172.21.15.97:0/2990645238 conn(0x7f4cb40fb4f0 msgr2=0x7f4cb40fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:02.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.110+0000 7f4cb3200640 1 -- 172.21.15.97:0/2990645238 shutdown_connections 2024-03-11T19:56:02.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.110+0000 7f4cb3200640 1 -- 172.21.15.97:0/2990645238 wait complete. 2024-03-11T19:56:02.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.111+0000 7f4cb3200640 1 Processor -- start 2024-03-11T19:56:02.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.111+0000 7f4cb3200640 1 -- start start 2024-03-11T19:56:02.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.111+0000 7f4cb3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb40ff8d0 0x7f4cb419d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:02.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.111+0000 7f4cb3200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cb4102240 0x7f4cb419dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:02.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.111+0000 7f4cb3200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4104bf0 0x7f4cb41a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:02.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.111+0000 7f4cb3200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb41a51a0 con 0x7f4cb4104bf0 2024-03-11T19:56:02.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.111+0000 7f4cb3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb41a5310 con 0x7f4cb40ff8d0 2024-03-11T19:56:02.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.111+0000 7f4cb3200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb41a54d0 con 0x7f4cb4102240 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.111+0000 7f4cb1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb40ff8d0 0x7f4cb419d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.111+0000 7f4cb1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb40ff8d0 0x7f4cb419d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:53200/0 (socket says 172.21.15.97:53200) 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.111+0000 7f4cb1e00640 1 -- 172.21.15.97:0/3864930830 learned_addr learned my addr 172.21.15.97:0/3864930830 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.111+0000 7f4cb2800640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4104bf0 0x7f4cb41a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.112+0000 7f4ca9400640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cb4102240 0x7f4cb419dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.112+0000 7f4cb1e00640 1 -- 172.21.15.97:0/3864930830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cb4102240 msgr2=0x7f4cb419dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.112+0000 7f4cb1e00640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cb4102240 0x7f4cb419dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.112+0000 7f4cb1e00640 1 -- 172.21.15.97:0/3864930830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4104bf0 msgr2=0x7f4cb41a4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.112+0000 7f4cb1e00640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4104bf0 0x7f4cb41a4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.112+0000 7f4cb1e00640 1 -- 172.21.15.97:0/3864930830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c009560 con 0x7f4cb40ff8d0 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.112+0000 7f4cb1e00640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb40ff8d0 0x7f4cb419d6b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400e9f0 tx=0x7f4ca400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.112+0000 7f4cab400640 1 -- 172.21.15.97:0/3864930830 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4009700 con 0x7f4cb40ff8d0 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.112+0000 7f4cab400640 1 -- 172.21.15.97:0/3864930830 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ca400cd80 con 0x7f4cb40ff8d0 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.112+0000 7f4cab400640 1 -- 172.21.15.97:0/3864930830 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4018920 con 0x7f4cb40ff8d0 2024-03-11T19:56:02.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.112+0000 7f4cb3200640 1 -- 172.21.15.97:0/3864930830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb41a57b0 con 0x7f4cb40ff8d0 2024-03-11T19:56:02.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.113+0000 7f4cb3200640 1 -- 172.21.15.97:0/3864930830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb41a5d10 con 0x7f4cb40ff8d0 2024-03-11T19:56:02.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.114+0000 7f4cb3200640 1 -- 172.21.15.97:0/3864930830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb40fcfe0 con 0x7f4cb40ff8d0 2024-03-11T19:56:02.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.115+0000 7f4cab400640 1 -- 172.21.15.97:0/3864930830 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4ca4018a80 con 0x7f4cb40ff8d0 2024-03-11T19:56:02.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.117+0000 7f4cab400640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c980a1ab0 0x7f4c980a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:02.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.117+0000 7f4cab400640 1 -- 172.21.15.97:0/3864930830 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4ca4014070 con 0x7f4cb40ff8d0 2024-03-11T19:56:02.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.117+0000 7f4ca9400640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c980a1ab0 0x7f4c980a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:02.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.118+0000 7f4ca9400640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c980a1ab0 0x7f4c980a3f70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4cb419ebc0 tx=0x7f4c9c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:02.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.119+0000 7f4cab400640 1 -- 172.21.15.97:0/3864930830 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4ca4096970 con 0x7f4cb40ff8d0 2024-03-11T19:56:02.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.314+0000 7f4cb3200640 1 -- 172.21.15.97:0/3864930830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4cb402cdd0 con 0x7f4cb40ff8d0 2024-03-11T19:56:02.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.315+0000 7f4cab400640 1 -- 172.21.15.97:0/3864930830 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ca409a030 con 0x7f4cb40ff8d0 2024-03-11T19:56:02.317 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-11T19:56:02.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.317+0000 7f4cb3200640 1 -- 172.21.15.97:0/3864930830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c980a1ab0 msgr2=0x7f4c980a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:02.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.317+0000 7f4cb3200640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c980a1ab0 0x7f4c980a3f70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4cb419ebc0 tx=0x7f4c9c000bc0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.317+0000 7f4cb3200640 1 -- 172.21.15.97:0/3864930830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb40ff8d0 msgr2=0x7f4cb419d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:02.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.317+0000 7f4cb3200640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb40ff8d0 0x7f4cb419d6b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400e9f0 tx=0x7f4ca400eeb0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.318+0000 7f4cb3200640 1 -- 172.21.15.97:0/3864930830 shutdown_connections 2024-03-11T19:56:02.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.318+0000 7f4cb3200640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4c980a1ab0 0x7f4c980a3f70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.318+0000 7f4cb3200640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cb4104bf0 0x7f4cb41a4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.318+0000 7f4cb3200640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cb4102240 0x7f4cb419dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.318+0000 7f4cb3200640 1 --2- 172.21.15.97:0/3864930830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb40ff8d0 0x7f4cb419d6b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.318+0000 7f4cb3200640 1 -- 172.21.15.97:0/3864930830 >> 172.21.15.97:0/3864930830 conn(0x7f4cb40fb4f0 msgr2=0x7f4cb4100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:02.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.318+0000 7f4cb3200640 1 -- 172.21.15.97:0/3864930830 shutdown_connections 2024-03-11T19:56:02.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.318+0000 7f4cb3200640 1 -- 172.21.15.97:0/3864930830 wait complete. 2024-03-11T19:56:02.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:56:02.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.552+0000 7f82b1800640 1 -- 172.21.15.97:0/2358759384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82ac102250 msgr2=0x7f82ac1026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:02.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.552+0000 7f82b1800640 1 --2- 172.21.15.97:0/2358759384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82ac102250 0x7f82ac1026d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8294009860 tx=0x7f829402bb70 comp rx=0 tx=0).stop 2024-03-11T19:56:02.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.552+0000 7f82b1800640 1 -- 172.21.15.97:0/2358759384 shutdown_connections 2024-03-11T19:56:02.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.552+0000 7f82b1800640 1 --2- 172.21.15.97:0/2358759384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82ac102c10 0x7f82ac1094a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.552+0000 7f82b1800640 1 --2- 172.21.15.97:0/2358759384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82ac102250 0x7f82ac1026d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.552+0000 7f82b1800640 1 --2- 172.21.15.97:0/2358759384 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ac101060 0x7f82ac101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.552+0000 7f82b1800640 1 -- 172.21.15.97:0/2358759384 >> 172.21.15.97:0/2358759384 conn(0x7f82ac0fc5f0 msgr2=0x7f82ac0fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:02.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.553+0000 7f82b1800640 1 -- 172.21.15.97:0/2358759384 shutdown_connections 2024-03-11T19:56:02.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.553+0000 7f82b1800640 1 -- 172.21.15.97:0/2358759384 wait complete. 2024-03-11T19:56:02.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.554+0000 7f82b1800640 1 Processor -- start 2024-03-11T19:56:02.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.554+0000 7f82b1800640 1 -- start start 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.554+0000 7f82b1800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82ac101060 0x7f82ac1970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.554+0000 7f82b1800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ac102250 0x7f82ac197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.554+0000 7f82b1800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82ac102c10 0x7f82ac19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.554+0000 7f82b1800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac19ebb0 con 0x7f82ac101060 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.554+0000 7f82b1800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac19ed20 con 0x7f82ac102c10 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.554+0000 7f82b1800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac19eee0 con 0x7f82ac102250 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.555+0000 7f82abe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82ac101060 0x7f82ac1970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.555+0000 7f82b0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82ac102c10 0x7f82ac19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.555+0000 7f82abe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82ac101060 0x7f82ac1970c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46152/0 (socket says 172.21.15.97:46152) 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.555+0000 7f82abe00640 1 -- 172.21.15.97:0/2134165536 learned_addr learned my addr 172.21.15.97:0/2134165536 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.555+0000 7f82a3400640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ac102250 0x7f82ac197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.555+0000 7f82b0e00640 1 -- 172.21.15.97:0/2134165536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ac102250 msgr2=0x7f82ac197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.555+0000 7f82b0e00640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ac102250 0x7f82ac197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.555+0000 7f82b0e00640 1 -- 172.21.15.97:0/2134165536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82ac101060 msgr2=0x7f82ac1970c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.555+0000 7f82b0e00640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82ac101060 0x7f82ac1970c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.555+0000 7f82b0e00640 1 -- 172.21.15.97:0/2134165536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8294009490 con 0x7f82ac102c10 2024-03-11T19:56:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.555+0000 7f82abe00640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82ac101060 0x7f82ac1970c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:02.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.555+0000 7f82b0e00640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82ac102c10 0x7f82ac19e670 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f82a400ef00 tx=0x7f82a400c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:02.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.556+0000 7f82a9600640 1 -- 172.21.15.97:0/2134165536 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a4004b20 con 0x7f82ac102c10 2024-03-11T19:56:02.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.556+0000 7f82a9600640 1 -- 172.21.15.97:0/2134165536 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82a4003e50 con 0x7f82ac102c10 2024-03-11T19:56:02.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.556+0000 7f82a9600640 1 -- 172.21.15.97:0/2134165536 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a400b570 con 0x7f82ac102c10 2024-03-11T19:56:02.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.556+0000 7f82b1800640 1 -- 172.21.15.97:0/2134165536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ac19f1c0 con 0x7f82ac102c10 2024-03-11T19:56:02.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.556+0000 7f82b1800640 1 -- 172.21.15.97:0/2134165536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ac19f6a0 con 0x7f82ac102c10 2024-03-11T19:56:02.559 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.557+0000 7f82b1800640 1 -- 172.21.15.97:0/2134165536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ac108380 con 0x7f82ac102c10 2024-03-11T19:56:02.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.559+0000 7f82a9600640 1 -- 172.21.15.97:0/2134165536 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f82a400a040 con 0x7f82ac102c10 2024-03-11T19:56:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.561+0000 7f82a9600640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f82900a1ab0 0x7f82900a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.561+0000 7f82a9600640 1 -- 172.21.15.97:0/2134165536 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f82a40cdf30 con 0x7f82ac102c10 2024-03-11T19:56:02.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.561+0000 7f82abe00640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f82900a1ab0 0x7f82900a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:02.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.562+0000 7f82abe00640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f82900a1ab0 0x7f82900a3f70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f829c0058c0 tx=0x7f829c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:02.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.563+0000 7f82a9600640 1 -- 172.21.15.97:0/2134165536 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f82a40fbd20 con 0x7f82ac102c10 2024-03-11T19:56:02.669 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:02 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1074629657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:02.669 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:02 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3864930830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:56:02.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.761+0000 7f82b1800640 1 -- 172.21.15.97:0/2134165536 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f82ac02cda0 con 0x7f82ac102c10 2024-03-11T19:56:02.763 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.761+0000 7f82a9600640 1 -- 172.21.15.97:0/2134165536 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f82a401a020 con 0x7f82ac102c10 2024-03-11T19:56:02.763 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T19:56:02.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.764+0000 7f82b1800640 1 -- 172.21.15.97:0/2134165536 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f82900a1ab0 msgr2=0x7f82900a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:02.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.764+0000 7f82b1800640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f82900a1ab0 0x7f82900a3f70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f829c0058c0 tx=0x7f829c005f90 comp rx=0 tx=0).stop 2024-03-11T19:56:02.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.764+0000 7f82b1800640 1 -- 172.21.15.97:0/2134165536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82ac102c10 msgr2=0x7f82ac19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:02.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.764+0000 7f82b1800640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82ac102c10 0x7f82ac19e670 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f82a400ef00 tx=0x7f82a400c740 comp rx=0 tx=0).stop 2024-03-11T19:56:02.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.764+0000 7f82b1800640 1 -- 172.21.15.97:0/2134165536 shutdown_connections 2024-03-11T19:56:02.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.764+0000 7f82b1800640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82ac102c10 0x7f82ac19e670 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.764+0000 7f82b1800640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82ac102250 0x7f82ac197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.764+0000 7f82b1800640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f82900a1ab0 0x7f82900a3f70 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.764+0000 7f82b1800640 1 --2- 172.21.15.97:0/2134165536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f82ac101060 0x7f82ac1970c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:02.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.764+0000 7f82b1800640 1 -- 172.21.15.97:0/2134165536 >> 172.21.15.97:0/2134165536 conn(0x7f82ac0fc5f0 msgr2=0x7f82ac105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:02.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.764+0000 7f82b1800640 1 -- 172.21.15.97:0/2134165536 shutdown_connections 2024-03-11T19:56:02.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:02.765+0000 7f82b1800640 1 -- 172.21.15.97:0/2134165536 wait complete. 2024-03-11T19:56:02.784 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:56:02.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:02 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1074629657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:02.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:02 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3864930830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:56:03.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.000+0000 7f390b200640 1 -- 172.21.15.97:0/588641262 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f390c104e50 msgr2=0x7f390c10b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:03.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.000+0000 7f390b200640 1 --2- 172.21.15.97:0/588641262 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f390c104e50 0x7f390c10b6e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3904009ca0 tx=0x7f390402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.001+0000 7f390b200640 1 -- 172.21.15.97:0/588641262 shutdown_connections 2024-03-11T19:56:03.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.001+0000 7f390b200640 1 --2- 172.21.15.97:0/588641262 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f390c104e50 0x7f390c10b6e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.001+0000 7f390b200640 1 --2- 172.21.15.97:0/588641262 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f390c104490 0x7f390c104910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.001+0000 7f390b200640 1 --2- 172.21.15.97:0/588641262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f390c1032a0 0x7f390c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.001+0000 7f390b200640 1 -- 172.21.15.97:0/588641262 >> 172.21.15.97:0/588641262 conn(0x7f390c0fe830 msgr2=0x7f390c100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:03.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.001+0000 7f390b200640 1 -- 172.21.15.97:0/588641262 shutdown_connections 2024-03-11T19:56:03.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.001+0000 7f390b200640 1 -- 172.21.15.97:0/588641262 wait complete. 2024-03-11T19:56:03.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.002+0000 7f390b200640 1 Processor -- start 2024-03-11T19:56:03.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.002+0000 7f390b200640 1 -- start start 2024-03-11T19:56:03.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.002+0000 7f390b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f390c1032a0 0x7f390c1992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:03.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.002+0000 7f390b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f390c104490 0x7f390c1997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.002+0000 7f390b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f390c104e50 0x7f390c1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.002+0000 7f390b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c1a0d90 con 0x7f390c104e50 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.002+0000 7f390b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c1a0f00 con 0x7f390c1032a0 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.002+0000 7f390b200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c1a10c0 con 0x7f390c104490 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.003+0000 7f3909e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f390c1032a0 0x7f390c1992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.003+0000 7f3909e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f390c1032a0 0x7f390c1992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:53248/0 (socket says 172.21.15.97:53248) 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.003+0000 7f3909e00640 1 -- 172.21.15.97:0/3553828891 learned_addr learned my addr 172.21.15.97:0/3553828891 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.003+0000 7f3901400640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f390c104490 0x7f390c1997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.003+0000 7f390a800640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f390c104e50 0x7f390c1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.003+0000 7f3909e00640 1 -- 172.21.15.97:0/3553828891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f390c104490 msgr2=0x7f390c1997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.003+0000 7f3909e00640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f390c104490 0x7f390c1997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.003+0000 7f3909e00640 1 -- 172.21.15.97:0/3553828891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f390c104e50 msgr2=0x7f390c1a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.003+0000 7f3909e00640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f390c104e50 0x7f390c1a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.003+0000 7f3909e00640 1 -- 172.21.15.97:0/3553828891 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f4009490 con 0x7f390c1032a0 2024-03-11T19:56:03.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.003+0000 7f3909e00640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f390c1032a0 0x7f390c1992a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00e9c0 tx=0x7f38fc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:03.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.004+0000 7f3903400640 1 -- 172.21.15.97:0/3553828891 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc009700 con 0x7f390c1032a0 2024-03-11T19:56:03.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.004+0000 7f3903400640 1 -- 172.21.15.97:0/3553828891 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38fc00cd80 con 0x7f390c1032a0 2024-03-11T19:56:03.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.004+0000 7f390b200640 1 -- 172.21.15.97:0/3553828891 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3904009900 con 0x7f390c1032a0 2024-03-11T19:56:03.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.004+0000 7f3903400640 1 -- 172.21.15.97:0/3553828891 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc010430 con 0x7f390c1032a0 2024-03-11T19:56:03.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.004+0000 7f390b200640 1 -- 172.21.15.97:0/3553828891 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c1a1770 con 0x7f390c1032a0 2024-03-11T19:56:03.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.005+0000 7f390b200640 1 -- 172.21.15.97:0/3553828891 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390c10a560 con 0x7f390c1032a0 2024-03-11T19:56:03.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.007+0000 7f3903400640 1 -- 172.21.15.97:0/3553828891 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f38fc003680 con 0x7f390c1032a0 2024-03-11T19:56:03.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.008+0000 7f3903400640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f38f00a1ab0 0x7f38f00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:03.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.008+0000 7f3903400640 1 -- 172.21.15.97:0/3553828891 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f38fc014070 con 0x7f390c1032a0 2024-03-11T19:56:03.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.008+0000 7f3901400640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f38f00a1ab0 0x7f38f00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:03.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.009+0000 7f3901400640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f38f00a1ab0 0x7f38f00a3f70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f390c19a7b0 tx=0x7f38f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:03.012 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.010+0000 7f3903400640 1 -- 172.21.15.97:0/3553828891 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f38fc096950 con 0x7f390c1032a0 2024-03-11T19:56:03.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:02 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1074629657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:03.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:02 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3864930830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:56:03.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.209+0000 7f390b200640 1 -- 172.21.15.97:0/3553828891 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f390c10ec80 con 0x7f390c1032a0 2024-03-11T19:56:03.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.210+0000 7f3903400640 1 -- 172.21.15.97:0/3553828891 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f38fc09a010 con 0x7f390c1032a0 2024-03-11T19:56:03.211 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-11T19:56:03.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.212+0000 7f390b200640 1 -- 172.21.15.97:0/3553828891 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f38f00a1ab0 msgr2=0x7f38f00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:03.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.212+0000 7f390b200640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f38f00a1ab0 0x7f38f00a3f70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f390c19a7b0 tx=0x7f38f4005f90 comp rx=0 tx=0).stop 2024-03-11T19:56:03.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.212+0000 7f390b200640 1 -- 172.21.15.97:0/3553828891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f390c1032a0 msgr2=0x7f390c1992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:03.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.212+0000 7f390b200640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f390c1032a0 0x7f390c1992a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00e9c0 tx=0x7f38fc00ee80 comp rx=0 tx=0).stop 2024-03-11T19:56:03.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.212+0000 7f390b200640 1 -- 172.21.15.97:0/3553828891 shutdown_connections 2024-03-11T19:56:03.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.212+0000 7f390b200640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f38f00a1ab0 0x7f38f00a3f70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.212+0000 7f390b200640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f390c104e50 0x7f390c1a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.212+0000 7f390b200640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f390c104490 0x7f390c1997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.212+0000 7f390b200640 1 --2- 172.21.15.97:0/3553828891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f390c1032a0 0x7f390c1992a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.212+0000 7f390b200640 1 -- 172.21.15.97:0/3553828891 >> 172.21.15.97:0/3553828891 conn(0x7f390c0fe830 msgr2=0x7f390c107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:03.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.212+0000 7f390b200640 1 -- 172.21.15.97:0/3553828891 shutdown_connections 2024-03-11T19:56:03.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.213+0000 7f390b200640 1 -- 172.21.15.97:0/3553828891 wait complete. 2024-03-11T19:56:03.232 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T19:56:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.441+0000 7f6484a00640 1 -- 172.21.15.97:0/1064579719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6480109050 msgr2=0x7f648010b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.441+0000 7f6484a00640 1 --2- 172.21.15.97:0/1064579719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6480109050 0x7f648010b4e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6478009be0 tx=0x7f647802bef0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.441+0000 7f6484a00640 1 -- 172.21.15.97:0/1064579719 shutdown_connections 2024-03-11T19:56:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.441+0000 7f6484a00640 1 --2- 172.21.15.97:0/1064579719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6480109050 0x7f648010b4e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.441+0000 7f6484a00640 1 --2- 172.21.15.97:0/1064579719 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6480073be0 0x7f6480074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.441+0000 7f6484a00640 1 --2- 172.21.15.97:0/1064579719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64800751c0 0x7f6480073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.441+0000 7f6484a00640 1 -- 172.21.15.97:0/1064579719 >> 172.21.15.97:0/1064579719 conn(0x7f64800fc700 msgr2=0x7f64800feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.441+0000 7f6484a00640 1 -- 172.21.15.97:0/1064579719 shutdown_connections 2024-03-11T19:56:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.441+0000 7f6484a00640 1 -- 172.21.15.97:0/1064579719 wait complete. 2024-03-11T19:56:03.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.442+0000 7f6484a00640 1 Processor -- start 2024-03-11T19:56:03.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.442+0000 7f6484a00640 1 -- start start 2024-03-11T19:56:03.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.442+0000 7f6484a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6480073be0 0x7f64801993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:03.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.442+0000 7f6484a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64800751c0 0x7f6480199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:03.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.442+0000 7f6484a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6480109050 0x7f64801a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:03.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.442+0000 7f6484a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64801a0eb0 con 0x7f6480073be0 2024-03-11T19:56:03.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.442+0000 7f6484a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64801a1020 con 0x7f64800751c0 2024-03-11T19:56:03.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.442+0000 7f6484a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64801a11e0 con 0x7f6480109050 2024-03-11T19:56:03.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.443+0000 7f647de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6480073be0 0x7f64801993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:03.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.443+0000 7f647de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6480073be0 0x7f64801993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:46190/0 (socket says 172.21.15.97:46190) 2024-03-11T19:56:03.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.443+0000 7f647de00640 1 -- 172.21.15.97:0/2669779812 learned_addr learned my addr 172.21.15.97:0/2669779812 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:03.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.443+0000 7f647e800640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6480109050 0x7f64801a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:03.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.443+0000 7f647d400640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64800751c0 0x7f6480199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:03.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.443+0000 7f647de00640 1 -- 172.21.15.97:0/2669779812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6480109050 msgr2=0x7f64801a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:03.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.443+0000 7f647de00640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6480109050 0x7f64801a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.443+0000 7f647de00640 1 -- 172.21.15.97:0/2669779812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64800751c0 msgr2=0x7f6480199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:03.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.443+0000 7f647de00640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64800751c0 0x7f6480199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.443+0000 7f647de00640 1 -- 172.21.15.97:0/2669779812 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6468009560 con 0x7f6480073be0 2024-03-11T19:56:03.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.443+0000 7f647de00640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6480073be0 0x7f64801993c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f647000e9f0 tx=0x7f647000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:03.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.444+0000 7f6476a00640 1 -- 172.21.15.97:0/2669779812 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470009720 con 0x7f6480073be0 2024-03-11T19:56:03.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.444+0000 7f6484a00640 1 -- 172.21.15.97:0/2669779812 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6478009840 con 0x7f6480073be0 2024-03-11T19:56:03.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.444+0000 7f6476a00640 1 -- 172.21.15.97:0/2669779812 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f647000cd80 con 0x7f6480073be0 2024-03-11T19:56:03.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.444+0000 7f6476a00640 1 -- 172.21.15.97:0/2669779812 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470010690 con 0x7f6480073be0 2024-03-11T19:56:03.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.444+0000 7f6484a00640 1 -- 172.21.15.97:0/2669779812 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64801a17f0 con 0x7f6480073be0 2024-03-11T19:56:03.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.445+0000 7f6484a00640 1 -- 172.21.15.97:0/2669779812 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6480107d80 con 0x7f6480073be0 2024-03-11T19:56:03.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.448+0000 7f6476a00640 1 -- 172.21.15.97:0/2669779812 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f64700046c0 con 0x7f6480073be0 2024-03-11T19:56:03.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.450+0000 7f6476a00640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f64640a1a30 0x7f64640a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:03.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.450+0000 7f6476a00640 1 -- 172.21.15.97:0/2669779812 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f6470014070 con 0x7f6480073be0 2024-03-11T19:56:03.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.450+0000 7f647d400640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f64640a1a30 0x7f64640a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:03.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.451+0000 7f647d400640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f64640a1a30 0x7f64640a3ef0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f648019a8d0 tx=0x7f6468009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:03.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.453+0000 7f6476a00640 1 -- 172.21.15.97:0/2669779812 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6470096ba0 con 0x7f6480073be0 2024-03-11T19:56:03.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.649+0000 7f6484a00640 1 -- 172.21.15.97:0/2669779812 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f648010ede0 con 0x7f6480073be0 2024-03-11T19:56:03.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.650+0000 7f6476a00640 1 -- 172.21.15.97:0/2669779812 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f647009a260 con 0x7f6480073be0 2024-03-11T19:56:03.652 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T19:56:03.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.653+0000 7f6484a00640 1 -- 172.21.15.97:0/2669779812 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f64640a1a30 msgr2=0x7f64640a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:03.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.653+0000 7f6484a00640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f64640a1a30 0x7f64640a3ef0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f648019a8d0 tx=0x7f6468009260 comp rx=0 tx=0).stop 2024-03-11T19:56:03.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.653+0000 7f6484a00640 1 -- 172.21.15.97:0/2669779812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6480073be0 msgr2=0x7f64801993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:03.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.653+0000 7f6484a00640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6480073be0 0x7f64801993c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f647000e9f0 tx=0x7f647000edf0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.653+0000 7f6484a00640 1 -- 172.21.15.97:0/2669779812 shutdown_connections 2024-03-11T19:56:03.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.653+0000 7f6484a00640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6480109050 0x7f64801a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.653+0000 7f6484a00640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64800751c0 0x7f6480199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.653+0000 7f6484a00640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f64640a1a30 0x7f64640a3ef0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.653+0000 7f6484a00640 1 --2- 172.21.15.97:0/2669779812 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6480073be0 0x7f64801993c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.653+0000 7f6484a00640 1 -- 172.21.15.97:0/2669779812 >> 172.21.15.97:0/2669779812 conn(0x7f64800fc700 msgr2=0x7f6480104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:03.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.653+0000 7f6484a00640 1 -- 172.21.15.97:0/2669779812 shutdown_connections 2024-03-11T19:56:03.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.653+0000 7f6484a00640 1 -- 172.21.15.97:0/2669779812 wait complete. 2024-03-11T19:56:03.672 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-11T19:56:03.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.873+0000 7eff65400640 1 -- 172.21.15.97:0/206916330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff601033c0 msgr2=0x7eff601037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:03.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.873+0000 7eff65400640 1 --2- 172.21.15.97:0/206916330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff601033c0 0x7eff601037e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7eff48009900 tx=0x7eff4802bc30 comp rx=0 tx=0).stop 2024-03-11T19:56:03.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.874+0000 7eff65400640 1 -- 172.21.15.97:0/206916330 shutdown_connections 2024-03-11T19:56:03.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.874+0000 7eff65400640 1 --2- 172.21.15.97:0/206916330 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff60104f70 0x7eff6010b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.874+0000 7eff65400640 1 --2- 172.21.15.97:0/206916330 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601045b0 0x7eff60104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.874+0000 7eff65400640 1 --2- 172.21.15.97:0/206916330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff601033c0 0x7eff601037e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.874+0000 7eff65400640 1 -- 172.21.15.97:0/206916330 >> 172.21.15.97:0/206916330 conn(0x7eff600fe930 msgr2=0x7eff60100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:03.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.874+0000 7eff65400640 1 -- 172.21.15.97:0/206916330 shutdown_connections 2024-03-11T19:56:03.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.874+0000 7eff65400640 1 -- 172.21.15.97:0/206916330 wait complete. 2024-03-11T19:56:03.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.875+0000 7eff65400640 1 Processor -- start 2024-03-11T19:56:03.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.875+0000 7eff65400640 1 -- start start 2024-03-11T19:56:03.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.875+0000 7eff65400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff601033c0 0x7eff601994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:03.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.875+0000 7eff65400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601045b0 0x7eff601999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:03.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff65400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff60104f70 0x7eff601a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:03.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff65400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff601a0fa0 con 0x7eff60104f70 2024-03-11T19:56:03.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff65400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff601a1110 con 0x7eff601033c0 2024-03-11T19:56:03.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff65400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff601a12d0 con 0x7eff601045b0 2024-03-11T19:56:03.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff5de00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601045b0 0x7eff601999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:03.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff5e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff601033c0 0x7eff601994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:03.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff5f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff60104f70 0x7eff601a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:03.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff5de00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601045b0 0x7eff601999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:35264/0 (socket says 172.21.15.97:35264) 2024-03-11T19:56:03.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff5e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff601033c0 0x7eff601994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:43026/0 (socket says 172.21.15.97:43026) 2024-03-11T19:56:03.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff5de00640 1 -- 172.21.15.97:0/1424709140 learned_addr learned my addr 172.21.15.97:0/1424709140 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:03.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff5e800640 1 -- 172.21.15.97:0/1424709140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601045b0 msgr2=0x7eff601999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:03.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff5e800640 1 --2- 172.21.15.97:0/1424709140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601045b0 0x7eff601999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff5e800640 1 -- 172.21.15.97:0/1424709140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff60104f70 msgr2=0x7eff601a0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:03.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff5e800640 1 --2- 172.21.15.97:0/1424709140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff60104f70 0x7eff601a0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:03.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.876+0000 7eff5e800640 1 -- 172.21.15.97:0/1424709140 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff48009560 con 0x7eff601033c0 2024-03-11T19:56:03.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.877+0000 7eff5e800640 1 --2- 172.21.15.97:0/1424709140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff601033c0 0x7eff601994b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7eff48036040 tx=0x7eff48005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:03.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.877+0000 7eff57400640 1 -- 172.21.15.97:0/1424709140 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff480043d0 con 0x7eff601033c0 2024-03-11T19:56:03.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.877+0000 7eff57400640 1 -- 172.21.15.97:0/1424709140 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff480052d0 con 0x7eff601033c0 2024-03-11T19:56:03.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.877+0000 7eff57400640 1 -- 172.21.15.97:0/1424709140 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff48003cd0 con 0x7eff601033c0 2024-03-11T19:56:03.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.877+0000 7eff65400640 1 -- 172.21.15.97:0/1424709140 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff601a1550 con 0x7eff601033c0 2024-03-11T19:56:03.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.877+0000 7eff65400640 1 -- 172.21.15.97:0/1424709140 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff601a1a80 con 0x7eff601033c0 2024-03-11T19:56:03.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.878+0000 7eff65400640 1 -- 172.21.15.97:0/1424709140 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff6010a6e0 con 0x7eff601033c0 2024-03-11T19:56:03.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.880+0000 7eff57400640 1 -- 172.21.15.97:0/1424709140 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7eff48005440 con 0x7eff601033c0 2024-03-11T19:56:03.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.881+0000 7eff57400640 1 --2- 172.21.15.97:0/1424709140 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7eff440a1ab0 0x7eff440a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:03.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.881+0000 7eff57400640 1 -- 172.21.15.97:0/1424709140 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7eff480ef710 con 0x7eff601033c0 2024-03-11T19:56:03.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.882+0000 7eff5de00640 1 --2- 172.21.15.97:0/1424709140 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7eff440a1ab0 0x7eff440a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:03.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.882+0000 7eff5de00640 1 --2- 172.21.15.97:0/1424709140 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7eff440a1ab0 0x7eff440a3f70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7eff6019a9c0 tx=0x7eff50009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:03.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:03.886+0000 7eff57400640 1 -- 172.21.15.97:0/1424709140 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7eff480b8680 con 0x7eff601033c0 2024-03-11T19:56:03.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:03 smithi097 ceph-mon[31898]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 359 B/s wr, 0 op/s 2024-03-11T19:56:03.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:03 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2134165536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:56:03.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:03 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3553828891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:56:03.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:03 smithi114 ceph-mon[36376]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 359 B/s wr, 0 op/s 2024-03-11T19:56:03.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:03 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2134165536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:56:03.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:03 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3553828891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:56:04.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:03 smithi161 ceph-mon[36332]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 359 B/s wr, 0 op/s 2024-03-11T19:56:04.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:03 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2134165536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:56:04.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:03 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3553828891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:56:04.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.094+0000 7eff65400640 1 -- 172.21.15.97:0/1424709140 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7eff6002cdd0 con 0x7eff601033c0 2024-03-11T19:56:04.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.095+0000 7eff57400640 1 -- 172.21.15.97:0/1424709140 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+590 (secure 0 0 0) 0x7eff480bbd40 con 0x7eff601033c0 2024-03-11T19:56:04.096 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/4129219534 2024-03-12T19:47:44.356323+0000 2024-03-11T19:56:04.097 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/3932792931 2024-03-12T19:47:44.356323+0000 2024-03-11T19:56:04.097 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6801/1708150324 2024-03-12T19:47:44.356323+0000 2024-03-11T19:56:04.097 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6800/1708150324 2024-03-12T19:47:44.356323+0000 2024-03-11T19:56:04.097 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6800/3216682632 2024-03-12T19:47:09.368819+0000 2024-03-11T19:56:04.097 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/3236990094 2024-03-12T19:47:09.368819+0000 2024-03-11T19:56:04.097 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/1301912133 2024-03-12T19:47:09.368819+0000 2024-03-11T19:56:04.097 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6801/3216682632 2024-03-12T19:47:09.368819+0000 2024-03-11T19:56:04.097 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/46019613 2024-03-12T19:47:44.356323+0000 2024-03-11T19:56:04.097 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/4194903751 2024-03-12T19:47:09.368819+0000 2024-03-11T19:56:04.097 INFO:teuthology.orchestra.run.smithi097.stderr:listed 10 entries 2024-03-11T19:56:04.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.097+0000 7eff65400640 1 -- 172.21.15.97:0/1424709140 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7eff440a1ab0 msgr2=0x7eff440a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:04.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.097+0000 7eff65400640 1 --2- 172.21.15.97:0/1424709140 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7eff440a1ab0 0x7eff440a3f70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7eff6019a9c0 tx=0x7eff50009500 comp rx=0 tx=0).stop 2024-03-11T19:56:04.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.097+0000 7eff65400640 1 -- 172.21.15.97:0/1424709140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff601033c0 msgr2=0x7eff601994b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:04.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.097+0000 7eff65400640 1 --2- 172.21.15.97:0/1424709140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff601033c0 0x7eff601994b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7eff48036040 tx=0x7eff48005e50 comp rx=0 tx=0).stop 2024-03-11T19:56:04.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.097+0000 7eff65400640 1 -- 172.21.15.97:0/1424709140 shutdown_connections 2024-03-11T19:56:04.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.097+0000 7eff65400640 1 --2- 172.21.15.97:0/1424709140 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7eff440a1ab0 0x7eff440a3f70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.097+0000 7eff65400640 1 --2- 172.21.15.97:0/1424709140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7eff60104f70 0x7eff601a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.097+0000 7eff65400640 1 --2- 172.21.15.97:0/1424709140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff601045b0 0x7eff601999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.097+0000 7eff65400640 1 --2- 172.21.15.97:0/1424709140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff601033c0 0x7eff601994b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.097+0000 7eff65400640 1 -- 172.21.15.97:0/1424709140 >> 172.21.15.97:0/1424709140 conn(0x7eff600fe930 msgr2=0x7eff601078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:04.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.098+0000 7eff65400640 1 -- 172.21.15.97:0/1424709140 shutdown_connections 2024-03-11T19:56:04.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.098+0000 7eff65400640 1 -- 172.21.15.97:0/1424709140 wait complete. 2024-03-11T19:56:04.119 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:56:04.119 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T19:56:04.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:56:04.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.371+0000 7face9a00640 1 -- 172.21.15.97:0/4008713087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face4075140 msgr2=0x7face4073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:04.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.371+0000 7face9a00640 1 --2- 172.21.15.97:0/4008713087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face4075140 0x7face4073590 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7facd4009900 tx=0x7facd402bcd0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.371+0000 7face9a00640 1 -- 172.21.15.97:0/4008713087 shutdown_connections 2024-03-11T19:56:04.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.371+0000 7face9a00640 1 --2- 172.21.15.97:0/4008713087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7face4108e90 0x7face410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.371+0000 7face9a00640 1 --2- 172.21.15.97:0/4008713087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7face4073b60 0x7face4073fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.371+0000 7face9a00640 1 --2- 172.21.15.97:0/4008713087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face4075140 0x7face4073590 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.371+0000 7face9a00640 1 -- 172.21.15.97:0/4008713087 >> 172.21.15.97:0/4008713087 conn(0x7face40fc550 msgr2=0x7face40fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:04.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.371+0000 7face9a00640 1 -- 172.21.15.97:0/4008713087 shutdown_connections 2024-03-11T19:56:04.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.371+0000 7face9a00640 1 -- 172.21.15.97:0/4008713087 wait complete. 2024-03-11T19:56:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.372+0000 7face9a00640 1 Processor -- start 2024-03-11T19:56:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.372+0000 7face9a00640 1 -- start start 2024-03-11T19:56:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face9a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7face4073b60 0x7face4199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face9a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face4075140 0x7face41997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face9a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7face4108e90 0x7face41a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face9a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face41a0d50 con 0x7face4073b60 2024-03-11T19:56:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face9a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face41a0ec0 con 0x7face4075140 2024-03-11T19:56:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face9a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face41a1080 con 0x7face4108e90 2024-03-11T19:56:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7face4073b60 0x7face4199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7face4073b60 0x7face4199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56628/0 (socket says 172.21.15.97:56628) 2024-03-11T19:56:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face3e00640 1 -- 172.21.15.97:0/3319562881 learned_addr learned my addr 172.21.15.97:0/3319562881 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face4075140 0x7face41997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:04.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face9000640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7face4108e90 0x7face41a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:04.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face3400640 1 -- 172.21.15.97:0/3319562881 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7face4108e90 msgr2=0x7face41a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:04.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face3400640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7face4108e90 0x7face41a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face3400640 1 -- 172.21.15.97:0/3319562881 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7face4073b60 msgr2=0x7face4199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:04.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face3400640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7face4073b60 0x7face4199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face3400640 1 -- 172.21.15.97:0/3319562881 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd4009560 con 0x7face4075140 2024-03-11T19:56:04.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.373+0000 7face3e00640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7face4073b60 0x7face4199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:04.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.374+0000 7face3400640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face4075140 0x7face41997a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7facd0005fd0 tx=0x7facd000dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:04.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.374+0000 7face0c00640 1 -- 172.21.15.97:0/3319562881 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd0009c30 con 0x7face4075140 2024-03-11T19:56:04.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.374+0000 7face0c00640 1 -- 172.21.15.97:0/3319562881 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facd0004510 con 0x7face4075140 2024-03-11T19:56:04.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.374+0000 7face9a00640 1 -- 172.21.15.97:0/3319562881 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face41a1360 con 0x7face4075140 2024-03-11T19:56:04.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.374+0000 7face0c00640 1 -- 172.21.15.97:0/3319562881 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd0003e70 con 0x7face4075140 2024-03-11T19:56:04.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.374+0000 7face9a00640 1 -- 172.21.15.97:0/3319562881 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face41a1870 con 0x7face4075140 2024-03-11T19:56:04.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.375+0000 7face9a00640 1 -- 172.21.15.97:0/3319562881 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face4107c50 con 0x7face4075140 2024-03-11T19:56:04.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.378+0000 7face0c00640 1 -- 172.21.15.97:0/3319562881 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7facd0004000 con 0x7face4075140 2024-03-11T19:56:04.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.379+0000 7face0c00640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7facc80a1a30 0x7facc80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:04.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.379+0000 7face0c00640 1 -- 172.21.15.97:0/3319562881 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7facd00cd4e0 con 0x7face4075140 2024-03-11T19:56:04.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.380+0000 7face3e00640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7facc80a1a30 0x7facc80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:04.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.380+0000 7face3e00640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7facc80a1a30 0x7facc80a3ef0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7facd4006010 tx=0x7facd402c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:04.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.381+0000 7face0c00640 1 -- 172.21.15.97:0/3319562881 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7facd0096500 con 0x7face4075140 2024-03-11T19:56:04.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.577+0000 7face9a00640 1 -- 172.21.15.97:0/3319562881 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7face410eea0 con 0x7face4075140 2024-03-11T19:56:04.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.580+0000 7face0c00640 1 -- 172.21.15.97:0/3319562881 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24152 (secure 0 0 0) 0x7facd0020020 con 0x7face4075140 2024-03-11T19:56:04.582 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:56:04.582 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":89,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:55:50.540865+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:56:04.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.583+0000 7face9a00640 1 -- 172.21.15.97:0/3319562881 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7facc80a1a30 msgr2=0x7facc80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:04.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.583+0000 7face9a00640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7facc80a1a30 0x7facc80a3ef0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7facd4006010 tx=0x7facd402c880 comp rx=0 tx=0).stop 2024-03-11T19:56:04.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.583+0000 7face9a00640 1 -- 172.21.15.97:0/3319562881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face4075140 msgr2=0x7face41997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:04.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.584+0000 7face9a00640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face4075140 0x7face41997a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7facd0005fd0 tx=0x7facd000dde0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.584+0000 7face9a00640 1 -- 172.21.15.97:0/3319562881 shutdown_connections 2024-03-11T19:56:04.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.584+0000 7face9a00640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7face4108e90 0x7face41a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.584+0000 7face9a00640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face4075140 0x7face41997a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.584+0000 7face9a00640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7facc80a1a30 0x7facc80a3ef0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.584+0000 7face9a00640 1 --2- 172.21.15.97:0/3319562881 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7face4073b60 0x7face4199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.584+0000 7face9a00640 1 -- 172.21.15.97:0/3319562881 >> 172.21.15.97:0/3319562881 conn(0x7face40fc550 msgr2=0x7face4104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:04.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.584+0000 7face9a00640 1 -- 172.21.15.97:0/3319562881 shutdown_connections 2024-03-11T19:56:04.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.584+0000 7face9a00640 1 -- 172.21.15.97:0/3319562881 wait complete. 2024-03-11T19:56:04.605 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:50:29.051985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:55:43.424076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:55:44.652160+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T19:55:48.598077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T19:56:04.605 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:56:04.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.819+0000 7f0a50000640 1 -- 172.21.15.97:0/3036105917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a48109350 msgr2=0x7f0a4810fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:04.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.819+0000 7f0a50000640 1 --2- 172.21.15.97:0/3036105917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a48109350 0x7f0a4810fbe0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0a40009ca0 tx=0x7f0a4002c100 comp rx=0 tx=0).stop 2024-03-11T19:56:04.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.820+0000 7f0a50000640 1 -- 172.21.15.97:0/3036105917 shutdown_connections 2024-03-11T19:56:04.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.820+0000 7f0a50000640 1 --2- 172.21.15.97:0/3036105917 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a48109350 0x7f0a4810fbe0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.820+0000 7f0a50000640 1 --2- 172.21.15.97:0/3036105917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a48108990 0x7f0a48108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.820+0000 7f0a50000640 1 --2- 172.21.15.97:0/3036105917 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a481077a0 0x7f0a48107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.820+0000 7f0a50000640 1 -- 172.21.15.97:0/3036105917 >> 172.21.15.97:0/3036105917 conn(0x7f0a48076740 msgr2=0x7f0a48078b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:04.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.820+0000 7f0a50000640 1 -- 172.21.15.97:0/3036105917 shutdown_connections 2024-03-11T19:56:04.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.820+0000 7f0a50000640 1 -- 172.21.15.97:0/3036105917 wait complete. 2024-03-11T19:56:04.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.821+0000 7f0a50000640 1 Processor -- start 2024-03-11T19:56:04.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.821+0000 7f0a50000640 1 -- start start 2024-03-11T19:56:04.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.821+0000 7f0a50000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a481077a0 0x7f0a4819d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:04.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.821+0000 7f0a50000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a48108990 0x7f0a4819dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:04.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.821+0000 7f0a50000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a48109350 0x7f0a481a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:04.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.821+0000 7f0a50000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a481a5310 con 0x7f0a48109350 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.821+0000 7f0a50000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a481a5480 con 0x7f0a48108990 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.821+0000 7f0a50000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a481a5640 con 0x7f0a481077a0 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.821+0000 7f0a4e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a48109350 0x7f0a481a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.821+0000 7f0a4e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a48109350 0x7f0a481a4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56648/0 (socket says 172.21.15.97:56648) 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.821+0000 7f0a4e000640 1 -- 172.21.15.97:0/2264644555 learned_addr learned my addr 172.21.15.97:0/2264644555 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.821+0000 7f0a4d600640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a481077a0 0x7f0a4819d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.821+0000 7f0a4cc00640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a48108990 0x7f0a4819dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.822+0000 7f0a4e000640 1 -- 172.21.15.97:0/2264644555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a481077a0 msgr2=0x7f0a4819d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.822+0000 7f0a4e000640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a481077a0 0x7f0a4819d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.822+0000 7f0a4e000640 1 -- 172.21.15.97:0/2264644555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a48108990 msgr2=0x7f0a4819dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.822+0000 7f0a4e000640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a48108990 0x7f0a4819dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.822+0000 7f0a4e000640 1 -- 172.21.15.97:0/2264644555 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a40009900 con 0x7f0a48109350 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.822+0000 7f0a4d600640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a481077a0 0x7f0a4819d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:04.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.822+0000 7f0a4e000640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a48109350 0x7f0a481a4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0a40037040 tx=0x7f0a40009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:04.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.822+0000 7f0a46000640 1 -- 172.21.15.97:0/2264644555 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40034850 con 0x7f0a48109350 2024-03-11T19:56:04.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.822+0000 7f0a46000640 1 -- 172.21.15.97:0/2264644555 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a40003e50 con 0x7f0a48109350 2024-03-11T19:56:04.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.822+0000 7f0a46000640 1 -- 172.21.15.97:0/2264644555 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40040450 con 0x7f0a48109350 2024-03-11T19:56:04.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.822+0000 7f0a50000640 1 -- 172.21.15.97:0/2264644555 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a481a58c0 con 0x7f0a48109350 2024-03-11T19:56:04.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.823+0000 7f0a50000640 1 -- 172.21.15.97:0/2264644555 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a481a5d20 con 0x7f0a48109350 2024-03-11T19:56:04.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.824+0000 7f0a50000640 1 -- 172.21.15.97:0/2264644555 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4810eac0 con 0x7f0a48109350 2024-03-11T19:56:04.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.826+0000 7f0a46000640 1 -- 172.21.15.97:0/2264644555 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0a4000a040 con 0x7f0a48109350 2024-03-11T19:56:04.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.827+0000 7f0a46000640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0a300a1a30 0x7f0a300a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:04.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.827+0000 7f0a46000640 1 -- 172.21.15.97:0/2264644555 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0a400f0850 con 0x7f0a48109350 2024-03-11T19:56:04.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.827+0000 7f0a4d600640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0a300a1a30 0x7f0a300a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:04.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.828+0000 7f0a4d600640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0a300a1a30 0x7f0a300a3ef0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0a481087f0 tx=0x7f0a3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:04.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:04.830+0000 7f0a46000640 1 -- 172.21.15.97:0/2264644555 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0a400b97c0 con 0x7f0a48109350 2024-03-11T19:56:04.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:04 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2669779812' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:56:04.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:04 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1424709140' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T19:56:04.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:04 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2669779812' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:56:04.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:04 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1424709140' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T19:56:05.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.025+0000 7f0a50000640 1 -- 172.21.15.97:0/2264644555 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a4802ce40 con 0x7f0a48109350 2024-03-11T19:56:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.026+0000 7f0a46000640 1 -- 172.21.15.97:0/2264644555 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a400bce80 con 0x7f0a48109350 2024-03-11T19:56:05.027 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-11T19:56:05.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.028+0000 7f0a50000640 1 -- 172.21.15.97:0/2264644555 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0a300a1a30 msgr2=0x7f0a300a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:05.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.028+0000 7f0a50000640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0a300a1a30 0x7f0a300a3ef0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0a481087f0 tx=0x7f0a3c005f90 comp rx=0 tx=0).stop 2024-03-11T19:56:05.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.028+0000 7f0a50000640 1 -- 172.21.15.97:0/2264644555 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a48109350 msgr2=0x7f0a481a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:05.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.028+0000 7f0a50000640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a48109350 0x7f0a481a4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0a40037040 tx=0x7f0a40009360 comp rx=0 tx=0).stop 2024-03-11T19:56:05.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.029+0000 7f0a50000640 1 -- 172.21.15.97:0/2264644555 shutdown_connections 2024-03-11T19:56:05.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.029+0000 7f0a50000640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0a300a1a30 0x7f0a300a3ef0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.029+0000 7f0a50000640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0a48109350 0x7f0a481a4dd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.029+0000 7f0a50000640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a48108990 0x7f0a4819dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.029+0000 7f0a50000640 1 --2- 172.21.15.97:0/2264644555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a481077a0 0x7f0a4819d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.029+0000 7f0a50000640 1 -- 172.21.15.97:0/2264644555 >> 172.21.15.97:0/2264644555 conn(0x7f0a48076740 msgr2=0x7f0a4810bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:05.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.029+0000 7f0a50000640 1 -- 172.21.15.97:0/2264644555 shutdown_connections 2024-03-11T19:56:05.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.029+0000 7f0a50000640 1 -- 172.21.15.97:0/2264644555 wait complete. 2024-03-11T19:56:05.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:04 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2669779812' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:56:05.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:04 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1424709140' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T19:56:05.052 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:56:05.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.264+0000 7fcdfb200640 1 -- 172.21.15.97:0/4211641863 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdf41045b0 msgr2=0x7fcdf4104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:05.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.264+0000 7fcdfb200640 1 --2- 172.21.15.97:0/4211641863 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdf41045b0 0x7fcdf4104a30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009ca0 tx=0x7fcddc02c100 comp rx=0 tx=0).stop 2024-03-11T19:56:05.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.264+0000 7fcdfb200640 1 -- 172.21.15.97:0/4211641863 shutdown_connections 2024-03-11T19:56:05.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.264+0000 7fcdfb200640 1 --2- 172.21.15.97:0/4211641863 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdf4104f70 0x7fcdf410b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.264+0000 7fcdfb200640 1 --2- 172.21.15.97:0/4211641863 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdf41045b0 0x7fcdf4104a30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.264+0000 7fcdfb200640 1 --2- 172.21.15.97:0/4211641863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf41033c0 0x7fcdf41037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.264+0000 7fcdfb200640 1 -- 172.21.15.97:0/4211641863 >> 172.21.15.97:0/4211641863 conn(0x7fcdf40fe930 msgr2=0x7fcdf4100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:05.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.264+0000 7fcdfb200640 1 -- 172.21.15.97:0/4211641863 shutdown_connections 2024-03-11T19:56:05.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.264+0000 7fcdfb200640 1 -- 172.21.15.97:0/4211641863 wait complete. 2024-03-11T19:56:05.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.265+0000 7fcdfb200640 1 Processor -- start 2024-03-11T19:56:05.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.265+0000 7fcdfb200640 1 -- start start 2024-03-11T19:56:05.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.265+0000 7fcdfb200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdf41033c0 0x7fcdf4199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:05.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.265+0000 7fcdfb200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdf41045b0 0x7fcdf4199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:05.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.265+0000 7fcdfb200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf4104f70 0x7fcdf41a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:05.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.265+0000 7fcdfb200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf41a0f00 con 0x7fcdf41045b0 2024-03-11T19:56:05.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.265+0000 7fcdfb200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf41a1070 con 0x7fcdf4104f70 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.265+0000 7fcdfb200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf41a1230 con 0x7fcdf41033c0 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.265+0000 7fcdf3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdf41045b0 0x7fcdf4199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.266+0000 7fcdf3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdf41045b0 0x7fcdf4199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56680/0 (socket says 172.21.15.97:56680) 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.266+0000 7fcdf9200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf4104f70 0x7fcdf41a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.266+0000 7fcdf3400640 1 -- 172.21.15.97:0/1230438843 learned_addr learned my addr 172.21.15.97:0/1230438843 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.266+0000 7fcdf3e00640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdf41033c0 0x7fcdf4199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.266+0000 7fcdf3400640 1 -- 172.21.15.97:0/1230438843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdf41033c0 msgr2=0x7fcdf4199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.266+0000 7fcdf3400640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdf41033c0 0x7fcdf4199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.266+0000 7fcdf3400640 1 -- 172.21.15.97:0/1230438843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf4104f70 msgr2=0x7fcdf41a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.266+0000 7fcdf3400640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf4104f70 0x7fcdf41a09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.266+0000 7fcdf3400640 1 -- 172.21.15.97:0/1230438843 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec009490 con 0x7fcdf41045b0 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.266+0000 7fcdf9200640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf4104f70 0x7fcdf41a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.266+0000 7fcdf3400640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdf41045b0 0x7fcdf4199950 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcddc032730 tx=0x7fcddc032fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.266+0000 7fcdf0c00640 1 -- 172.21.15.97:0/1230438843 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc034640 con 0x7fcdf41045b0 2024-03-11T19:56:05.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.267+0000 7fcdfb200640 1 -- 172.21.15.97:0/1230438843 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcddc009900 con 0x7fcdf41045b0 2024-03-11T19:56:05.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.267+0000 7fcdf0c00640 1 -- 172.21.15.97:0/1230438843 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcddc003e50 con 0x7fcdf41045b0 2024-03-11T19:56:05.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.267+0000 7fcdf0c00640 1 -- 172.21.15.97:0/1230438843 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc040450 con 0x7fcdf41045b0 2024-03-11T19:56:05.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.267+0000 7fcdfb200640 1 -- 172.21.15.97:0/1230438843 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf41a17a0 con 0x7fcdf41045b0 2024-03-11T19:56:05.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.268+0000 7fcdfb200640 1 -- 172.21.15.97:0/1230438843 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf410a760 con 0x7fcdf41045b0 2024-03-11T19:56:05.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.270+0000 7fcdf0c00640 1 -- 172.21.15.97:0/1230438843 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcddc00a040 con 0x7fcdf41045b0 2024-03-11T19:56:05.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.272+0000 7fcdf0c00640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcdd80a1a30 0x7fcdd80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:05.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.272+0000 7fcdf0c00640 1 -- 172.21.15.97:0/1230438843 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fcddc0f0040 con 0x7fcdf41045b0 2024-03-11T19:56:05.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.272+0000 7fcdf3e00640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcdd80a1a30 0x7fcdd80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:05.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.273+0000 7fcdf3e00640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcdd80a1a30 0x7fcdd80a3ef0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4104410 tx=0x7fcdec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:05.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.276+0000 7fcdf0c00640 1 -- 172.21.15.97:0/1230438843 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcddc0b9010 con 0x7fcdf41045b0 2024-03-11T19:56:05.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.473+0000 7fcdfb200640 1 -- 172.21.15.97:0/1230438843 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcdf402ce40 con 0x7fcdf41045b0 2024-03-11T19:56:05.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.474+0000 7fcdf0c00640 1 -- 172.21.15.97:0/1230438843 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcddc0bc6d0 con 0x7fcdf41045b0 2024-03-11T19:56:05.475 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T19:56:05.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.478+0000 7fcdfb200640 1 -- 172.21.15.97:0/1230438843 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcdd80a1a30 msgr2=0x7fcdd80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:05.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.478+0000 7fcdfb200640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcdd80a1a30 0x7fcdd80a3ef0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4104410 tx=0x7fcdec005f90 comp rx=0 tx=0).stop 2024-03-11T19:56:05.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.478+0000 7fcdfb200640 1 -- 172.21.15.97:0/1230438843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdf41045b0 msgr2=0x7fcdf4199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:05.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.478+0000 7fcdfb200640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdf41045b0 0x7fcdf4199950 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcddc032730 tx=0x7fcddc032fa0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.478+0000 7fcdfb200640 1 -- 172.21.15.97:0/1230438843 shutdown_connections 2024-03-11T19:56:05.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.478+0000 7fcdfb200640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdf4104f70 0x7fcdf41a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.478+0000 7fcdfb200640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcdd80a1a30 0x7fcdd80a3ef0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.478+0000 7fcdfb200640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcdf41045b0 0x7fcdf4199950 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.478+0000 7fcdfb200640 1 --2- 172.21.15.97:0/1230438843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcdf41033c0 0x7fcdf4199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.478+0000 7fcdfb200640 1 -- 172.21.15.97:0/1230438843 >> 172.21.15.97:0/1230438843 conn(0x7fcdf40fe930 msgr2=0x7fcdf4107930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:05.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.478+0000 7fcdfb200640 1 -- 172.21.15.97:0/1230438843 shutdown_connections 2024-03-11T19:56:05.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.478+0000 7fcdfb200640 1 -- 172.21.15.97:0/1230438843 wait complete. 2024-03-11T19:56:05.502 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:56:05.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.711+0000 7f0fe8a00640 1 -- 172.21.15.97:0/2315639087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe4104f70 msgr2=0x7f0fe410b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:05.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.711+0000 7f0fe8a00640 1 --2- 172.21.15.97:0/2315639087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe4104f70 0x7f0fe410b800 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009be0 tx=0x7f0fdc02bef0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.711+0000 7f0fe8a00640 1 -- 172.21.15.97:0/2315639087 shutdown_connections 2024-03-11T19:56:05.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.711+0000 7f0fe8a00640 1 --2- 172.21.15.97:0/2315639087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe4104f70 0x7f0fe410b800 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.711+0000 7f0fe8a00640 1 --2- 172.21.15.97:0/2315639087 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fe41045b0 0x7f0fe4104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.711+0000 7f0fe8a00640 1 --2- 172.21.15.97:0/2315639087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fe41033c0 0x7f0fe41037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.711+0000 7f0fe8a00640 1 -- 172.21.15.97:0/2315639087 >> 172.21.15.97:0/2315639087 conn(0x7f0fe40fe930 msgr2=0x7f0fe4100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:05.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.711+0000 7f0fe8a00640 1 -- 172.21.15.97:0/2315639087 shutdown_connections 2024-03-11T19:56:05.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.711+0000 7f0fe8a00640 1 -- 172.21.15.97:0/2315639087 wait complete. 2024-03-11T19:56:05.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.712+0000 7f0fe8a00640 1 Processor -- start 2024-03-11T19:56:05.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.712+0000 7f0fe8a00640 1 -- start start 2024-03-11T19:56:05.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.712+0000 7f0fe8a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fe41033c0 0x7f0fe41993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:05.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.712+0000 7f0fe8a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fe41045b0 0x7f0fe4199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:05.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.712+0000 7f0fe8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe4104f70 0x7f0fe41a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:05.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.712+0000 7f0fe8a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe41a0ee0 con 0x7f0fe41045b0 2024-03-11T19:56:05.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.712+0000 7f0fe8a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe41a1050 con 0x7f0fe4104f70 2024-03-11T19:56:05.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.712+0000 7f0fe8a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe41a1210 con 0x7f0fe41033c0 2024-03-11T19:56:05.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.713+0000 7f0fd9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fe41045b0 0x7f0fe4199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:05.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.713+0000 7f0fd9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fe41045b0 0x7f0fe4199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56698/0 (socket says 172.21.15.97:56698) 2024-03-11T19:56:05.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.713+0000 7f0fd9400640 1 -- 172.21.15.97:0/3792212288 learned_addr learned my addr 172.21.15.97:0/3792212288 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:05.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.713+0000 7f0fe2800640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe4104f70 0x7f0fe41a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:05.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.713+0000 7f0fe1e00640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fe41033c0 0x7f0fe41993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:05.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.713+0000 7f0fd9400640 1 -- 172.21.15.97:0/3792212288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fe41033c0 msgr2=0x7f0fe41993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:05.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.713+0000 7f0fd9400640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fe41033c0 0x7f0fe41993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.713+0000 7f0fd9400640 1 -- 172.21.15.97:0/3792212288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe4104f70 msgr2=0x7f0fe41a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:05.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.713+0000 7f0fd9400640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe4104f70 0x7f0fe41a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.713+0000 7f0fd9400640 1 -- 172.21.15.97:0/3792212288 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc009840 con 0x7f0fe41045b0 2024-03-11T19:56:05.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.713+0000 7f0fd9400640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fe41045b0 0x7f0fe4199930 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00e9c0 tx=0x7f0fcc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:05.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.714+0000 7f0fdb400640 1 -- 172.21.15.97:0/3792212288 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc00ccc0 con 0x7f0fe41045b0 2024-03-11T19:56:05.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.714+0000 7f0fdb400640 1 -- 172.21.15.97:0/3792212288 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fcc00ce20 con 0x7f0fe41045b0 2024-03-11T19:56:05.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.714+0000 7f0fe8a00640 1 -- 172.21.15.97:0/3792212288 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe41a14f0 con 0x7f0fe41045b0 2024-03-11T19:56:05.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.714+0000 7f0fdb400640 1 -- 172.21.15.97:0/3792212288 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc010430 con 0x7f0fe41045b0 2024-03-11T19:56:05.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.714+0000 7f0fe8a00640 1 -- 172.21.15.97:0/3792212288 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe41a1a00 con 0x7f0fe41045b0 2024-03-11T19:56:05.717 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.715+0000 7f0fe8a00640 1 -- 172.21.15.97:0/3792212288 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe410a680 con 0x7f0fe41045b0 2024-03-11T19:56:05.718 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.717+0000 7f0fdb400640 1 -- 172.21.15.97:0/3792212288 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0fcc004f80 con 0x7f0fe41045b0 2024-03-11T19:56:05.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.718+0000 7f0fdb400640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0fc40a1a30 0x7f0fc40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:05.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.718+0000 7f0fdb400640 1 -- 172.21.15.97:0/3792212288 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0fcc014070 con 0x7f0fe41045b0 2024-03-11T19:56:05.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.718+0000 7f0fe1e00640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0fc40a1a30 0x7f0fc40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:05.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.719+0000 7f0fe1e00640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0fc40a1a30 0x7f0fc40a3ef0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4104410 tx=0x7f0fd4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:05.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.720+0000 7f0fdb400640 1 -- 172.21.15.97:0/3792212288 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0fcc096af0 con 0x7f0fe41045b0 2024-03-11T19:56:05.892 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:05 smithi114 ceph-mon[36376]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T19:56:05.892 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:05 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3319562881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:05.892 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:05 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2264644555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:56:05.893 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:05 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1230438843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:56:05.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.905+0000 7f0fe8a00640 1 -- 172.21.15.97:0/3792212288 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0fe410efb0 con 0x7f0fe41045b0 2024-03-11T19:56:05.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.906+0000 7f0fdb400640 1 -- 172.21.15.97:0/3792212288 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0fcc09a1b0 con 0x7f0fe41045b0 2024-03-11T19:56:05.908 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-11T19:56:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.908+0000 7f0fe8a00640 1 -- 172.21.15.97:0/3792212288 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0fc40a1a30 msgr2=0x7f0fc40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.908+0000 7f0fe8a00640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0fc40a1a30 0x7f0fc40a3ef0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4104410 tx=0x7f0fd4005f90 comp rx=0 tx=0).stop 2024-03-11T19:56:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.908+0000 7f0fe8a00640 1 -- 172.21.15.97:0/3792212288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fe41045b0 msgr2=0x7f0fe4199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.908+0000 7f0fe8a00640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fe41045b0 0x7f0fe4199930 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00e9c0 tx=0x7f0fcc00ee80 comp rx=0 tx=0).stop 2024-03-11T19:56:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.908+0000 7f0fe8a00640 1 -- 172.21.15.97:0/3792212288 shutdown_connections 2024-03-11T19:56:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.908+0000 7f0fe8a00640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe4104f70 0x7f0fe41a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.908+0000 7f0fe8a00640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0fc40a1a30 0x7f0fc40a3ef0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.908+0000 7f0fe8a00640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0fe41045b0 0x7f0fe4199930 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.908+0000 7f0fe8a00640 1 --2- 172.21.15.97:0/3792212288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0fe41033c0 0x7f0fe41993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.908+0000 7f0fe8a00640 1 -- 172.21.15.97:0/3792212288 >> 172.21.15.97:0/3792212288 conn(0x7f0fe40fe930 msgr2=0x7f0fe4107850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:05.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.909+0000 7f0fe8a00640 1 -- 172.21.15.97:0/3792212288 shutdown_connections 2024-03-11T19:56:05.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:05.909+0000 7f0fe8a00640 1 -- 172.21.15.97:0/3792212288 wait complete. 2024-03-11T19:56:05.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:05 smithi097 ceph-mon[31898]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T19:56:05.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:05 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3319562881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:05.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:05 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2264644555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:56:05.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:05 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1230438843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:56:05.928 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T19:56:06.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:05 smithi161 ceph-mon[36332]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T19:56:06.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:05 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3319562881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:06.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:05 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2264644555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:56:06.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:05 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1230438843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:56:06.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.144+0000 7f66cb800640 1 -- 172.21.15.97:0/3540803346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66c4100080 msgr2=0x7f66c41004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:06.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.144+0000 7f66cb800640 1 --2- 172.21.15.97:0/3540803346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66c4100080 0x7f66c41004a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f66b4009900 tx=0x7f66b402bc30 comp rx=0 tx=0).stop 2024-03-11T19:56:06.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.144+0000 7f66cb800640 1 -- 172.21.15.97:0/3540803346 shutdown_connections 2024-03-11T19:56:06.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.144+0000 7f66cb800640 1 --2- 172.21.15.97:0/3540803346 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66c4101c30 0x7f66c41084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.144+0000 7f66cb800640 1 --2- 172.21.15.97:0/3540803346 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66c4101270 0x7f66c41016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.144+0000 7f66cb800640 1 --2- 172.21.15.97:0/3540803346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66c4100080 0x7f66c41004a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.144+0000 7f66cb800640 1 -- 172.21.15.97:0/3540803346 >> 172.21.15.97:0/3540803346 conn(0x7f66c40fb610 msgr2=0x7f66c40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:06.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.144+0000 7f66cb800640 1 -- 172.21.15.97:0/3540803346 shutdown_connections 2024-03-11T19:56:06.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.144+0000 7f66cb800640 1 -- 172.21.15.97:0/3540803346 wait complete. 2024-03-11T19:56:06.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.145+0000 7f66cb800640 1 Processor -- start 2024-03-11T19:56:06.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.145+0000 7f66cb800640 1 -- start start 2024-03-11T19:56:06.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.145+0000 7f66cb800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66c4100080 0x7f66c4195070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:06.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66cb800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66c4101270 0x7f66c41955b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:06.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66cb800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66c4101c30 0x7f66c419c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:06.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66cb800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c419cb60 con 0x7f66c4100080 2024-03-11T19:56:06.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66cb800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c419ccd0 con 0x7f66c4101270 2024-03-11T19:56:06.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66cb800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c419ce90 con 0x7f66c4101c30 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66c8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66c4100080 0x7f66c4195070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66c8e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66c4100080 0x7f66c4195070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56722/0 (socket says 172.21.15.97:56722) 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66c8e00640 1 -- 172.21.15.97:0/2886023379 learned_addr learned my addr 172.21.15.97:0/2886023379 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66c9800640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66c4101c30 0x7f66c419c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66bbe00640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66c4101270 0x7f66c41955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66c8e00640 1 -- 172.21.15.97:0/2886023379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66c4101c30 msgr2=0x7f66c419c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66c8e00640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66c4101c30 0x7f66c419c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66c8e00640 1 -- 172.21.15.97:0/2886023379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66c4101270 msgr2=0x7f66c41955b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66c8e00640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66c4101270 0x7f66c41955b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66c8e00640 1 -- 172.21.15.97:0/2886023379 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b4009560 con 0x7f66c4100080 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.146+0000 7f66bbe00640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66c4101270 0x7f66c41955b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.147+0000 7f66c8e00640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66c4100080 0x7f66c4195070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f66b4000c00 tx=0x7f66b4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.147+0000 7f66c2000640 1 -- 172.21.15.97:0/2886023379 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b40043d0 con 0x7f66c4100080 2024-03-11T19:56:06.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.147+0000 7f66cb800640 1 -- 172.21.15.97:0/2886023379 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c419d110 con 0x7f66c4100080 2024-03-11T19:56:06.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.147+0000 7f66c2000640 1 -- 172.21.15.97:0/2886023379 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66b40052d0 con 0x7f66c4100080 2024-03-11T19:56:06.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.147+0000 7f66cb800640 1 -- 172.21.15.97:0/2886023379 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c419d670 con 0x7f66c4100080 2024-03-11T19:56:06.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.147+0000 7f66c2000640 1 -- 172.21.15.97:0/2886023379 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b4003d40 con 0x7f66c4100080 2024-03-11T19:56:06.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.148+0000 7f66cb800640 1 -- 172.21.15.97:0/2886023379 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c41073f0 con 0x7f66c4100080 2024-03-11T19:56:06.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.150+0000 7f66c2000640 1 -- 172.21.15.97:0/2886023379 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f66b4005440 con 0x7f66c4100080 2024-03-11T19:56:06.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.152+0000 7f66c2000640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66a80a3c40 0x7f66a80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:06.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.152+0000 7f66c2000640 1 -- 172.21.15.97:0/2886023379 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f66b40efdb0 con 0x7f66c4100080 2024-03-11T19:56:06.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.152+0000 7f66bbe00640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66a80a3c40 0x7f66a80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:06.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.153+0000 7f66bbe00640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66a80a3c40 0x7f66a80a6100 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f66c4196580 tx=0x7f66b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:06.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.156+0000 7f66c2000640 1 -- 172.21.15.97:0/2886023379 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f66b40f4070 con 0x7f66c4100080 2024-03-11T19:56:06.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.352+0000 7f66cb800640 1 -- 172.21.15.97:0/2886023379 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f66c419d2a0 con 0x7f66c4100080 2024-03-11T19:56:06.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.353+0000 7f66c2000640 1 -- 172.21.15.97:0/2886023379 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f66c419d2a0 con 0x7f66c4100080 2024-03-11T19:56:06.354 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T19:56:06.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.355+0000 7f66cb800640 1 -- 172.21.15.97:0/2886023379 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66a80a3c40 msgr2=0x7f66a80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:06.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.355+0000 7f66cb800640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66a80a3c40 0x7f66a80a6100 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f66c4196580 tx=0x7f66b0005f90 comp rx=0 tx=0).stop 2024-03-11T19:56:06.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.355+0000 7f66cb800640 1 -- 172.21.15.97:0/2886023379 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66c4100080 msgr2=0x7f66c4195070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:06.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.355+0000 7f66cb800640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66c4100080 0x7f66c4195070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f66b4000c00 tx=0x7f66b4005e80 comp rx=0 tx=0).stop 2024-03-11T19:56:06.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.355+0000 7f66cb800640 1 -- 172.21.15.97:0/2886023379 shutdown_connections 2024-03-11T19:56:06.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.355+0000 7f66cb800640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66c4101c30 0x7f66c419c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.355+0000 7f66cb800640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66c4101270 0x7f66c41955b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.355+0000 7f66cb800640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66a80a3c40 0x7f66a80a6100 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.355+0000 7f66cb800640 1 --2- 172.21.15.97:0/2886023379 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66c4100080 0x7f66c4195070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.357 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.355+0000 7f66cb800640 1 -- 172.21.15.97:0/2886023379 >> 172.21.15.97:0/2886023379 conn(0x7f66c40fb610 msgr2=0x7f66c41045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:06.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.356+0000 7f66cb800640 1 -- 172.21.15.97:0/2886023379 shutdown_connections 2024-03-11T19:56:06.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.356+0000 7f66cb800640 1 -- 172.21.15.97:0/2886023379 wait complete. 2024-03-11T19:56:06.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-11T19:56:06.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.573+0000 7f4309200640 1 -- 172.21.15.97:0/2977747198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4304109350 msgr2=0x7f430410fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:06.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.573+0000 7f4309200640 1 --2- 172.21.15.97:0/2977747198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4304109350 0x7f430410fbe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f42fc009be0 tx=0x7f42fc02bef0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.573+0000 7f4309200640 1 -- 172.21.15.97:0/2977747198 shutdown_connections 2024-03-11T19:56:06.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.573+0000 7f4309200640 1 --2- 172.21.15.97:0/2977747198 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4304109350 0x7f430410fbe0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.573+0000 7f4309200640 1 --2- 172.21.15.97:0/2977747198 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4304108990 0x7f4304108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.573+0000 7f4309200640 1 --2- 172.21.15.97:0/2977747198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43041077a0 0x7f4304107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.573+0000 7f4309200640 1 -- 172.21.15.97:0/2977747198 >> 172.21.15.97:0/2977747198 conn(0x7f4304076740 msgr2=0x7f4304078b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:06.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.574+0000 7f4309200640 1 -- 172.21.15.97:0/2977747198 shutdown_connections 2024-03-11T19:56:06.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.574+0000 7f4309200640 1 -- 172.21.15.97:0/2977747198 wait complete. 2024-03-11T19:56:06.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.575+0000 7f4309200640 1 Processor -- start 2024-03-11T19:56:06.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.575+0000 7f4309200640 1 -- start start 2024-03-11T19:56:06.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.575+0000 7f4309200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43041077a0 0x7f430419d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:06.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.575+0000 7f4309200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4304108990 0x7f430419dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:06.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f4309200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4304109350 0x7f43041a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:06.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f4309200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43041a52b0 con 0x7f4304108990 2024-03-11T19:56:06.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f4309200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43041a5420 con 0x7f4304109350 2024-03-11T19:56:06.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f4309200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43041a55e0 con 0x7f43041077a0 2024-03-11T19:56:06.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f4301e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43041077a0 0x7f430419d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:06.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f42f9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4304108990 0x7f430419dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:06.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f4301e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43041077a0 0x7f430419d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:35378/0 (socket says 172.21.15.97:35378) 2024-03-11T19:56:06.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f4301e00640 1 -- 172.21.15.97:0/3063418657 learned_addr learned my addr 172.21.15.97:0/3063418657 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:06.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f42f9400640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4304108990 0x7f430419dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56748/0 (socket says 172.21.15.97:56748) 2024-03-11T19:56:06.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f4302800640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4304109350 0x7f43041a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:06.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f4301e00640 1 -- 172.21.15.97:0/3063418657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4304109350 msgr2=0x7f43041a4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:06.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f4301e00640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4304109350 0x7f43041a4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f4301e00640 1 -- 172.21.15.97:0/3063418657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4304108990 msgr2=0x7f430419dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:06.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f4301e00640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4304108990 0x7f430419dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.576+0000 7f4301e00640 1 -- 172.21.15.97:0/3063418657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42fc009840 con 0x7f43041077a0 2024-03-11T19:56:06.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.577+0000 7f4301e00640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43041077a0 0x7f430419d7c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f42f400e9c0 tx=0x7f42f400edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:06.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.581+0000 7f42fb400640 1 -- 172.21.15.97:0/3063418657 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f4009700 con 0x7f43041077a0 2024-03-11T19:56:06.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.581+0000 7f42fb400640 1 -- 172.21.15.97:0/3063418657 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42f400ccc0 con 0x7f43041077a0 2024-03-11T19:56:06.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.581+0000 7f4309200640 1 -- 172.21.15.97:0/3063418657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43041a58c0 con 0x7f43041077a0 2024-03-11T19:56:06.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.581+0000 7f42fb400640 1 -- 172.21.15.97:0/3063418657 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f4010430 con 0x7f43041077a0 2024-03-11T19:56:06.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.581+0000 7f4309200640 1 -- 172.21.15.97:0/3063418657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43041a5e20 con 0x7f43041077a0 2024-03-11T19:56:06.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.583+0000 7f4309200640 1 -- 172.21.15.97:0/3063418657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f430410eac0 con 0x7f43041077a0 2024-03-11T19:56:06.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.584+0000 7f42fb400640 1 -- 172.21.15.97:0/3063418657 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f42f4003f60 con 0x7f43041077a0 2024-03-11T19:56:06.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.586+0000 7f42fb400640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f42e80a1ab0 0x7f42e80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:06.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.586+0000 7f42f9400640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f42e80a1ab0 0x7f42e80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:06.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.587+0000 7f42fb400640 1 -- 172.21.15.97:0/3063418657 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f42f4014070 con 0x7f43041077a0 2024-03-11T19:56:06.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.587+0000 7f42f9400640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f42e80a1ab0 0x7f42e80a3f70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f430419ecd0 tx=0x7f42ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:06.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.592+0000 7f42fb400640 1 -- 172.21.15.97:0/3063418657 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f42f40969b0 con 0x7f43041077a0 2024-03-11T19:56:06.669 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:06 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3792212288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:56:06.669 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:06 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2886023379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:56:06.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.802+0000 7f4309200640 1 -- 172.21.15.97:0/3063418657 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4304113180 con 0x7f43041077a0 2024-03-11T19:56:06.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.803+0000 7f42fb400640 1 -- 172.21.15.97:0/3063418657 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+590 (secure 0 0 0) 0x7f42f409a070 con 0x7f43041077a0 2024-03-11T19:56:06.804 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/4129219534 2024-03-12T19:47:44.356323+0000 2024-03-11T19:56:06.804 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/3932792931 2024-03-12T19:47:44.356323+0000 2024-03-11T19:56:06.804 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6801/1708150324 2024-03-12T19:47:44.356323+0000 2024-03-11T19:56:06.804 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6800/1708150324 2024-03-12T19:47:44.356323+0000 2024-03-11T19:56:06.805 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6800/3216682632 2024-03-12T19:47:09.368819+0000 2024-03-11T19:56:06.805 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/3236990094 2024-03-12T19:47:09.368819+0000 2024-03-11T19:56:06.805 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/1301912133 2024-03-12T19:47:09.368819+0000 2024-03-11T19:56:06.805 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6801/3216682632 2024-03-12T19:47:09.368819+0000 2024-03-11T19:56:06.805 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/46019613 2024-03-12T19:47:44.356323+0000 2024-03-11T19:56:06.805 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/4194903751 2024-03-12T19:47:09.368819+0000 2024-03-11T19:56:06.805 INFO:teuthology.orchestra.run.smithi097.stderr:listed 10 entries 2024-03-11T19:56:06.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.805+0000 7f4309200640 1 -- 172.21.15.97:0/3063418657 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f42e80a1ab0 msgr2=0x7f42e80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:06.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.805+0000 7f4309200640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f42e80a1ab0 0x7f42e80a3f70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f430419ecd0 tx=0x7f42ec009260 comp rx=0 tx=0).stop 2024-03-11T19:56:06.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.805+0000 7f4309200640 1 -- 172.21.15.97:0/3063418657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43041077a0 msgr2=0x7f430419d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:06.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.805+0000 7f4309200640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43041077a0 0x7f430419d7c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f42f400e9c0 tx=0x7f42f400edc0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.805+0000 7f4309200640 1 -- 172.21.15.97:0/3063418657 shutdown_connections 2024-03-11T19:56:06.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.805+0000 7f4309200640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4304109350 0x7f43041a4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.805+0000 7f4309200640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f42e80a1ab0 0x7f42e80a3f70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.807 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.805+0000 7f4309200640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4304108990 0x7f430419dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.805+0000 7f4309200640 1 --2- 172.21.15.97:0/3063418657 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43041077a0 0x7f430419d7c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:06.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.805+0000 7f4309200640 1 -- 172.21.15.97:0/3063418657 >> 172.21.15.97:0/3063418657 conn(0x7f4304076740 msgr2=0x7f430410bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:06.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.805+0000 7f4309200640 1 -- 172.21.15.97:0/3063418657 shutdown_connections 2024-03-11T19:56:06.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:06.806+0000 7f4309200640 1 -- 172.21.15.97:0/3063418657 wait complete. 2024-03-11T19:56:06.824 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi097.front.sepia.ceph.com... 2024-03-11T19:56:06.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:56:06.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-11T19:56:06.911 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi097.stderr:ceph-fuse[75909]: fuse finished with error 0 and tester_r 0 2024-03-11T19:56:06.919 INFO:teuthology.orchestra.run:waiting for 300 2024-03-11T19:56:06.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:06 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3792212288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:56:06.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:06 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2886023379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:56:07.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:06 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3792212288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:56:07.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:06 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2886023379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:56:07.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:07 smithi097 ceph-mon[31898]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T19:56:07.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:07 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3063418657' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T19:56:07.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:07 smithi114 ceph-mon[36376]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T19:56:07.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:07 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3063418657' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T19:56:08.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:07 smithi161 ceph-mon[36332]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T19:56:08.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:07 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3063418657' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T19:56:09.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:09 smithi097 ceph-mon[31898]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-11T19:56:09.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:09 smithi114 ceph-mon[36376]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-11T19:56:10.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:09 smithi161 ceph-mon[36332]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-11T19:56:11.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:11 smithi097 ceph-mon[31898]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-11T19:56:11.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:11 smithi114 ceph-mon[36376]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-11T19:56:12.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:11 smithi161 ceph-mon[36332]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-11T19:56:13.021 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-11T19:56:13.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:56:13.022 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-11T19:56:13.102 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:56:13.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.322+0000 7f02c3e00640 1 -- 172.21.15.97:0/4207017206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02c41043c0 msgr2=0x7f02c4104840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:13.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.322+0000 7f02c3e00640 1 --2- 172.21.15.97:0/4207017206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02c41043c0 0x7f02c4104840 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f02ac009860 tx=0x7f02ac02bb70 comp rx=0 tx=0).stop 2024-03-11T19:56:13.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.322+0000 7f02c3e00640 1 -- 172.21.15.97:0/4207017206 shutdown_connections 2024-03-11T19:56:13.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.322+0000 7f02c3e00640 1 --2- 172.21.15.97:0/4207017206 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02c4104d80 0x7f02c410b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.322+0000 7f02c3e00640 1 --2- 172.21.15.97:0/4207017206 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02c41043c0 0x7f02c4104840 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.322+0000 7f02c3e00640 1 --2- 172.21.15.97:0/4207017206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c41031d0 0x7f02c41035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.322+0000 7f02c3e00640 1 -- 172.21.15.97:0/4207017206 >> 172.21.15.97:0/4207017206 conn(0x7f02c40fe760 msgr2=0x7f02c4100b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:13.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.322+0000 7f02c3e00640 1 -- 172.21.15.97:0/4207017206 shutdown_connections 2024-03-11T19:56:13.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.322+0000 7f02c3e00640 1 -- 172.21.15.97:0/4207017206 wait complete. 2024-03-11T19:56:13.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.323+0000 7f02c3e00640 1 Processor -- start 2024-03-11T19:56:13.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.323+0000 7f02c3e00640 1 -- start start 2024-03-11T19:56:13.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.323+0000 7f02c3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02c41031d0 0x7f02c4199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:13.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.323+0000 7f02c3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c41043c0 0x7f02c4199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:13.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.323+0000 7f02c3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02c4104d80 0x7f02c41a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:13.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.323+0000 7f02c3e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c41a0cf0 con 0x7f02c41031d0 2024-03-11T19:56:13.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.323+0000 7f02c3e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c41a0e60 con 0x7f02c41043c0 2024-03-11T19:56:13.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.324+0000 7f02c3e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c41a1020 con 0x7f02c4104d80 2024-03-11T19:56:13.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.324+0000 7f02c3400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02c4104d80 0x7f02c41a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:13.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.324+0000 7f02c3400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02c4104d80 0x7f02c41a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:35392/0 (socket says 172.21.15.97:35392) 2024-03-11T19:56:13.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.324+0000 7f02c3400640 1 -- 172.21.15.97:0/2475753398 learned_addr learned my addr 172.21.15.97:0/2475753398 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:13.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.324+0000 7f02c2a00640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02c41031d0 0x7f02c4199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:13.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.324+0000 7f02bbe00640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c41043c0 0x7f02c4199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:13.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.324+0000 7f02c3400640 1 -- 172.21.15.97:0/2475753398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c41043c0 msgr2=0x7f02c4199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:13.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.324+0000 7f02c3400640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c41043c0 0x7f02c4199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.324+0000 7f02c3400640 1 -- 172.21.15.97:0/2475753398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02c41031d0 msgr2=0x7f02c4199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:13.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.324+0000 7f02c3400640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02c41031d0 0x7f02c4199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.324+0000 7f02c3400640 1 -- 172.21.15.97:0/2475753398 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac009490 con 0x7f02c4104d80 2024-03-11T19:56:13.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.325+0000 7f02c3400640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02c4104d80 0x7f02c41a07b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00eb10 tx=0x7f02bc00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:13.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.325+0000 7f02bb400640 1 -- 172.21.15.97:0/2475753398 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc00ce20 con 0x7f02c4104d80 2024-03-11T19:56:13.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.325+0000 7f02c3e00640 1 -- 172.21.15.97:0/2475753398 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c41a1300 con 0x7f02c4104d80 2024-03-11T19:56:13.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.325+0000 7f02bb400640 1 -- 172.21.15.97:0/2475753398 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02bc005020 con 0x7f02c4104d80 2024-03-11T19:56:13.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.325+0000 7f02c3e00640 1 -- 172.21.15.97:0/2475753398 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c41a1830 con 0x7f02c4104d80 2024-03-11T19:56:13.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.325+0000 7f02bb400640 1 -- 172.21.15.97:0/2475753398 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc010670 con 0x7f02c4104d80 2024-03-11T19:56:13.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.326+0000 7f02c3e00640 1 -- 172.21.15.97:0/2475753398 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c410a490 con 0x7f02c4104d80 2024-03-11T19:56:13.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.328+0000 7f02bb400640 1 -- 172.21.15.97:0/2475753398 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f02bc005190 con 0x7f02c4104d80 2024-03-11T19:56:13.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.329+0000 7f02bb400640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f02a80a1ab0 0x7f02a80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:13.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.330+0000 7f02bb400640 1 -- 172.21.15.97:0/2475753398 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f02bc014070 con 0x7f02c4104d80 2024-03-11T19:56:13.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.330+0000 7f02c2a00640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f02a80a1ab0 0x7f02a80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:13.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.330+0000 7f02c2a00640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f02a80a1ab0 0x7f02a80a3f70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f02c4104220 tx=0x7f02b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:13.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.332+0000 7f02bb400640 1 -- 172.21.15.97:0/2475753398 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f02bc096c50 con 0x7f02c4104d80 2024-03-11T19:56:13.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.529+0000 7f02c3e00640 1 -- 172.21.15.97:0/2475753398 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02c410ebe0 con 0x7f02c4104d80 2024-03-11T19:56:13.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.532+0000 7f02bb400640 1 -- 172.21.15.97:0/2475753398 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24152 (secure 0 0 0) 0x7f02bc09a310 con 0x7f02c4104d80 2024-03-11T19:56:13.534 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:56:13.534 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":89,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:55:50.540865+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:56:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.534+0000 7f02c3e00640 1 -- 172.21.15.97:0/2475753398 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f02a80a1ab0 msgr2=0x7f02a80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.534+0000 7f02c3e00640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f02a80a1ab0 0x7f02a80a3f70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f02c4104220 tx=0x7f02b4009260 comp rx=0 tx=0).stop 2024-03-11T19:56:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.534+0000 7f02c3e00640 1 -- 172.21.15.97:0/2475753398 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02c4104d80 msgr2=0x7f02c41a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.534+0000 7f02c3e00640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02c4104d80 0x7f02c41a07b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00eb10 tx=0x7f02bc00ef10 comp rx=0 tx=0).stop 2024-03-11T19:56:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.535+0000 7f02c3e00640 1 -- 172.21.15.97:0/2475753398 shutdown_connections 2024-03-11T19:56:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.535+0000 7f02c3e00640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02c4104d80 0x7f02c41a07b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.535+0000 7f02c3e00640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c41043c0 0x7f02c4199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.535+0000 7f02c3e00640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f02a80a1ab0 0x7f02a80a3f70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.535+0000 7f02c3e00640 1 --2- 172.21.15.97:0/2475753398 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02c41031d0 0x7f02c4199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.535+0000 7f02c3e00640 1 -- 172.21.15.97:0/2475753398 >> 172.21.15.97:0/2475753398 conn(0x7f02c40fe760 msgr2=0x7f02c4107660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:13.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.535+0000 7f02c3e00640 1 -- 172.21.15.97:0/2475753398 shutdown_connections 2024-03-11T19:56:13.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.535+0000 7f02c3e00640 1 -- 172.21.15.97:0/2475753398 wait complete. 2024-03-11T19:56:13.557 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:56:13.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.769+0000 7fef4b200640 1 -- 172.21.15.97:0/2329113983 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4c1021d0 msgr2=0x7fef4c102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:13.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.769+0000 7fef4b200640 1 --2- 172.21.15.97:0/2329113983 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4c1021d0 0x7fef4c102650 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fef34009900 tx=0x7fef3402bc30 comp rx=0 tx=0).stop 2024-03-11T19:56:13.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.769+0000 7fef4b200640 1 -- 172.21.15.97:0/2329113983 shutdown_connections 2024-03-11T19:56:13.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.769+0000 7fef4b200640 1 --2- 172.21.15.97:0/2329113983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef4c102b90 0x7fef4c109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.769+0000 7fef4b200640 1 --2- 172.21.15.97:0/2329113983 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4c1021d0 0x7fef4c102650 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.769+0000 7fef4b200640 1 --2- 172.21.15.97:0/2329113983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4c100fe0 0x7fef4c101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.769+0000 7fef4b200640 1 -- 172.21.15.97:0/2329113983 >> 172.21.15.97:0/2329113983 conn(0x7fef4c0fc550 msgr2=0x7fef4c0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:13.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.769+0000 7fef4b200640 1 -- 172.21.15.97:0/2329113983 shutdown_connections 2024-03-11T19:56:13.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.769+0000 7fef4b200640 1 -- 172.21.15.97:0/2329113983 wait complete. 2024-03-11T19:56:13.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.770+0000 7fef4b200640 1 Processor -- start 2024-03-11T19:56:13.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.770+0000 7fef4b200640 1 -- start start 2024-03-11T19:56:13.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.770+0000 7fef4b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4c100fe0 0x7fef4c197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:13.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.770+0000 7fef4b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef4c1021d0 0x7fef4c197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:13.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.770+0000 7fef4b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4c102b90 0x7fef4c19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:13.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.770+0000 7fef4b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c19eb40 con 0x7fef4c102b90 2024-03-11T19:56:13.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.770+0000 7fef4b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c19ecb0 con 0x7fef4c100fe0 2024-03-11T19:56:13.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.770+0000 7fef4b200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c19ee70 con 0x7fef4c1021d0 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.771+0000 7fef4a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4c102b90 0x7fef4c19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.771+0000 7fef49e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4c100fe0 0x7fef4c197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.771+0000 7fef4a800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4c102b90 0x7fef4c19e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56782/0 (socket says 172.21.15.97:56782) 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.771+0000 7fef4a800640 1 -- 172.21.15.97:0/2190519681 learned_addr learned my addr 172.21.15.97:0/2190519681 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.771+0000 7fef43e00640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef4c1021d0 0x7fef4c197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.771+0000 7fef49e00640 1 -- 172.21.15.97:0/2190519681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef4c1021d0 msgr2=0x7fef4c197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.771+0000 7fef49e00640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef4c1021d0 0x7fef4c197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.771+0000 7fef49e00640 1 -- 172.21.15.97:0/2190519681 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4c102b90 msgr2=0x7fef4c19e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.771+0000 7fef49e00640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4c102b90 0x7fef4c19e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.771+0000 7fef49e00640 1 -- 172.21.15.97:0/2190519681 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef34009560 con 0x7fef4c100fe0 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.771+0000 7fef4a800640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4c102b90 0x7fef4c19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.771+0000 7fef49e00640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4c100fe0 0x7fef4c197050 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fef3c00e9c0 tx=0x7fef3c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.772+0000 7fef42a00640 1 -- 172.21.15.97:0/2190519681 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c00ccc0 con 0x7fef4c100fe0 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.772+0000 7fef42a00640 1 -- 172.21.15.97:0/2190519681 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef3c00ce20 con 0x7fef4c100fe0 2024-03-11T19:56:13.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.772+0000 7fef42a00640 1 -- 172.21.15.97:0/2190519681 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c010430 con 0x7fef4c100fe0 2024-03-11T19:56:13.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.772+0000 7fef4b200640 1 -- 172.21.15.97:0/2190519681 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4c19f150 con 0x7fef4c100fe0 2024-03-11T19:56:13.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.772+0000 7fef4b200640 1 -- 172.21.15.97:0/2190519681 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef4c19f680 con 0x7fef4c100fe0 2024-03-11T19:56:13.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.773+0000 7fef4b200640 1 -- 172.21.15.97:0/2190519681 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4c108300 con 0x7fef4c100fe0 2024-03-11T19:56:13.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.775+0000 7fef42a00640 1 -- 172.21.15.97:0/2190519681 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fef3c003f60 con 0x7fef4c100fe0 2024-03-11T19:56:13.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.776+0000 7fef42a00640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fef2c0a1ab0 0x7fef2c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:13.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.776+0000 7fef42a00640 1 -- 172.21.15.97:0/2190519681 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fef3c014070 con 0x7fef4c100fe0 2024-03-11T19:56:13.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.776+0000 7fef43e00640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fef2c0a1ab0 0x7fef2c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:13.778 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.777+0000 7fef43e00640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fef2c0a1ab0 0x7fef2c0a3f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fef34009230 tx=0x7fef34005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:13.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.780+0000 7fef42a00640 1 -- 172.21.15.97:0/2190519681 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fef3c096a10 con 0x7fef4c100fe0 2024-03-11T19:56:13.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:13 smithi097 ceph-mon[31898]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-11T19:56:13.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:13 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2475753398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:13.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.977+0000 7fef4b200640 1 -- 172.21.15.97:0/2190519681 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef4c10c9e0 con 0x7fef4c100fe0 2024-03-11T19:56:13.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.979+0000 7fef42a00640 1 -- 172.21.15.97:0/2190519681 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24152 (secure 0 0 0) 0x7fef3c09a0d0 con 0x7fef4c100fe0 2024-03-11T19:56:13.982 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:56:13.982 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":89,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:55:50.540865+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:56:13.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.982+0000 7fef4b200640 1 -- 172.21.15.97:0/2190519681 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fef2c0a1ab0 msgr2=0x7fef2c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:13.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.982+0000 7fef4b200640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fef2c0a1ab0 0x7fef2c0a3f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fef34009230 tx=0x7fef34005fd0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.982+0000 7fef4b200640 1 -- 172.21.15.97:0/2190519681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4c100fe0 msgr2=0x7fef4c197050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:13.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.982+0000 7fef4b200640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4c100fe0 0x7fef4c197050 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fef3c00e9c0 tx=0x7fef3c00ee80 comp rx=0 tx=0).stop 2024-03-11T19:56:13.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.982+0000 7fef4b200640 1 -- 172.21.15.97:0/2190519681 shutdown_connections 2024-03-11T19:56:13.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.982+0000 7fef4b200640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fef2c0a1ab0 0x7fef2c0a3f70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.982+0000 7fef4b200640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fef4c102b90 0x7fef4c19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.982+0000 7fef4b200640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef4c1021d0 0x7fef4c197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.982+0000 7fef4b200640 1 --2- 172.21.15.97:0/2190519681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef4c100fe0 0x7fef4c197050 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:13.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.982+0000 7fef4b200640 1 -- 172.21.15.97:0/2190519681 >> 172.21.15.97:0/2190519681 conn(0x7fef4c0fc550 msgr2=0x7fef4c1054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:13.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.982+0000 7fef4b200640 1 -- 172.21.15.97:0/2190519681 shutdown_connections 2024-03-11T19:56:13.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:13.982+0000 7fef4b200640 1 -- 172.21.15.97:0/2190519681 wait complete. 2024-03-11T19:56:13.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:13 smithi114 ceph-mon[36376]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-11T19:56:13.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:13 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2475753398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:14.004 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-11T19:56:14.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:13 smithi161 ceph-mon[36332]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-11T19:56:14.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:13 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2475753398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:14.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.223+0000 7f3fa5800640 1 -- 172.21.15.97:0/595675550 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fa0104f50 msgr2=0x7f3fa010b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:14.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.223+0000 7f3fa5800640 1 --2- 172.21.15.97:0/595675550 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fa0104f50 0x7f3fa010b7e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009ca0 tx=0x7f3f9802c100 comp rx=0 tx=0).stop 2024-03-11T19:56:14.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.223+0000 7f3fa5800640 1 -- 172.21.15.97:0/595675550 shutdown_connections 2024-03-11T19:56:14.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.223+0000 7f3fa5800640 1 --2- 172.21.15.97:0/595675550 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fa0104f50 0x7f3fa010b7e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.223+0000 7f3fa5800640 1 --2- 172.21.15.97:0/595675550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fa0104590 0x7f3fa0104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.223+0000 7f3fa5800640 1 --2- 172.21.15.97:0/595675550 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3fa01033a0 0x7f3fa01037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.223+0000 7f3fa5800640 1 -- 172.21.15.97:0/595675550 >> 172.21.15.97:0/595675550 conn(0x7f3fa00fe930 msgr2=0x7f3fa0100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:14.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.223+0000 7f3fa5800640 1 -- 172.21.15.97:0/595675550 shutdown_connections 2024-03-11T19:56:14.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.223+0000 7f3fa5800640 1 -- 172.21.15.97:0/595675550 wait complete. 2024-03-11T19:56:14.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.224+0000 7f3fa5800640 1 Processor -- start 2024-03-11T19:56:14.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.224+0000 7f3fa5800640 1 -- start start 2024-03-11T19:56:14.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3fa5800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fa01033a0 0x7f3fa0199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:14.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3fa5800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fa0104590 0x7f3fa0199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:14.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3fa5800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3fa0104f50 0x7f3fa01a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:14.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3fa5800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa01a0f20 con 0x7f3fa01033a0 2024-03-11T19:56:14.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3fa5800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa01a1090 con 0x7f3fa0104590 2024-03-11T19:56:14.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3fa5800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa01a1250 con 0x7f3fa0104f50 2024-03-11T19:56:14.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3f9ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fa01033a0 0x7f3fa0199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:14.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3f9ea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fa01033a0 0x7f3fa0199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59652/0 (socket says 172.21.15.97:59652) 2024-03-11T19:56:14.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3f9ea00640 1 -- 172.21.15.97:0/2798126348 learned_addr learned my addr 172.21.15.97:0/2798126348 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:14.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3f9e000640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fa0104590 0x7f3fa0199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:14.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3f9f400640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3fa0104f50 0x7f3fa01a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:14.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3f9ea00640 1 -- 172.21.15.97:0/2798126348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3fa0104f50 msgr2=0x7f3fa01a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:14.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3f9ea00640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3fa0104f50 0x7f3fa01a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3f9ea00640 1 -- 172.21.15.97:0/2798126348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fa0104590 msgr2=0x7f3fa0199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:14.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.225+0000 7f3f9ea00640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fa0104590 0x7f3fa0199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.226+0000 7f3f9ea00640 1 -- 172.21.15.97:0/2798126348 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f98009900 con 0x7f3fa01033a0 2024-03-11T19:56:14.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.226+0000 7f3f9e000640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fa0104590 0x7f3fa0199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:14.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.226+0000 7f3f9ea00640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fa01033a0 0x7f3fa0199430 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3f8800e9c0 tx=0x7f3f8800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:14.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.226+0000 7f3f97400640 1 -- 172.21.15.97:0/2798126348 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f880096e0 con 0x7f3fa01033a0 2024-03-11T19:56:14.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.226+0000 7f3f97400640 1 -- 172.21.15.97:0/2798126348 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f8800cd80 con 0x7f3fa01033a0 2024-03-11T19:56:14.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.226+0000 7f3fa5800640 1 -- 172.21.15.97:0/2798126348 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa01a1530 con 0x7f3fa01033a0 2024-03-11T19:56:14.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.226+0000 7f3f97400640 1 -- 172.21.15.97:0/2798126348 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88010430 con 0x7f3fa01033a0 2024-03-11T19:56:14.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.226+0000 7f3fa5800640 1 -- 172.21.15.97:0/2798126348 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa01a19f0 con 0x7f3fa01033a0 2024-03-11T19:56:14.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.228+0000 7f3fa5800640 1 -- 172.21.15.97:0/2798126348 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa010a660 con 0x7f3fa01033a0 2024-03-11T19:56:14.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.229+0000 7f3f97400640 1 -- 172.21.15.97:0/2798126348 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3f88003680 con 0x7f3fa01033a0 2024-03-11T19:56:14.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.231+0000 7f3f97400640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3f840a1a30 0x7f3f840a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:14.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.231+0000 7f3f97400640 1 -- 172.21.15.97:0/2798126348 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3f88014070 con 0x7f3fa01033a0 2024-03-11T19:56:14.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.231+0000 7f3f9e000640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3f840a1a30 0x7f3f840a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:14.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.232+0000 7f3f9e000640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3f840a1a30 0x7f3f840a3ef0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3fa019a940 tx=0x7f3f90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:14.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.235+0000 7f3f97400640 1 -- 172.21.15.97:0/2798126348 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3f880969a0 con 0x7f3fa01033a0 2024-03-11T19:56:14.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.470+0000 7f3fa5800640 1 -- 172.21.15.97:0/2798126348 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3fa002ce40 con 0x7f3fa01033a0 2024-03-11T19:56:14.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.601+0000 7f3f97400640 1 -- 172.21.15.97:0/2798126348 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3f8809a060 con 0x7f3fa01033a0 2024-03-11T19:56:14.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.605+0000 7f3fa5800640 1 -- 172.21.15.97:0/2798126348 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3f840a1a30 msgr2=0x7f3f840a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:14.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.605+0000 7f3fa5800640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3f840a1a30 0x7f3f840a3ef0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3fa019a940 tx=0x7f3f90005f90 comp rx=0 tx=0).stop 2024-03-11T19:56:14.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.605+0000 7f3fa5800640 1 -- 172.21.15.97:0/2798126348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fa01033a0 msgr2=0x7f3fa0199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:14.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.605+0000 7f3fa5800640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fa01033a0 0x7f3fa0199430 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3f8800e9c0 tx=0x7f3f8800ee80 comp rx=0 tx=0).stop 2024-03-11T19:56:14.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.605+0000 7f3fa5800640 1 -- 172.21.15.97:0/2798126348 shutdown_connections 2024-03-11T19:56:14.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.605+0000 7f3fa5800640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3fa0104f50 0x7f3fa01a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.605+0000 7f3fa5800640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fa0104590 0x7f3fa0199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.605+0000 7f3fa5800640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3f840a1a30 0x7f3f840a3ef0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.605+0000 7f3fa5800640 1 --2- 172.21.15.97:0/2798126348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3fa01033a0 0x7f3fa0199430 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.605+0000 7f3fa5800640 1 -- 172.21.15.97:0/2798126348 >> 172.21.15.97:0/2798126348 conn(0x7f3fa00fe930 msgr2=0x7f3fa0107830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:14.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.605+0000 7f3fa5800640 1 -- 172.21.15.97:0/2798126348 shutdown_connections 2024-03-11T19:56:14.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.606+0000 7f3fa5800640 1 -- 172.21.15.97:0/2798126348 wait complete. 2024-03-11T19:56:14.628 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-11T19:56:14.628 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-11T19:56:14.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.837+0000 7f3b48a00640 1 -- 172.21.15.97:0/3204185281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b441077a0 msgr2=0x7f3b44107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:14.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.837+0000 7f3b48a00640 1 --2- 172.21.15.97:0/3204185281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b441077a0 0x7f3b44107bc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009900 tx=0x7f3b3402bc30 comp rx=0 tx=0).stop 2024-03-11T19:56:14.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.838+0000 7f3b48a00640 1 -- 172.21.15.97:0/3204185281 shutdown_connections 2024-03-11T19:56:14.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.838+0000 7f3b48a00640 1 --2- 172.21.15.97:0/3204185281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b44109350 0x7f3b4410fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.838+0000 7f3b48a00640 1 --2- 172.21.15.97:0/3204185281 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b44108990 0x7f3b44108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.838+0000 7f3b48a00640 1 --2- 172.21.15.97:0/3204185281 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b441077a0 0x7f3b44107bc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.838+0000 7f3b48a00640 1 -- 172.21.15.97:0/3204185281 >> 172.21.15.97:0/3204185281 conn(0x7f3b44076740 msgr2=0x7f3b44078b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:14.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.838+0000 7f3b48a00640 1 -- 172.21.15.97:0/3204185281 shutdown_connections 2024-03-11T19:56:14.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.838+0000 7f3b48a00640 1 -- 172.21.15.97:0/3204185281 wait complete. 2024-03-11T19:56:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.839+0000 7f3b48a00640 1 Processor -- start 2024-03-11T19:56:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.839+0000 7f3b48a00640 1 -- start start 2024-03-11T19:56:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.839+0000 7f3b48a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b441077a0 0x7f3b4419d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.839+0000 7f3b48a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b44108990 0x7f3b4419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.839+0000 7f3b48a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b44109350 0x7f3b441a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.839+0000 7f3b48a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b441a53c0 con 0x7f3b44109350 2024-03-11T19:56:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.839+0000 7f3b48a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b441a5530 con 0x7f3b44108990 2024-03-11T19:56:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.839+0000 7f3b48a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b441a56f0 con 0x7f3b441077a0 2024-03-11T19:56:14.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.839+0000 7f3b42800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b44109350 0x7f3b441a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:14.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.839+0000 7f3b42800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b44109350 0x7f3b441a4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59682/0 (socket says 172.21.15.97:59682) 2024-03-11T19:56:14.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.839+0000 7f3b41e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b441077a0 0x7f3b4419d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:14.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.839+0000 7f3b42800640 1 -- 172.21.15.97:0/2767050389 learned_addr learned my addr 172.21.15.97:0/2767050389 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:14.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.839+0000 7f3b41400640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b44108990 0x7f3b4419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:14.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.840+0000 7f3b42800640 1 -- 172.21.15.97:0/2767050389 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b441077a0 msgr2=0x7f3b4419d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:14.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.840+0000 7f3b42800640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b441077a0 0x7f3b4419d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.840+0000 7f3b42800640 1 -- 172.21.15.97:0/2767050389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b44108990 msgr2=0x7f3b4419de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:14.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.840+0000 7f3b42800640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b44108990 0x7f3b4419de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:14.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.840+0000 7f3b42800640 1 -- 172.21.15.97:0/2767050389 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b34009560 con 0x7f3b44109350 2024-03-11T19:56:14.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.840+0000 7f3b41e00640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b441077a0 0x7f3b4419d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:14.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.840+0000 7f3b42800640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b44109350 0x7f3b441a4e80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c009fd0 tx=0x7f3b3c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:14.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.840+0000 7f3b3aa00640 1 -- 172.21.15.97:0/2767050389 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c00bea0 con 0x7f3b44109350 2024-03-11T19:56:14.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.840+0000 7f3b3aa00640 1 -- 172.21.15.97:0/2767050389 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b3c009940 con 0x7f3b44109350 2024-03-11T19:56:14.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.840+0000 7f3b48a00640 1 -- 172.21.15.97:0/2767050389 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b441a59d0 con 0x7f3b44109350 2024-03-11T19:56:14.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.840+0000 7f3b3aa00640 1 -- 172.21.15.97:0/2767050389 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c0106c0 con 0x7f3b44109350 2024-03-11T19:56:14.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.841+0000 7f3b48a00640 1 -- 172.21.15.97:0/2767050389 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b441a5e90 con 0x7f3b44109350 2024-03-11T19:56:14.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.842+0000 7f3b48a00640 1 -- 172.21.15.97:0/2767050389 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b4410eb90 con 0x7f3b44109350 2024-03-11T19:56:14.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.844+0000 7f3b3aa00640 1 -- 172.21.15.97:0/2767050389 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3b3c004240 con 0x7f3b44109350 2024-03-11T19:56:14.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.845+0000 7f3b3aa00640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3b280a1a30 0x7f3b280a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:14.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.845+0000 7f3b3aa00640 1 -- 172.21.15.97:0/2767050389 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3b3c0cd290 con 0x7f3b44109350 2024-03-11T19:56:14.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.846+0000 7f3b41e00640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3b280a1a30 0x7f3b280a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:14.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.846+0000 7f3b41e00640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3b280a1a30 0x7f3b280a3ef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3b34005ab0 tx=0x7f3b34005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:14.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:14.848+0000 7f3b3aa00640 1 -- 172.21.15.97:0/2767050389 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3b3c096310 con 0x7f3b44109350 2024-03-11T19:56:14.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:14 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2190519681' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:14.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:14 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2798126348' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-11T19:56:14.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:14 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2190519681' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:14.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:14 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2798126348' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-11T19:56:15.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:14 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2190519681' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:15.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:14 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2798126348' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-11T19:56:15.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.082+0000 7f3b48a00640 1 -- 172.21.15.97:0/2767050389 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f3b44113350 con 0x7f3b44109350 2024-03-11T19:56:15.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.622+0000 7f3b3aa00640 1 -- 172.21.15.97:0/2767050389 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3b3c0999d0 con 0x7f3b44109350 2024-03-11T19:56:15.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.625+0000 7f3b48a00640 1 -- 172.21.15.97:0/2767050389 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3b280a1a30 msgr2=0x7f3b280a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:15.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.625+0000 7f3b48a00640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3b280a1a30 0x7f3b280a3ef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3b34005ab0 tx=0x7f3b34005fd0 comp rx=0 tx=0).stop 2024-03-11T19:56:15.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.625+0000 7f3b48a00640 1 -- 172.21.15.97:0/2767050389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b44109350 msgr2=0x7f3b441a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:15.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.625+0000 7f3b48a00640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b44109350 0x7f3b441a4e80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c009fd0 tx=0x7f3b3c00b260 comp rx=0 tx=0).stop 2024-03-11T19:56:15.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.625+0000 7f3b48a00640 1 -- 172.21.15.97:0/2767050389 shutdown_connections 2024-03-11T19:56:15.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.625+0000 7f3b48a00640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3b280a1a30 0x7f3b280a3ef0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:15.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.625+0000 7f3b48a00640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b44109350 0x7f3b441a4e80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:15.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.625+0000 7f3b48a00640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b44108990 0x7f3b4419de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:15.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.625+0000 7f3b48a00640 1 --2- 172.21.15.97:0/2767050389 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b441077a0 0x7f3b4419d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:15.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.626+0000 7f3b48a00640 1 -- 172.21.15.97:0/2767050389 >> 172.21.15.97:0/2767050389 conn(0x7f3b44076740 msgr2=0x7f3b4410bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:15.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.626+0000 7f3b48a00640 1 -- 172.21.15.97:0/2767050389 shutdown_connections 2024-03-11T19:56:15.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.626+0000 7f3b48a00640 1 -- 172.21.15.97:0/2767050389 wait complete. 2024-03-11T19:56:15.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-11T19:56:15.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.839+0000 7ff6fde00640 1 -- 172.21.15.97:0/2404998557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f81033c0 msgr2=0x7ff6f81037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:15.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.839+0000 7ff6fde00640 1 --2- 172.21.15.97:0/2404998557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f81033c0 0x7ff6f81037e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009900 tx=0x7ff6e802bc30 comp rx=0 tx=0).stop 2024-03-11T19:56:15.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.839+0000 7ff6fde00640 1 -- 172.21.15.97:0/2404998557 shutdown_connections 2024-03-11T19:56:15.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.839+0000 7ff6fde00640 1 --2- 172.21.15.97:0/2404998557 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6f8104f70 0x7ff6f810b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:15.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.839+0000 7ff6fde00640 1 --2- 172.21.15.97:0/2404998557 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f81045b0 0x7ff6f8104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:15.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.839+0000 7ff6fde00640 1 --2- 172.21.15.97:0/2404998557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f81033c0 0x7ff6f81037e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:15.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.839+0000 7ff6fde00640 1 -- 172.21.15.97:0/2404998557 >> 172.21.15.97:0/2404998557 conn(0x7ff6f80fe930 msgr2=0x7ff6f8100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:15.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.839+0000 7ff6fde00640 1 -- 172.21.15.97:0/2404998557 shutdown_connections 2024-03-11T19:56:15.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.840+0000 7ff6fde00640 1 -- 172.21.15.97:0/2404998557 wait complete. 2024-03-11T19:56:15.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.841+0000 7ff6fde00640 1 Processor -- start 2024-03-11T19:56:15.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.841+0000 7ff6fde00640 1 -- start start 2024-03-11T19:56:15.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.841+0000 7ff6fde00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f81033c0 0x7ff6f8199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:15.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.841+0000 7ff6fde00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6f81045b0 0x7ff6f81999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:15.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.841+0000 7ff6fde00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f8104f70 0x7ff6f81a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.841+0000 7ff6fde00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81a0f60 con 0x7ff6f81045b0 2024-03-11T19:56:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.841+0000 7ff6fde00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81a10d0 con 0x7ff6f81033c0 2024-03-11T19:56:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.841+0000 7ff6fde00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81a1290 con 0x7ff6f8104f70 2024-03-11T19:56:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.841+0000 7ff6f6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6f81045b0 0x7ff6f81999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.841+0000 7ff6f6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f81033c0 0x7ff6f8199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.842+0000 7ff6f6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6f81045b0 0x7ff6f81999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59700/0 (socket says 172.21.15.97:59700) 2024-03-11T19:56:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.842+0000 7ff6f6000640 1 -- 172.21.15.97:0/4287536929 learned_addr learned my addr 172.21.15.97:0/4287536929 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.842+0000 7ff6f7400640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f8104f70 0x7ff6f81a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.842+0000 7ff6f6a00640 1 -- 172.21.15.97:0/4287536929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f8104f70 msgr2=0x7ff6f81a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.842+0000 7ff6f6a00640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f8104f70 0x7ff6f81a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.842+0000 7ff6f6a00640 1 -- 172.21.15.97:0/4287536929 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6f81045b0 msgr2=0x7ff6f81999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.842+0000 7ff6f6a00640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6f81045b0 0x7ff6f81999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:15.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.842+0000 7ff6f6a00640 1 -- 172.21.15.97:0/4287536929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e8009560 con 0x7ff6f81033c0 2024-03-11T19:56:15.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.842+0000 7ff6f6000640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6f81045b0 0x7ff6f81999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:15.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.842+0000 7ff6f6a00640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f81033c0 0x7ff6f8199470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8036040 tx=0x7ff6e8005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:15.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.843+0000 7ff6fca00640 1 -- 172.21.15.97:0/4287536929 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e80043d0 con 0x7ff6f81033c0 2024-03-11T19:56:15.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.843+0000 7ff6fca00640 1 -- 172.21.15.97:0/4287536929 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6e80052d0 con 0x7ff6f81033c0 2024-03-11T19:56:15.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.843+0000 7ff6fca00640 1 -- 172.21.15.97:0/4287536929 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8003c60 con 0x7ff6f81033c0 2024-03-11T19:56:15.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.843+0000 7ff6fde00640 1 -- 172.21.15.97:0/4287536929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f81a1510 con 0x7ff6f81033c0 2024-03-11T19:56:15.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.843+0000 7ff6fde00640 1 -- 172.21.15.97:0/4287536929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f81a1a40 con 0x7ff6f81033c0 2024-03-11T19:56:15.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.844+0000 7ff6fde00640 1 -- 172.21.15.97:0/4287536929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f810a6e0 con 0x7ff6f81033c0 2024-03-11T19:56:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.846+0000 7ff6fca00640 1 -- 172.21.15.97:0/4287536929 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff6e8004df0 con 0x7ff6f81033c0 2024-03-11T19:56:15.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.848+0000 7ff6fca00640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff6d80a1ab0 0x7ff6d80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:15.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.848+0000 7ff6fca00640 1 -- 172.21.15.97:0/4287536929 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff6e80f0420 con 0x7ff6f81033c0 2024-03-11T19:56:15.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.848+0000 7ff6f6000640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff6d80a1ab0 0x7ff6d80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:15.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.849+0000 7ff6f6000640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff6d80a1ab0 0x7ff6d80a3f70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff6f819a980 tx=0x7ff6e0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:15.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:15.850+0000 7ff6fca00640 1 -- 172.21.15.97:0/4287536929 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff6e80b9390 con 0x7ff6f81033c0 2024-03-11T19:56:15.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:15 smithi097 ceph-mon[31898]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-11T19:56:15.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:15 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2798126348' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-11T19:56:15.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:15 smithi097 ceph-mon[31898]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:56:15.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:15 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2767050389' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-11T19:56:15.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:15 smithi114 ceph-mon[36376]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-11T19:56:15.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:15 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2798126348' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-11T19:56:15.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:15 smithi114 ceph-mon[36376]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:56:15.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:15 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2767050389' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-11T19:56:16.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:15 smithi161 ceph-mon[36332]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-11T19:56:16.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:15 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2798126348' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-11T19:56:16.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:15 smithi161 ceph-mon[36332]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:56:16.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:15 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2767050389' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-11T19:56:16.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.083+0000 7ff6fde00640 1 -- 172.21.15.97:0/4287536929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7ff6f810ee20 con 0x7ff6f81033c0 2024-03-11T19:56:16.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.636+0000 7ff6fca00640 1 -- 172.21.15.97:0/4287536929 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7ff6e8043020 con 0x7ff6f81033c0 2024-03-11T19:56:16.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.639+0000 7ff6fde00640 1 -- 172.21.15.97:0/4287536929 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff6d80a1ab0 msgr2=0x7ff6d80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:16.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.639+0000 7ff6fde00640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff6d80a1ab0 0x7ff6d80a3f70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff6f819a980 tx=0x7ff6e0009500 comp rx=0 tx=0).stop 2024-03-11T19:56:16.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.639+0000 7ff6fde00640 1 -- 172.21.15.97:0/4287536929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f81033c0 msgr2=0x7ff6f8199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:16.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.639+0000 7ff6fde00640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f81033c0 0x7ff6f8199470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8036040 tx=0x7ff6e8005e80 comp rx=0 tx=0).stop 2024-03-11T19:56:16.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.639+0000 7ff6fde00640 1 -- 172.21.15.97:0/4287536929 shutdown_connections 2024-03-11T19:56:16.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.639+0000 7ff6fde00640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff6f8104f70 0x7ff6f81a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:16.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.639+0000 7ff6fde00640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff6d80a1ab0 0x7ff6d80a3f70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:16.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.639+0000 7ff6fde00640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff6f81045b0 0x7ff6f81999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:16.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.639+0000 7ff6fde00640 1 --2- 172.21.15.97:0/4287536929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6f81033c0 0x7ff6f8199470 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:16.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.639+0000 7ff6fde00640 1 -- 172.21.15.97:0/4287536929 >> 172.21.15.97:0/4287536929 conn(0x7ff6f80fe930 msgr2=0x7ff6f81078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:16.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.640+0000 7ff6fde00640 1 -- 172.21.15.97:0/4287536929 shutdown_connections 2024-03-11T19:56:16.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.640+0000 7ff6fde00640 1 -- 172.21.15.97:0/4287536929 wait complete. 2024-03-11T19:56:16.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-11T19:56:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.861+0000 7f3662800640 1 -- 172.21.15.97:0/1860673505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c104420 msgr2=0x7f365c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:16.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.861+0000 7f3662800640 1 --2- 172.21.15.97:0/1860673505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c104420 0x7f365c106810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3648009900 tx=0x7f364802bc40 comp rx=0 tx=0).stop 2024-03-11T19:56:16.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.861+0000 7f3662800640 1 -- 172.21.15.97:0/1860673505 shutdown_connections 2024-03-11T19:56:16.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.861+0000 7f3662800640 1 --2- 172.21.15.97:0/1860673505 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c106d50 0x7f365c1091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:16.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.861+0000 7f3662800640 1 --2- 172.21.15.97:0/1860673505 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c104420 0x7f365c106810 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:16.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.861+0000 7f3662800640 1 --2- 172.21.15.97:0/1860673505 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c101af0 0x7f365c103ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:16.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.861+0000 7f3662800640 1 -- 172.21.15.97:0/1860673505 >> 172.21.15.97:0/1860673505 conn(0x7f365c0fb500 msgr2=0x7f365c0fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:16.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.861+0000 7f3662800640 1 -- 172.21.15.97:0/1860673505 shutdown_connections 2024-03-11T19:56:16.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.861+0000 7f3662800640 1 -- 172.21.15.97:0/1860673505 wait complete. 2024-03-11T19:56:16.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.862+0000 7f3662800640 1 Processor -- start 2024-03-11T19:56:16.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.862+0000 7f3662800640 1 -- start start 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.863+0000 7f3662800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c101af0 0x7f365c1993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.863+0000 7f3662800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c104420 0x7f365c1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.863+0000 7f3662800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c106d50 0x7f365c1a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.863+0000 7f3662800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c1a0e30 con 0x7f365c104420 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.863+0000 7f3662800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c1a0fa0 con 0x7f365c101af0 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.863+0000 7f3662800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c1a1160 con 0x7f365c106d50 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.863+0000 7f3660a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c104420 0x7f365c1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.863+0000 7f3660a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c104420 0x7f365c1998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59708/0 (socket says 172.21.15.97:59708) 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.863+0000 7f3660a00640 1 -- 172.21.15.97:0/3558800593 learned_addr learned my addr 172.21.15.97:0/3558800593 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.863+0000 7f3661e00640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c106d50 0x7f365c1a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.863+0000 7f3661400640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c101af0 0x7f365c1993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.864+0000 7f3660a00640 1 -- 172.21.15.97:0/3558800593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c106d50 msgr2=0x7f365c1a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.864+0000 7f3660a00640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c106d50 0x7f365c1a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:16.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.864+0000 7f3660a00640 1 -- 172.21.15.97:0/3558800593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c101af0 msgr2=0x7f365c1993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:16.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.864+0000 7f3660a00640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c101af0 0x7f365c1993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:16.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.864+0000 7f3660a00640 1 -- 172.21.15.97:0/3558800593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3654009490 con 0x7f365c104420 2024-03-11T19:56:16.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.864+0000 7f3661e00640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c106d50 0x7f365c1a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:16.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.864+0000 7f3661400640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c101af0 0x7f365c1993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:16.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.864+0000 7f3660a00640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c104420 0x7f365c1998f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3648000c00 tx=0x7f3648005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:16.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.864+0000 7f365a000640 1 -- 172.21.15.97:0/3558800593 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36480043d0 con 0x7f365c104420 2024-03-11T19:56:16.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.864+0000 7f365a000640 1 -- 172.21.15.97:0/3558800593 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36480052d0 con 0x7f365c104420 2024-03-11T19:56:16.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.864+0000 7f365a000640 1 -- 172.21.15.97:0/3558800593 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3648003ca0 con 0x7f365c104420 2024-03-11T19:56:16.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.864+0000 7f3662800640 1 -- 172.21.15.97:0/3558800593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3648009560 con 0x7f365c104420 2024-03-11T19:56:16.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.864+0000 7f3662800640 1 -- 172.21.15.97:0/3558800593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365c1a1710 con 0x7f365c104420 2024-03-11T19:56:16.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.866+0000 7f3662800640 1 -- 172.21.15.97:0/3558800593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365c1088a0 con 0x7f365c104420 2024-03-11T19:56:16.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.867+0000 7f365a000640 1 -- 172.21.15.97:0/3558800593 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3648005440 con 0x7f365c104420 2024-03-11T19:56:16.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.869+0000 7f365a000640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f36440a1a30 0x7f36440a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:16.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.869+0000 7f365a000640 1 -- 172.21.15.97:0/3558800593 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f36480efdf0 con 0x7f365c104420 2024-03-11T19:56:16.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.869+0000 7f3661400640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f36440a1a30 0x7f36440a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:16.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.870+0000 7f3661400640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f36440a1a30 0x7f36440a3ef0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3650005fd0 tx=0x7f3650009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:16.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:16.873+0000 7f365a000640 1 -- 172.21.15.97:0/3558800593 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f36480b8d60 con 0x7f365c104420 2024-03-11T19:56:16.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:16 smithi097 ceph-mon[31898]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T19:56:16.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:16 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2767050389' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-11T19:56:16.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:16 smithi097 ceph-mon[31898]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-11T19:56:16.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:16 smithi097 ceph-mon[31898]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:56:16.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:16 smithi097 ceph-mon[31898]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-11T19:56:16.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:16 smithi097 ceph-mon[31898]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-11T19:56:16.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:16 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/4287536929' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T19:56:16.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:16 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T19:56:16.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:16 smithi114 ceph-mon[36376]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T19:56:16.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:16 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2767050389' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-11T19:56:16.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:16 smithi114 ceph-mon[36376]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-11T19:56:16.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:16 smithi114 ceph-mon[36376]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:56:16.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:16 smithi114 ceph-mon[36376]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-11T19:56:16.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:16 smithi114 ceph-mon[36376]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-11T19:56:16.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:16 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/4287536929' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T19:56:16.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:16 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T19:56:17.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:16 smithi161 ceph-mon[36332]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T19:56:17.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:16 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2767050389' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-11T19:56:17.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:16 smithi161 ceph-mon[36332]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-11T19:56:17.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:16 smithi161 ceph-mon[36332]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T19:56:17.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:16 smithi161 ceph-mon[36332]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-11T19:56:17.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:16 smithi161 ceph-mon[36332]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-11T19:56:17.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:16 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/4287536929' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T19:56:17.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:16 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T19:56:17.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.135+0000 7f3662800640 1 -- 172.21.15.97:0/3558800593 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f365c02cdd0 con 0x7f365c104420 2024-03-11T19:56:17.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.645+0000 7f365a000640 1 -- 172.21.15.97:0/3558800593 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f3648043020 con 0x7f365c104420 2024-03-11T19:56:17.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.648+0000 7f3662800640 1 -- 172.21.15.97:0/3558800593 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f36440a1a30 msgr2=0x7f36440a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:17.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.648+0000 7f3662800640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f36440a1a30 0x7f36440a3ef0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3650005fd0 tx=0x7f3650009500 comp rx=0 tx=0).stop 2024-03-11T19:56:17.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.648+0000 7f3662800640 1 -- 172.21.15.97:0/3558800593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c104420 msgr2=0x7f365c1998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.648+0000 7f3662800640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c104420 0x7f365c1998f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3648000c00 tx=0x7f3648005e80 comp rx=0 tx=0).stop 2024-03-11T19:56:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.648+0000 7f3662800640 1 -- 172.21.15.97:0/3558800593 shutdown_connections 2024-03-11T19:56:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.648+0000 7f3662800640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c106d50 0x7f365c1a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.648+0000 7f3662800640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f36440a1a30 0x7f36440a3ef0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.648+0000 7f3662800640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c104420 0x7f365c1998f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.648+0000 7f3662800640 1 --2- 172.21.15.97:0/3558800593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c101af0 0x7f365c1993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.648+0000 7f3662800640 1 -- 172.21.15.97:0/3558800593 >> 172.21.15.97:0/3558800593 conn(0x7f365c0fb500 msgr2=0x7f365c101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.648+0000 7f3662800640 1 -- 172.21.15.97:0/3558800593 shutdown_connections 2024-03-11T19:56:17.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.648+0000 7f3662800640 1 -- 172.21.15.97:0/3558800593 wait complete. 2024-03-11T19:56:17.671 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-03-11T19:56:17.672 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-11T19:56:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.859+0000 7fe67da00640 1 -- 172.21.15.97:0/1316503868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe678075160 msgr2=0x7fe6780735b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.859+0000 7fe67da00640 1 --2- 172.21.15.97:0/1316503868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe678075160 0x7fe6780735b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe668009860 tx=0x7fe66802bc10 comp rx=0 tx=0).stop 2024-03-11T19:56:17.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.859+0000 7fe67da00640 1 -- 172.21.15.97:0/1316503868 shutdown_connections 2024-03-11T19:56:17.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.859+0000 7fe67da00640 1 --2- 172.21.15.97:0/1316503868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe678108f10 0x7fe67810b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:17.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.859+0000 7fe67da00640 1 --2- 172.21.15.97:0/1316503868 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe678073b80 0x7fe678073fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:17.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.859+0000 7fe67da00640 1 --2- 172.21.15.97:0/1316503868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe678075160 0x7fe6780735b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:17.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.859+0000 7fe67da00640 1 -- 172.21.15.97:0/1316503868 >> 172.21.15.97:0/1316503868 conn(0x7fe6780fc5f0 msgr2=0x7fe6780fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:17.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.859+0000 7fe67da00640 1 -- 172.21.15.97:0/1316503868 shutdown_connections 2024-03-11T19:56:17.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.859+0000 7fe67da00640 1 -- 172.21.15.97:0/1316503868 wait complete. 2024-03-11T19:56:17.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.860+0000 7fe67da00640 1 Processor -- start 2024-03-11T19:56:17.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.860+0000 7fe67da00640 1 -- start start 2024-03-11T19:56:17.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.860+0000 7fe67da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe678073b80 0x7fe678199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.860+0000 7fe67da00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe678075160 0x7fe678199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.860+0000 7fe67da00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe678108f10 0x7fe6781a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.860+0000 7fe67da00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6781a0e10 con 0x7fe678075160 2024-03-11T19:56:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.860+0000 7fe67da00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6781a0f80 con 0x7fe678073b80 2024-03-11T19:56:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.860+0000 7fe67da00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6781a1140 con 0x7fe678108f10 2024-03-11T19:56:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.861+0000 7fe66fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe678075160 0x7fe678199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.861+0000 7fe66fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe678075160 0x7fe678199860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59734/0 (socket says 172.21.15.97:59734) 2024-03-11T19:56:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.861+0000 7fe66fe00640 1 -- 172.21.15.97:0/1707451954 learned_addr learned my addr 172.21.15.97:0/1707451954 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.861+0000 7fe677e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe678073b80 0x7fe678199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.861+0000 7fe67d000640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe678108f10 0x7fe6781a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.861+0000 7fe677e00640 1 -- 172.21.15.97:0/1707451954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe678108f10 msgr2=0x7fe6781a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.861+0000 7fe677e00640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe678108f10 0x7fe6781a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.861+0000 7fe677e00640 1 -- 172.21.15.97:0/1707451954 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe678075160 msgr2=0x7fe678199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.861+0000 7fe677e00640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe678075160 0x7fe678199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.861+0000 7fe677e00640 1 -- 172.21.15.97:0/1707451954 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe660009560 con 0x7fe678073b80 2024-03-11T19:56:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.861+0000 7fe66fe00640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe678075160 0x7fe678199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.861+0000 7fe677e00640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe678073b80 0x7fe678199320 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe668009780 tx=0x7fe6680346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.862+0000 7fe675600640 1 -- 172.21.15.97:0/1707451954 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668005250 con 0x7fe678073b80 2024-03-11T19:56:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.862+0000 7fe67da00640 1 -- 172.21.15.97:0/1707451954 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe668009490 con 0x7fe678073b80 2024-03-11T19:56:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.862+0000 7fe675600640 1 -- 172.21.15.97:0/1707451954 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe66802ca90 con 0x7fe678073b80 2024-03-11T19:56:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.862+0000 7fe675600640 1 -- 172.21.15.97:0/1707451954 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668004c60 con 0x7fe678073b80 2024-03-11T19:56:17.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.862+0000 7fe67da00640 1 -- 172.21.15.97:0/1707451954 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6781a1700 con 0x7fe678073b80 2024-03-11T19:56:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.863+0000 7fe67da00640 1 -- 172.21.15.97:0/1707451954 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe678107cd0 con 0x7fe678073b80 2024-03-11T19:56:17.866 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.865+0000 7fe675600640 1 -- 172.21.15.97:0/1707451954 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe668005440 con 0x7fe678073b80 2024-03-11T19:56:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.866+0000 7fe675600640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe65c0a1a30 0x7fe65c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.866+0000 7fe675600640 1 -- 172.21.15.97:0/1707451954 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe6680ef940 con 0x7fe678073b80 2024-03-11T19:56:17.867 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.866+0000 7fe66fe00640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe65c0a1a30 0x7fe65c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:17.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.867+0000 7fe66fe00640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe65c0a1a30 0x7fe65c0a3ef0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe67819a830 tx=0x7fe660009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:17.869 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:17.868+0000 7fe675600640 1 -- 172.21.15.97:0/1707451954 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe6680b88b0 con 0x7fe678073b80 2024-03-11T19:56:17.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:17 smithi097 ceph-mon[31898]: pgmap v335: 129 pgs: 129 active+clean; 582 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T19:56:17.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:17 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-11T19:56:17.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:17 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] up:active 2024-03-11T19:56:17.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:17 smithi097 ceph-mon[31898]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-11T19:56:17.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:17 smithi097 ceph-mon[31898]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-11T19:56:17.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:17 smithi097 ceph-mon[31898]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-03-11T19:56:17.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:17 smithi097 ceph-mon[31898]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-11T19:56:17.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:17 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3558800593' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-11T19:56:17.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:17 smithi114 ceph-mon[36376]: pgmap v335: 129 pgs: 129 active+clean; 582 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T19:56:17.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:17 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-11T19:56:17.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:17 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] up:active 2024-03-11T19:56:17.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:17 smithi114 ceph-mon[36376]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-11T19:56:17.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:17 smithi114 ceph-mon[36376]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-11T19:56:17.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:17 smithi114 ceph-mon[36376]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-03-11T19:56:17.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:17 smithi114 ceph-mon[36376]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-11T19:56:17.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:17 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3558800593' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-11T19:56:18.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:17 smithi161 ceph-mon[36332]: pgmap v335: 129 pgs: 129 active+clean; 582 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T19:56:18.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:17 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-11T19:56:18.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:17 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] up:active 2024-03-11T19:56:18.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:17 smithi161 ceph-mon[36332]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-11T19:56:18.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:17 smithi161 ceph-mon[36332]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-11T19:56:18.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:17 smithi161 ceph-mon[36332]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-03-11T19:56:18.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:17 smithi161 ceph-mon[36332]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-11T19:56:18.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:17 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3558800593' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-11T19:56:18.085 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.084+0000 7fe67da00640 1 -- 172.21.15.97:0/1707451954 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6780615c0 con 0x7fe65c0a1a30 2024-03-11T19:56:18.419 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:56:18.092+0000 7f1fe3e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:56:18.419 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:56:18.092+0000 7f1fe3e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:56:18.419 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:56:18.092+0000 7f1fe3e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:56:18.419 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:56:18.092+0000 7f1fe3e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:56:18.419 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:56:18.092+0000 7f1fe3e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:56:18.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.651+0000 7fe675600640 1 -- 172.21.15.97:0/1707451954 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe6780615c0 con 0x7fe65c0a1a30 2024-03-11T19:56:18.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.654+0000 7fe67da00640 1 -- 172.21.15.97:0/1707451954 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe65c0a1a30 msgr2=0x7fe65c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:18.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.654+0000 7fe67da00640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe65c0a1a30 0x7fe65c0a3ef0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe67819a830 tx=0x7fe660009260 comp rx=0 tx=0).stop 2024-03-11T19:56:18.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.654+0000 7fe67da00640 1 -- 172.21.15.97:0/1707451954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe678073b80 msgr2=0x7fe678199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:18.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.654+0000 7fe67da00640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe678073b80 0x7fe678199320 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe668009780 tx=0x7fe6680346f0 comp rx=0 tx=0).stop 2024-03-11T19:56:18.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.655+0000 7fe67da00640 1 -- 172.21.15.97:0/1707451954 shutdown_connections 2024-03-11T19:56:18.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.655+0000 7fe67da00640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe678108f10 0x7fe6781a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:18.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.655+0000 7fe67da00640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe65c0a1a30 0x7fe65c0a3ef0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:18.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.655+0000 7fe67da00640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe678075160 0x7fe678199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:18.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.655+0000 7fe67da00640 1 --2- 172.21.15.97:0/1707451954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe678073b80 0x7fe678199320 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:18.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.655+0000 7fe67da00640 1 -- 172.21.15.97:0/1707451954 >> 172.21.15.97:0/1707451954 conn(0x7fe6780fc5f0 msgr2=0x7fe678104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:18.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.655+0000 7fe67da00640 1 -- 172.21.15.97:0/1707451954 shutdown_connections 2024-03-11T19:56:18.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.655+0000 7fe67da00640 1 -- 172.21.15.97:0/1707451954 wait complete. 2024-03-11T19:56:18.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-11T19:56:18.738 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3558800593' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-11T19:56:18.739 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] up:active 2024-03-11T19:56:18.739 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-mon[31898]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-11T19:56:18.739 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-mon[31898]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-03-11T19:56:18.739 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-mon[31898]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-03-11T19:56:18.739 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-mon[31898]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-11T19:56:18.739 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] up:active 2024-03-11T19:56:18.739 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-mon[31898]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-11T19:56:18.739 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-mon[31898]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T19:56:18.739 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-mon[31898]: Cluster is now healthy 2024-03-11T19:56:18.739 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-mon[31898]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=i=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-03-11T19:56:18.739 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-mon[31898]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-11T19:56:18.739 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:18 smithi097 ceph-mon[31898]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-11T19:56:18.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.874+0000 7faef6400640 1 -- 172.21.15.97:0/4169850006 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faef0104f70 msgr2=0x7faef010b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:18.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.874+0000 7faef6400640 1 --2- 172.21.15.97:0/4169850006 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faef0104f70 0x7faef010b800 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faee8009900 tx=0x7faee802bc30 comp rx=0 tx=0).stop 2024-03-11T19:56:18.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.875+0000 7faef6400640 1 -- 172.21.15.97:0/4169850006 shutdown_connections 2024-03-11T19:56:18.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.875+0000 7faef6400640 1 --2- 172.21.15.97:0/4169850006 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faef0104f70 0x7faef010b800 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:18.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.875+0000 7faef6400640 1 --2- 172.21.15.97:0/4169850006 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faef01045b0 0x7faef0104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:18.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.875+0000 7faef6400640 1 --2- 172.21.15.97:0/4169850006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef01033c0 0x7faef01037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:18.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.875+0000 7faef6400640 1 -- 172.21.15.97:0/4169850006 >> 172.21.15.97:0/4169850006 conn(0x7faef00fe930 msgr2=0x7faef0100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:18.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.875+0000 7faef6400640 1 -- 172.21.15.97:0/4169850006 shutdown_connections 2024-03-11T19:56:18.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.875+0000 7faef6400640 1 -- 172.21.15.97:0/4169850006 wait complete. 2024-03-11T19:56:18.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.876+0000 7faef6400640 1 Processor -- start 2024-03-11T19:56:18.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.876+0000 7faef6400640 1 -- start start 2024-03-11T19:56:18.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.876+0000 7faef6400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faef01033c0 0x7faef0199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:18.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.876+0000 7faef6400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef01045b0 0x7faef0199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:18.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.876+0000 7faef6400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faef0104f70 0x7faef01a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:18.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.876+0000 7faef6400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef01a0f10 con 0x7faef01033c0 2024-03-11T19:56:18.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.876+0000 7faef6400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef01a1080 con 0x7faef01045b0 2024-03-11T19:56:18.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.876+0000 7faef6400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef01a1240 con 0x7faef0104f70 2024-03-11T19:56:18.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.876+0000 7faeef400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faef01033c0 0x7faef0199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:18.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.876+0000 7faeeea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef01045b0 0x7faef0199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:18.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.876+0000 7faeef400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faef01033c0 0x7faef0199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59764/0 (socket says 172.21.15.97:59764) 2024-03-11T19:56:18.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.876+0000 7faeef400640 1 -- 172.21.15.97:0/1278743733 learned_addr learned my addr 172.21.15.97:0/1278743733 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:18.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.876+0000 7faeefe00640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faef0104f70 0x7faef01a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:18.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.877+0000 7faeeea00640 1 -- 172.21.15.97:0/1278743733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faef0104f70 msgr2=0x7faef01a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:18.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.877+0000 7faeeea00640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faef0104f70 0x7faef01a09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:18.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.877+0000 7faeeea00640 1 -- 172.21.15.97:0/1278743733 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faef01033c0 msgr2=0x7faef0199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:18.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.877+0000 7faeeea00640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faef01033c0 0x7faef0199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:18.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.877+0000 7faeeea00640 1 -- 172.21.15.97:0/1278743733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee8009560 con 0x7faef01045b0 2024-03-11T19:56:18.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.877+0000 7faeef400640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faef01033c0 0x7faef0199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:18.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.877+0000 7faeeea00640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef01045b0 0x7faef0199960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faed800e9f0 tx=0x7faed800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:18.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.877+0000 7faee7e00640 1 -- 172.21.15.97:0/1278743733 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed800ccc0 con 0x7faef01045b0 2024-03-11T19:56:18.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.877+0000 7faee7e00640 1 -- 172.21.15.97:0/1278743733 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faed800ce20 con 0x7faef01045b0 2024-03-11T19:56:18.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.877+0000 7faef6400640 1 -- 172.21.15.97:0/1278743733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef01a1520 con 0x7faef01045b0 2024-03-11T19:56:18.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.877+0000 7faee7e00640 1 -- 172.21.15.97:0/1278743733 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8018920 con 0x7faef01045b0 2024-03-11T19:56:18.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.878+0000 7faef6400640 1 -- 172.21.15.97:0/1278743733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef01a1a30 con 0x7faef01045b0 2024-03-11T19:56:18.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.878+0000 7faef6400640 1 -- 172.21.15.97:0/1278743733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef010a6e0 con 0x7faef01045b0 2024-03-11T19:56:18.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.880+0000 7faee7e00640 1 -- 172.21.15.97:0/1278743733 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7faed8018a80 con 0x7faef01045b0 2024-03-11T19:56:18.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.882+0000 7faee7e00640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7faed40a1a30 0x7faed40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:18.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.882+0000 7faee7e00640 1 -- 172.21.15.97:0/1278743733 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7faed8014070 con 0x7faef01045b0 2024-03-11T19:56:18.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.882+0000 7faeef400640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7faed40a1a30 0x7faed40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:18.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.882+0000 7faeef400640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7faed40a1a30 0x7faed40a3ef0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faee000f7b0 tx=0x7faee0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:18.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:18.886+0000 7faee7e00640 1 -- 172.21.15.97:0/1278743733 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faed8052d20 con 0x7faef01045b0 2024-03-11T19:56:18.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:18 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3558800593' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-11T19:56:18.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:18 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] up:active 2024-03-11T19:56:18.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:18 smithi114 ceph-mon[36376]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-11T19:56:18.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:18 smithi114 ceph-mon[36376]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-03-11T19:56:18.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:18 smithi114 ceph-mon[36376]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-03-11T19:56:18.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:18 smithi114 ceph-mon[36376]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-11T19:56:18.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:18 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] up:active 2024-03-11T19:56:18.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:18 smithi114 ceph-mon[36376]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-11T19:56:18.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:18 smithi114 ceph-mon[36376]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T19:56:18.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:18 smithi114 ceph-mon[36376]: Cluster is now healthy 2024-03-11T19:56:18.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:18 smithi114 ceph-mon[36376]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=i=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-03-11T19:56:18.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:18 smithi114 ceph-mon[36376]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-11T19:56:18.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:18 smithi114 ceph-mon[36376]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-11T19:56:19.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:18 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3558800593' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-11T19:56:19.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:18 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] up:active 2024-03-11T19:56:19.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:18 smithi161 ceph-mon[36332]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-11T19:56:19.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:18 smithi161 ceph-mon[36332]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-03-11T19:56:19.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:18 smithi161 ceph-mon[36332]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-03-11T19:56:19.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:18 smithi161 ceph-mon[36332]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-11T19:56:19.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:18 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] up:active 2024-03-11T19:56:19.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:18 smithi161 ceph-mon[36332]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-11T19:56:19.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:18 smithi161 ceph-mon[36332]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T19:56:19.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:18 smithi161 ceph-mon[36332]: Cluster is now healthy 2024-03-11T19:56:19.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:18 smithi161 ceph-mon[36332]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=i=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-03-11T19:56:19.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:18 smithi161 ceph-mon[36332]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-11T19:56:19.038 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:18 smithi161 ceph-mon[36332]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-11T19:56:19.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.096+0000 7faef6400640 1 -- 172.21.15.97:0/1278743733 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7faef002ceb0 con 0x7faed40a1a30 2024-03-11T19:56:19.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.125+0000 7faee7e00640 1 -- 172.21.15.97:0/1278743733 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faef002ceb0 con 0x7faed40a1a30 2024-03-11T19:56:19.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.128+0000 7faef6400640 1 -- 172.21.15.97:0/1278743733 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7faed40a1a30 msgr2=0x7faed40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:19.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.128+0000 7faef6400640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7faed40a1a30 0x7faed40a3ef0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faee000f7b0 tx=0x7faee0009500 comp rx=0 tx=0).stop 2024-03-11T19:56:19.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.128+0000 7faef6400640 1 -- 172.21.15.97:0/1278743733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef01045b0 msgr2=0x7faef0199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.128+0000 7faef6400640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef01045b0 0x7faef0199960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faed800e9f0 tx=0x7faed800eeb0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.128+0000 7faef6400640 1 -- 172.21.15.97:0/1278743733 shutdown_connections 2024-03-11T19:56:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.128+0000 7faef6400640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faef0104f70 0x7faef01a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.128+0000 7faef6400640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faef01045b0 0x7faef0199960 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.128+0000 7faef6400640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7faed40a1a30 0x7faed40a3ef0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.128+0000 7faef6400640 1 --2- 172.21.15.97:0/1278743733 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faef01033c0 0x7faef0199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.128+0000 7faef6400640 1 -- 172.21.15.97:0/1278743733 >> 172.21.15.97:0/1278743733 conn(0x7faef00fe930 msgr2=0x7faef01078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.128+0000 7faef6400640 1 -- 172.21.15.97:0/1278743733 shutdown_connections 2024-03-11T19:56:19.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.128+0000 7faef6400640 1 -- 172.21.15.97:0/1278743733 wait complete. 2024-03-11T19:56:19.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-11T19:56:19.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.357+0000 7fae9f200640 1 -- 172.21.15.97:0/3265274922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae98104590 msgr2=0x7fae98104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:19.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.357+0000 7fae9f200640 1 --2- 172.21.15.97:0/3265274922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae98104590 0x7fae98104a10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fae80009900 tx=0x7fae8002bc30 comp rx=0 tx=0).stop 2024-03-11T19:56:19.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.357+0000 7fae9f200640 1 -- 172.21.15.97:0/3265274922 shutdown_connections 2024-03-11T19:56:19.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.357+0000 7fae9f200640 1 --2- 172.21.15.97:0/3265274922 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae98104f50 0x7fae9810b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.357+0000 7fae9f200640 1 --2- 172.21.15.97:0/3265274922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae98104590 0x7fae98104a10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.357+0000 7fae9f200640 1 --2- 172.21.15.97:0/3265274922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae981033a0 0x7fae981037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.357+0000 7fae9f200640 1 -- 172.21.15.97:0/3265274922 >> 172.21.15.97:0/3265274922 conn(0x7fae980fe930 msgr2=0x7fae98100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:19.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.357+0000 7fae9f200640 1 -- 172.21.15.97:0/3265274922 shutdown_connections 2024-03-11T19:56:19.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.357+0000 7fae9f200640 1 -- 172.21.15.97:0/3265274922 wait complete. 2024-03-11T19:56:19.361 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.358+0000 7fae9f200640 1 Processor -- start 2024-03-11T19:56:19.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.358+0000 7fae9f200640 1 -- start start 2024-03-11T19:56:19.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.358+0000 7fae9f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae981033a0 0x7fae98199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:19.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.358+0000 7fae9f200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae98104590 0x7fae98199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:19.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.358+0000 7fae9f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae98104f50 0x7fae981a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:19.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.358+0000 7fae9f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae981a0f00 con 0x7fae981033a0 2024-03-11T19:56:19.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.358+0000 7fae9f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae981a1070 con 0x7fae98104f50 2024-03-11T19:56:19.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.358+0000 7fae9f200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae981a1230 con 0x7fae98104590 2024-03-11T19:56:19.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.359+0000 7fae97e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae981033a0 0x7fae98199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:19.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.359+0000 7fae8f400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae98104590 0x7fae98199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:19.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.359+0000 7fae8f400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae98104590 0x7fae98199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:57610/0 (socket says 172.21.15.97:57610) 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.359+0000 7fae8f400640 1 -- 172.21.15.97:0/3110263730 learned_addr learned my addr 172.21.15.97:0/3110263730 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.359+0000 7fae9d200640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae98104f50 0x7fae981a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.359+0000 7fae8f400640 1 -- 172.21.15.97:0/3110263730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae98104f50 msgr2=0x7fae981a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.359+0000 7fae8f400640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae98104f50 0x7fae981a09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.359+0000 7fae8f400640 1 -- 172.21.15.97:0/3110263730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae981033a0 msgr2=0x7fae98199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.359+0000 7fae8f400640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae981033a0 0x7fae98199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.359+0000 7fae8f400640 1 -- 172.21.15.97:0/3110263730 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae90009840 con 0x7fae98104590 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.359+0000 7fae9d200640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae98104f50 0x7fae981a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.359+0000 7fae8f400640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae98104590 0x7fae98199950 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fae80006010 tx=0x7fae80005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.360+0000 7fae95600640 1 -- 172.21.15.97:0/3110263730 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae800043d0 con 0x7fae98104590 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.360+0000 7fae9f200640 1 -- 172.21.15.97:0/3110263730 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae80009560 con 0x7fae98104590 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.360+0000 7fae95600640 1 -- 172.21.15.97:0/3110263730 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae800052d0 con 0x7fae98104590 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.360+0000 7fae95600640 1 -- 172.21.15.97:0/3110263730 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80003bc0 con 0x7fae98104590 2024-03-11T19:56:19.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.360+0000 7fae9f200640 1 -- 172.21.15.97:0/3110263730 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae981a1830 con 0x7fae98104590 2024-03-11T19:56:19.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.361+0000 7fae9f200640 1 -- 172.21.15.97:0/3110263730 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9810a790 con 0x7fae98104590 2024-03-11T19:56:19.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.363+0000 7fae95600640 1 -- 172.21.15.97:0/3110263730 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fae80003d20 con 0x7fae98104590 2024-03-11T19:56:19.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.365+0000 7fae95600640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fae7c0a1a30 0x7fae7c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:19.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.365+0000 7fae95600640 1 -- 172.21.15.97:0/3110263730 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fae800efe80 con 0x7fae98104590 2024-03-11T19:56:19.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.365+0000 7fae97e00640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fae7c0a1a30 0x7fae7c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:19.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.366+0000 7fae97e00640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fae7c0a1a30 0x7fae7c0a3ef0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fae981043f0 tx=0x7fae88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:19.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.367+0000 7fae95600640 1 -- 172.21.15.97:0/3110263730 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fae800b8df0 con 0x7fae98104590 2024-03-11T19:56:19.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.595+0000 7fae9f200640 1 -- 172.21.15.97:0/3110263730 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fae9802ce40 con 0x7fae7c0a1a30 2024-03-11T19:56:19.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.603+0000 7fae95600640 1 -- 172.21.15.97:0/3110263730 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fae9802ce40 con 0x7fae7c0a1a30 2024-03-11T19:56:19.605 INFO:teuthology.orchestra.run.smithi097.stdout:/volumes/qa/sv_0/ac076e8e-4189-4d5e-a393-3598fe078c9f 2024-03-11T19:56:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.606+0000 7fae9f200640 1 -- 172.21.15.97:0/3110263730 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fae7c0a1a30 msgr2=0x7fae7c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.606+0000 7fae9f200640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fae7c0a1a30 0x7fae7c0a3ef0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fae981043f0 tx=0x7fae88005f90 comp rx=0 tx=0).stop 2024-03-11T19:56:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.606+0000 7fae9f200640 1 -- 172.21.15.97:0/3110263730 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae98104590 msgr2=0x7fae98199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.606+0000 7fae9f200640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae98104590 0x7fae98199950 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fae80006010 tx=0x7fae80005e80 comp rx=0 tx=0).stop 2024-03-11T19:56:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.606+0000 7fae9f200640 1 -- 172.21.15.97:0/3110263730 shutdown_connections 2024-03-11T19:56:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.606+0000 7fae9f200640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae98104f50 0x7fae981a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.606+0000 7fae9f200640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae98104590 0x7fae98199950 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.606+0000 7fae9f200640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fae7c0a1a30 0x7fae7c0a3ef0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.606+0000 7fae9f200640 1 --2- 172.21.15.97:0/3110263730 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fae981033a0 0x7fae98199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.606+0000 7fae9f200640 1 -- 172.21.15.97:0/3110263730 >> 172.21.15.97:0/3110263730 conn(0x7fae980fe930 msgr2=0x7fae98107960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.606+0000 7fae9f200640 1 -- 172.21.15.97:0/3110263730 shutdown_connections 2024-03-11T19:56:19.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.606+0000 7fae9f200640 1 -- 172.21.15.97:0/3110263730 wait complete. 2024-03-11T19:56:19.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-11T19:56:19.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.833+0000 7f9c76800640 1 -- 172.21.15.97:0/1682036038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c70104f70 msgr2=0x7f9c7010b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:19.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.833+0000 7f9c76800640 1 --2- 172.21.15.97:0/1682036038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c70104f70 0x7f9c7010b800 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9c68009be0 tx=0x7f9c6802bef0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.834+0000 7f9c76800640 1 -- 172.21.15.97:0/1682036038 shutdown_connections 2024-03-11T19:56:19.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.834+0000 7f9c76800640 1 --2- 172.21.15.97:0/1682036038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c70104f70 0x7f9c7010b800 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.834+0000 7f9c76800640 1 --2- 172.21.15.97:0/1682036038 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c701045b0 0x7f9c70104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.834+0000 7f9c76800640 1 --2- 172.21.15.97:0/1682036038 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c701033c0 0x7f9c701037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.834+0000 7f9c76800640 1 -- 172.21.15.97:0/1682036038 >> 172.21.15.97:0/1682036038 conn(0x7f9c700fe930 msgr2=0x7f9c70100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:19.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.834+0000 7f9c76800640 1 -- 172.21.15.97:0/1682036038 shutdown_connections 2024-03-11T19:56:19.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.834+0000 7f9c76800640 1 -- 172.21.15.97:0/1682036038 wait complete. 2024-03-11T19:56:19.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.835+0000 7f9c76800640 1 Processor -- start 2024-03-11T19:56:19.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.835+0000 7f9c76800640 1 -- start start 2024-03-11T19:56:19.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.835+0000 7f9c76800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c701033c0 0x7f9c70199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.835+0000 7f9c76800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c701045b0 0x7f9c701999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.835+0000 7f9c76800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c70104f70 0x7f9c701a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.835+0000 7f9c76800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c701a0f60 con 0x7f9c70104f70 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.835+0000 7f9c76800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c701a10d0 con 0x7f9c701033c0 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.835+0000 7f9c76800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c701a1290 con 0x7f9c701045b0 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.835+0000 7f9c6fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c70104f70 0x7f9c701a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.835+0000 7f9c66a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c701045b0 0x7f9c701999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.835+0000 7f9c6f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c701033c0 0x7f9c70199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.835+0000 7f9c66a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c701045b0 0x7f9c701999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:57624/0 (socket says 172.21.15.97:57624) 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.835+0000 7f9c66a00640 1 -- 172.21.15.97:0/1891061077 learned_addr learned my addr 172.21.15.97:0/1891061077 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.836+0000 7f9c66a00640 1 -- 172.21.15.97:0/1891061077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c701033c0 msgr2=0x7f9c70199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.836+0000 7f9c66a00640 1 --2- 172.21.15.97:0/1891061077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c701033c0 0x7f9c70199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.836+0000 7f9c66a00640 1 -- 172.21.15.97:0/1891061077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c70104f70 msgr2=0x7f9c701a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.836+0000 7f9c66a00640 1 --2- 172.21.15.97:0/1891061077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c70104f70 0x7f9c701a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:19.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.836+0000 7f9c66a00640 1 -- 172.21.15.97:0/1891061077 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c60009560 con 0x7f9c701045b0 2024-03-11T19:56:19.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.836+0000 7f9c66a00640 1 --2- 172.21.15.97:0/1891061077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c701045b0 0x7f9c701999b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800e9f0 tx=0x7f9c5800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:19.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.836+0000 7f9c6cc00640 1 -- 172.21.15.97:0/1891061077 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58009700 con 0x7f9c701045b0 2024-03-11T19:56:19.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.836+0000 7f9c6cc00640 1 -- 172.21.15.97:0/1891061077 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c5800cd80 con 0x7f9c701045b0 2024-03-11T19:56:19.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.836+0000 7f9c6cc00640 1 -- 172.21.15.97:0/1891061077 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58018920 con 0x7f9c701045b0 2024-03-11T19:56:19.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.836+0000 7f9c76800640 1 -- 172.21.15.97:0/1891061077 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c68009840 con 0x7f9c701045b0 2024-03-11T19:56:19.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.836+0000 7f9c76800640 1 -- 172.21.15.97:0/1891061077 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c701a18f0 con 0x7f9c701045b0 2024-03-11T19:56:19.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.837+0000 7f9c76800640 1 -- 172.21.15.97:0/1891061077 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c7010a680 con 0x7f9c701045b0 2024-03-11T19:56:19.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.839+0000 7f9c6cc00640 1 -- 172.21.15.97:0/1891061077 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f9c58018a80 con 0x7f9c701045b0 2024-03-11T19:56:19.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.840+0000 7f9c6cc00640 1 --2- 172.21.15.97:0/1891061077 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9c540a1ac0 0x7f9c540a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:19.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.840+0000 7f9c6cc00640 1 -- 172.21.15.97:0/1891061077 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9c58014070 con 0x7f9c701045b0 2024-03-11T19:56:19.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.841+0000 7f9c6f400640 1 --2- 172.21.15.97:0/1891061077 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9c540a1ac0 0x7f9c540a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:19.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.841+0000 7f9c6f400640 1 --2- 172.21.15.97:0/1891061077 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9c540a1ac0 0x7f9c540a3f80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009530 tx=0x7f9c60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:19.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:19.845+0000 7f9c6cc00640 1 -- 172.21.15.97:0/1891061077 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9c58052d20 con 0x7f9c701045b0 2024-03-11T19:56:19.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:19 smithi097 ceph-mon[31898]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:19.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:19 smithi097 ceph-mon[31898]: pgmap v336: 129 pgs: 129 active+clean; 584 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 5 op/s 2024-03-11T19:56:19.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:19 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] up:active 2024-03-11T19:56:19.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:19 smithi097 ceph-mon[31898]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-11T19:56:19.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:19 smithi097 ceph-mon[31898]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-11T19:56:19.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:19 smithi114 ceph-mon[36376]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:19.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:19 smithi114 ceph-mon[36376]: pgmap v336: 129 pgs: 129 active+clean; 584 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 5 op/s 2024-03-11T19:56:19.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:19 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] up:active 2024-03-11T19:56:19.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:19 smithi114 ceph-mon[36376]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-11T19:56:19.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:19 smithi114 ceph-mon[36376]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-11T19:56:19.991 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 19:56:19 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b[66693]: 2024-03-11T19:56:19.565+0000 7f7bc72f8700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-11T19:56:19.991 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 19:56:19 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b[66693]: 2024-03-11T19:56:19.565+0000 7f7bc72f8700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-11T19:56:20.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:19 smithi161 ceph-mon[36332]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:20.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:19 smithi161 ceph-mon[36332]: pgmap v336: 129 pgs: 129 active+clean; 584 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 5 op/s 2024-03-11T19:56:20.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:19 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] up:active 2024-03-11T19:56:20.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:19 smithi161 ceph-mon[36332]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-11T19:56:20.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:19 smithi161 ceph-mon[36332]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-11T19:56:20.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.071+0000 7f9c76800640 1 -- 172.21.15.97:0/1891061077 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c7002ceb0 con 0x7f9c540a1ac0 2024-03-11T19:56:20.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.102+0000 7f9c6cc00640 1 -- 172.21.15.97:0/1891061077 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9c7002ceb0 con 0x7f9c540a1ac0 2024-03-11T19:56:20.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.105+0000 7f9c76800640 1 -- 172.21.15.97:0/1891061077 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9c540a1ac0 msgr2=0x7f9c540a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:20.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.105+0000 7f9c76800640 1 --2- 172.21.15.97:0/1891061077 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9c540a1ac0 0x7f9c540a3f80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9c60009530 tx=0x7f9c60009260 comp rx=0 tx=0).stop 2024-03-11T19:56:20.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.105+0000 7f9c76800640 1 -- 172.21.15.97:0/1891061077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c701045b0 msgr2=0x7f9c701999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:20.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.105+0000 7f9c76800640 1 --2- 172.21.15.97:0/1891061077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c701045b0 0x7f9c701999b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800e9f0 tx=0x7f9c5800eeb0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.105+0000 7f9c76800640 1 -- 172.21.15.97:0/1891061077 shutdown_connections 2024-03-11T19:56:20.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.105+0000 7f9c76800640 1 --2- 172.21.15.97:0/1891061077 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9c540a1ac0 0x7f9c540a3f80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.105+0000 7f9c76800640 1 --2- 172.21.15.97:0/1891061077 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9c70104f70 0x7f9c701a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.105+0000 7f9c76800640 1 --2- 172.21.15.97:0/1891061077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9c701045b0 0x7f9c701999b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.105+0000 7f9c76800640 1 --2- 172.21.15.97:0/1891061077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c701033c0 0x7f9c70199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.105+0000 7f9c76800640 1 -- 172.21.15.97:0/1891061077 >> 172.21.15.97:0/1891061077 conn(0x7f9c700fe930 msgr2=0x7f9c70107850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:20.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.105+0000 7f9c76800640 1 -- 172.21.15.97:0/1891061077 shutdown_connections 2024-03-11T19:56:20.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.106+0000 7f9c76800640 1 -- 172.21.15.97:0/1891061077 wait complete. 2024-03-11T19:56:20.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-11T19:56:20.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.340+0000 7f8cc5600640 1 -- 172.21.15.97:0/3034976706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc00ffa80 msgr2=0x7f8cc00ffea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:20.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.340+0000 7f8cc5600640 1 --2- 172.21.15.97:0/3034976706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc00ffa80 0x7f8cc00ffea0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0009900 tx=0x7f8cb002bcc0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.341+0000 7f8cc5600640 1 -- 172.21.15.97:0/3034976706 shutdown_connections 2024-03-11T19:56:20.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.341+0000 7f8cc5600640 1 --2- 172.21.15.97:0/3034976706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc0100d80 0x7f8cc0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.341+0000 7f8cc5600640 1 --2- 172.21.15.97:0/3034976706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc01003e0 0x7f8cc0100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.341+0000 7f8cc5600640 1 --2- 172.21.15.97:0/3034976706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc00ffa80 0x7f8cc00ffea0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.341+0000 7f8cc5600640 1 -- 172.21.15.97:0/3034976706 >> 172.21.15.97:0/3034976706 conn(0x7f8cc00fb620 msgr2=0x7f8cc00fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:20.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.341+0000 7f8cc5600640 1 -- 172.21.15.97:0/3034976706 shutdown_connections 2024-03-11T19:56:20.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.341+0000 7f8cc5600640 1 -- 172.21.15.97:0/3034976706 wait complete. 2024-03-11T19:56:20.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.342+0000 7f8cc5600640 1 Processor -- start 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.342+0000 7f8cc5600640 1 -- start start 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.342+0000 7f8cc5600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc00ffa80 0x7f8cc0199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.342+0000 7f8cc5600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc01003e0 0x7f8cc01999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.342+0000 7f8cc5600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc0100d80 0x7f8cc01a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.342+0000 7f8cc5600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc01a0f60 con 0x7f8cc0100d80 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.342+0000 7f8cc5600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc01a10d0 con 0x7f8cc01003e0 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.342+0000 7f8cc5600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc01a1290 con 0x7f8cc00ffa80 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.342+0000 7f8cbf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc0100d80 0x7f8cc01a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.342+0000 7f8cbf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc0100d80 0x7f8cc01a0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59824/0 (socket says 172.21.15.97:59824) 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.342+0000 7f8cbf400640 1 -- 172.21.15.97:0/1126859119 learned_addr learned my addr 172.21.15.97:0/1126859119 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.342+0000 7f8cbea00640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc00ffa80 0x7f8cc0199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.342+0000 7f8cb7e00640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc01003e0 0x7f8cc01999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.343+0000 7f8cbea00640 1 -- 172.21.15.97:0/1126859119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc01003e0 msgr2=0x7f8cc01999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.343+0000 7f8cbea00640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc01003e0 0x7f8cc01999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.343+0000 7f8cbea00640 1 -- 172.21.15.97:0/1126859119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc0100d80 msgr2=0x7f8cc01a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:20.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.343+0000 7f8cbea00640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc0100d80 0x7f8cc01a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.343+0000 7f8cbea00640 1 -- 172.21.15.97:0/1126859119 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca8009490 con 0x7f8cc00ffa80 2024-03-11T19:56:20.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.343+0000 7f8cbf400640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc0100d80 0x7f8cc01a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:20.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.343+0000 7f8cbea00640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc00ffa80 0x7f8cc0199470 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0000c00 tx=0x7f8cb0005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:20.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.343+0000 7f8cb7400640 1 -- 172.21.15.97:0/1126859119 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb00043d0 con 0x7f8cc00ffa80 2024-03-11T19:56:20.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.343+0000 7f8cc5600640 1 -- 172.21.15.97:0/1126859119 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb0009560 con 0x7f8cc00ffa80 2024-03-11T19:56:20.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.343+0000 7f8cb7400640 1 -- 172.21.15.97:0/1126859119 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8cb00052d0 con 0x7f8cc00ffa80 2024-03-11T19:56:20.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.344+0000 7f8cb7400640 1 -- 172.21.15.97:0/1126859119 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0003d40 con 0x7f8cc00ffa80 2024-03-11T19:56:20.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.344+0000 7f8cc5600640 1 -- 172.21.15.97:0/1126859119 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc01a1860 con 0x7f8cc00ffa80 2024-03-11T19:56:20.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.345+0000 7f8cc5600640 1 -- 172.21.15.97:0/1126859119 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc010aa00 con 0x7f8cc00ffa80 2024-03-11T19:56:20.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.347+0000 7f8cb7400640 1 -- 172.21.15.97:0/1126859119 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f8cb0005440 con 0x7f8cc00ffa80 2024-03-11T19:56:20.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.348+0000 7f8cb7400640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8ca40a1ac0 0x7f8ca40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:20.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.348+0000 7f8cb7400640 1 -- 172.21.15.97:0/1126859119 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8cb00efdd0 con 0x7f8cc00ffa80 2024-03-11T19:56:20.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.348+0000 7f8cb7e00640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8ca40a1ac0 0x7f8ca40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:20.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.349+0000 7f8cb7e00640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8ca40a1ac0 0x7f8ca40a3f80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8cc019a980 tx=0x7f8ca8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:20.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.352+0000 7f8cb7400640 1 -- 172.21.15.97:0/1126859119 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8cb00b8cf0 con 0x7f8cc00ffa80 2024-03-11T19:56:20.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.578+0000 7f8cc5600640 1 -- 172.21.15.97:0/1126859119 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cc002ceb0 con 0x7f8ca40a1ac0 2024-03-11T19:56:20.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.587+0000 7f8cb7400640 1 -- 172.21.15.97:0/1126859119 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f8cc002ceb0 con 0x7f8ca40a1ac0 2024-03-11T19:56:20.589 INFO:teuthology.orchestra.run.smithi097.stdout:/volumes/qa/sv_1/326c5357-1c14-4d24-96f1-f194f5b2ba4f 2024-03-11T19:56:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.590+0000 7f8cc5600640 1 -- 172.21.15.97:0/1126859119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8ca40a1ac0 msgr2=0x7f8ca40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.590+0000 7f8cc5600640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8ca40a1ac0 0x7f8ca40a3f80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8cc019a980 tx=0x7f8ca8005f90 comp rx=0 tx=0).stop 2024-03-11T19:56:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.590+0000 7f8cc5600640 1 -- 172.21.15.97:0/1126859119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc00ffa80 msgr2=0x7f8cc0199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.590+0000 7f8cc5600640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc00ffa80 0x7f8cc0199470 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0000c00 tx=0x7f8cb0005e50 comp rx=0 tx=0).stop 2024-03-11T19:56:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.590+0000 7f8cc5600640 1 -- 172.21.15.97:0/1126859119 shutdown_connections 2024-03-11T19:56:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.590+0000 7f8cc5600640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8ca40a1ac0 0x7f8ca40a3f80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.590+0000 7f8cc5600640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8cc0100d80 0x7f8cc01a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.590+0000 7f8cc5600640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8cc01003e0 0x7f8cc01999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.590+0000 7f8cc5600640 1 --2- 172.21.15.97:0/1126859119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cc00ffa80 0x7f8cc0199470 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.590+0000 7f8cc5600640 1 -- 172.21.15.97:0/1126859119 >> 172.21.15.97:0/1126859119 conn(0x7f8cc00fb620 msgr2=0x7f8cc00fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.590+0000 7f8cc5600640 1 -- 172.21.15.97:0/1126859119 shutdown_connections 2024-03-11T19:56:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.590+0000 7f8cc5600640 1 -- 172.21.15.97:0/1126859119 wait complete. 2024-03-11T19:56:20.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:56:20.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.825+0000 7fe9e5e00640 1 -- 172.21.15.97:0/3343321745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e01077c0 msgr2=0x7fe9e0107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.825+0000 7fe9e5e00640 1 --2- 172.21.15.97:0/3343321745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e01077c0 0x7fe9e0107be0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009900 tx=0x7fe9d002bc30 comp rx=0 tx=0).stop 2024-03-11T19:56:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.825+0000 7fe9e5e00640 1 -- 172.21.15.97:0/3343321745 shutdown_connections 2024-03-11T19:56:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.825+0000 7fe9e5e00640 1 --2- 172.21.15.97:0/3343321745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9e0109370 0x7fe9e010fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.825+0000 7fe9e5e00640 1 --2- 172.21.15.97:0/3343321745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e01089b0 0x7fe9e0108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.825+0000 7fe9e5e00640 1 --2- 172.21.15.97:0/3343321745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e01077c0 0x7fe9e0107be0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.825+0000 7fe9e5e00640 1 -- 172.21.15.97:0/3343321745 >> 172.21.15.97:0/3343321745 conn(0x7fe9e0076790 msgr2=0x7fe9e0078bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.825+0000 7fe9e5e00640 1 -- 172.21.15.97:0/3343321745 shutdown_connections 2024-03-11T19:56:20.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.825+0000 7fe9e5e00640 1 -- 172.21.15.97:0/3343321745 wait complete. 2024-03-11T19:56:20.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.826+0000 7fe9e5e00640 1 Processor -- start 2024-03-11T19:56:20.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.826+0000 7fe9e5e00640 1 -- start start 2024-03-11T19:56:20.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9e5e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9e01077c0 0x7fe9e019d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:20.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9e5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e01089b0 0x7fe9e019dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:20.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9e5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e0109370 0x7fe9e01a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:20.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9e5e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e01a5340 con 0x7fe9e01089b0 2024-03-11T19:56:20.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9e5e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e01a54b0 con 0x7fe9e0109370 2024-03-11T19:56:20.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9e5e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e01a5670 con 0x7fe9e01077c0 2024-03-11T19:56:20.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9d6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e01089b0 0x7fe9e019dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:20.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9d6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e01089b0 0x7fe9e019dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59836/0 (socket says 172.21.15.97:59836) 2024-03-11T19:56:20.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9d6000640 1 -- 172.21.15.97:0/1715310700 learned_addr learned my addr 172.21.15.97:0/1715310700 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:20.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9dea00640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9e01077c0 0x7fe9e019d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:20.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9df400640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e0109370 0x7fe9e01a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:20.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9d6000640 1 -- 172.21.15.97:0/1715310700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9e01077c0 msgr2=0x7fe9e019d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:20.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9d6000640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9e01077c0 0x7fe9e019d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9d6000640 1 -- 172.21.15.97:0/1715310700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e0109370 msgr2=0x7fe9e01a4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:20.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9d6000640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e0109370 0x7fe9e01a4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9d6000640 1 -- 172.21.15.97:0/1715310700 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c8009490 con 0x7fe9e01089b0 2024-03-11T19:56:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.827+0000 7fe9df400640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e0109370 0x7fe9e01a4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T19:56:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.828+0000 7fe9d6000640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e01089b0 0x7fe9e019dd90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800ea60 tx=0x7fe9c800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.828+0000 7fe9e4a00640 1 -- 172.21.15.97:0/1715310700 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c800cde0 con 0x7fe9e01089b0 2024-03-11T19:56:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.828+0000 7fe9e4a00640 1 -- 172.21.15.97:0/1715310700 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9c80035f0 con 0x7fe9e01089b0 2024-03-11T19:56:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.828+0000 7fe9e5e00640 1 -- 172.21.15.97:0/1715310700 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d0009560 con 0x7fe9e01089b0 2024-03-11T19:56:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.828+0000 7fe9e4a00640 1 -- 172.21.15.97:0/1715310700 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8010690 con 0x7fe9e01089b0 2024-03-11T19:56:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.828+0000 7fe9e5e00640 1 -- 172.21.15.97:0/1715310700 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e01a5d20 con 0x7fe9e01089b0 2024-03-11T19:56:20.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.829+0000 7fe9e5e00640 1 -- 172.21.15.97:0/1715310700 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e010ea80 con 0x7fe9e01089b0 2024-03-11T19:56:20.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.831+0000 7fe9e4a00640 1 -- 172.21.15.97:0/1715310700 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fe9c8003af0 con 0x7fe9e01089b0 2024-03-11T19:56:20.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.832+0000 7fe9e4a00640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9c40a1b40 0x7fe9c40a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:20.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.832+0000 7fe9e4a00640 1 -- 172.21.15.97:0/1715310700 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe9c8014070 con 0x7fe9e01089b0 2024-03-11T19:56:20.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.833+0000 7fe9dea00640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9c40a1b40 0x7fe9c40a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:20.834 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.833+0000 7fe9dea00640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9c40a1b40 0x7fe9c40a4000 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe9d002c240 tx=0x7fe9d002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:20.837+0000 7fe9e4a00640 1 -- 172.21.15.97:0/1715310700 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe9c8096b00 con 0x7fe9e01089b0 2024-03-11T19:56:20.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:20 smithi097 ceph-mon[31898]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:20.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:20 smithi097 ceph-mon[31898]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:20.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:20 smithi097 ceph-mon[31898]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-11T19:56:20.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:20 smithi114 ceph-mon[36376]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:20.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:20 smithi114 ceph-mon[36376]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:20.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:20 smithi114 ceph-mon[36376]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-11T19:56:21.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:20 smithi161 ceph-mon[36332]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:21.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:20 smithi161 ceph-mon[36332]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:21.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:20 smithi161 ceph-mon[36332]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-11T19:56:21.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.073+0000 7fe9e5e00640 1 -- 172.21.15.97:0/1715310700 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9e01131e0 con 0x7fe9e01089b0 2024-03-11T19:56:21.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.074+0000 7fe9e4a00640 1 -- 172.21.15.97:0/1715310700 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11401 (secure 0 0 0) 0x7fe9c809a1c0 con 0x7fe9e01089b0 2024-03-11T19:56:21.076 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:56:21.076 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:55:45.149335+0000","modified":"2024-03-11T19:56:19.631638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24583,"mds_2":24547,"mds_3":24545,"mds_4":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24583":{"gid":24583,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.97:6841/2531110572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2531110572},{"type":"v1","addr":"172.21.15.97:6841","nonce":2531110572}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6839/1624532248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":1624532248},{"type":"v1","addr":"172.21.15.114:6839","nonce":1624532248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:56:21.077 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-03-11T19:56:21.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.077+0000 7fe9e5e00640 1 -- 172.21.15.97:0/1715310700 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9c40a1b40 msgr2=0x7fe9c40a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:21.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.077+0000 7fe9e5e00640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9c40a1b40 0x7fe9c40a4000 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe9d002c240 tx=0x7fe9d002c7e0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.077+0000 7fe9e5e00640 1 -- 172.21.15.97:0/1715310700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e01089b0 msgr2=0x7fe9e019dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:21.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.077+0000 7fe9e5e00640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e01089b0 0x7fe9e019dd90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800ea60 tx=0x7fe9c800ef20 comp rx=0 tx=0).stop 2024-03-11T19:56:21.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.077+0000 7fe9e5e00640 1 -- 172.21.15.97:0/1715310700 shutdown_connections 2024-03-11T19:56:21.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.077+0000 7fe9e5e00640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e0109370 0x7fe9e01a4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.077+0000 7fe9e5e00640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9c40a1b40 0x7fe9c40a4000 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.077+0000 7fe9e5e00640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9e01089b0 0x7fe9e019dd90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.077+0000 7fe9e5e00640 1 --2- 172.21.15.97:0/1715310700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9e01077c0 0x7fe9e019d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.077+0000 7fe9e5e00640 1 -- 172.21.15.97:0/1715310700 >> 172.21.15.97:0/1715310700 conn(0x7fe9e0076790 msgr2=0x7fe9e010bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:21.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.077+0000 7fe9e5e00640 1 -- 172.21.15.97:0/1715310700 shutdown_connections 2024-03-11T19:56:21.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.077+0000 7fe9e5e00640 1 -- 172.21.15.97:0/1715310700 wait complete. 2024-03-11T19:56:21.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:56:21.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.307+0000 7fce20400640 1 -- 172.21.15.97:0/388360028 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce181033c0 msgr2=0x7fce181037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:21.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.307+0000 7fce20400640 1 --2- 172.21.15.97:0/388360028 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce181033c0 0x7fce181037e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fce10009ca0 tx=0x7fce1002c100 comp rx=0 tx=0).stop 2024-03-11T19:56:21.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.307+0000 7fce20400640 1 -- 172.21.15.97:0/388360028 shutdown_connections 2024-03-11T19:56:21.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.307+0000 7fce20400640 1 --2- 172.21.15.97:0/388360028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce18104f70 0x7fce1810b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.307+0000 7fce20400640 1 --2- 172.21.15.97:0/388360028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce181045b0 0x7fce18104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.307+0000 7fce20400640 1 --2- 172.21.15.97:0/388360028 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce181033c0 0x7fce181037e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.307+0000 7fce20400640 1 -- 172.21.15.97:0/388360028 >> 172.21.15.97:0/388360028 conn(0x7fce180fe970 msgr2=0x7fce18100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:21.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.307+0000 7fce20400640 1 -- 172.21.15.97:0/388360028 shutdown_connections 2024-03-11T19:56:21.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.307+0000 7fce20400640 1 -- 172.21.15.97:0/388360028 wait complete. 2024-03-11T19:56:21.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.308+0000 7fce20400640 1 Processor -- start 2024-03-11T19:56:21.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.308+0000 7fce20400640 1 -- start start 2024-03-11T19:56:21.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.308+0000 7fce20400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce181033c0 0x7fce18199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:21.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.308+0000 7fce20400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce181045b0 0x7fce181999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:21.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.308+0000 7fce20400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce18104f70 0x7fce181a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:21.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.308+0000 7fce20400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce181a0f70 con 0x7fce18104f70 2024-03-11T19:56:21.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.308+0000 7fce20400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce181a10e0 con 0x7fce181033c0 2024-03-11T19:56:21.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.308+0000 7fce20400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce181a12a0 con 0x7fce181045b0 2024-03-11T19:56:21.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.309+0000 7fce1e400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce18104f70 0x7fce181a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:21.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.309+0000 7fce1e400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce18104f70 0x7fce181a0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59850/0 (socket says 172.21.15.97:59850) 2024-03-11T19:56:21.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.309+0000 7fce1e400640 1 -- 172.21.15.97:0/3282555261 learned_addr learned my addr 172.21.15.97:0/3282555261 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:21.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.309+0000 7fce1d000640 1 --2- 172.21.15.97:0/3282555261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce181045b0 0x7fce181999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:21.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.309+0000 7fce1da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce181033c0 0x7fce18199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:21.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.309+0000 7fce1e400640 1 -- 172.21.15.97:0/3282555261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce181045b0 msgr2=0x7fce181999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:21.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.309+0000 7fce1e400640 1 --2- 172.21.15.97:0/3282555261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce181045b0 0x7fce181999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.309+0000 7fce1e400640 1 -- 172.21.15.97:0/3282555261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce181033c0 msgr2=0x7fce18199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:21.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.309+0000 7fce1e400640 1 --2- 172.21.15.97:0/3282555261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce181033c0 0x7fce18199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.309+0000 7fce1e400640 1 -- 172.21.15.97:0/3282555261 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce10009900 con 0x7fce18104f70 2024-03-11T19:56:21.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.310+0000 7fce1e400640 1 --2- 172.21.15.97:0/3282555261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce18104f70 0x7fce181a0a30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fce0800e970 tx=0x7fce0800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:21.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.310+0000 7fce16000640 1 -- 172.21.15.97:0/3282555261 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08009720 con 0x7fce18104f70 2024-03-11T19:56:21.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.310+0000 7fce16000640 1 -- 172.21.15.97:0/3282555261 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce08005500 con 0x7fce18104f70 2024-03-11T19:56:21.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.310+0000 7fce20400640 1 -- 172.21.15.97:0/3282555261 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce181a1580 con 0x7fce18104f70 2024-03-11T19:56:21.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.310+0000 7fce16000640 1 -- 172.21.15.97:0/3282555261 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08010430 con 0x7fce18104f70 2024-03-11T19:56:21.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.310+0000 7fce20400640 1 -- 172.21.15.97:0/3282555261 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce181a1ae0 con 0x7fce18104f70 2024-03-11T19:56:21.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.311+0000 7fce20400640 1 -- 172.21.15.97:0/3282555261 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1810a6e0 con 0x7fce18104f70 2024-03-11T19:56:21.314 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.313+0000 7fce16000640 1 -- 172.21.15.97:0/3282555261 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fce08004ff0 con 0x7fce18104f70 2024-03-11T19:56:21.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.314+0000 7fce16000640 1 --2- 172.21.15.97:0/3282555261 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcdfc0a1b40 0x7fcdfc0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:21.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.314+0000 7fce16000640 1 -- 172.21.15.97:0/3282555261 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fce08014070 con 0x7fce18104f70 2024-03-11T19:56:21.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.314+0000 7fce1da00640 1 --2- 172.21.15.97:0/3282555261 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcdfc0a1b40 0x7fcdfc0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:21.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.315+0000 7fce1da00640 1 --2- 172.21.15.97:0/3282555261 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcdfc0a1b40 0x7fcdfc0a4000 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fce1002c640 tx=0x7fce10009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:21.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.318+0000 7fce16000640 1 -- 172.21.15.97:0/3282555261 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fce08096900 con 0x7fce18104f70 2024-03-11T19:56:21.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.502+0000 7fce20400640 1 -- 172.21.15.97:0/3282555261 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce1802ce40 con 0x7fce18104f70 2024-03-11T19:56:21.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.505+0000 7fce16000640 1 -- 172.21.15.97:0/3282555261 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24152 (secure 0 0 0) 0x7fce08099fc0 con 0x7fce18104f70 2024-03-11T19:56:21.506 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:56:21.506 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":89,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:55:50.540865+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:56:21.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.507+0000 7fce20400640 1 -- 172.21.15.97:0/3282555261 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcdfc0a1b40 msgr2=0x7fcdfc0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:21.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.507+0000 7fce20400640 1 --2- 172.21.15.97:0/3282555261 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcdfc0a1b40 0x7fcdfc0a4000 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fce1002c640 tx=0x7fce10009800 comp rx=0 tx=0).stop 2024-03-11T19:56:21.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.507+0000 7fce20400640 1 -- 172.21.15.97:0/3282555261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce18104f70 msgr2=0x7fce181a0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:21.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.507+0000 7fce20400640 1 --2- 172.21.15.97:0/3282555261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce18104f70 0x7fce181a0a30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fce0800e970 tx=0x7fce0800ed70 comp rx=0 tx=0).stop 2024-03-11T19:56:21.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.507+0000 7fce20400640 1 -- 172.21.15.97:0/3282555261 shutdown_connections 2024-03-11T19:56:21.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.507+0000 7fce20400640 1 --2- 172.21.15.97:0/3282555261 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcdfc0a1b40 0x7fcdfc0a4000 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.507+0000 7fce20400640 1 --2- 172.21.15.97:0/3282555261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fce18104f70 0x7fce181a0a30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.507+0000 7fce20400640 1 --2- 172.21.15.97:0/3282555261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce181045b0 0x7fce181999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.507+0000 7fce20400640 1 --2- 172.21.15.97:0/3282555261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce181033c0 0x7fce18199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.507+0000 7fce20400640 1 -- 172.21.15.97:0/3282555261 >> 172.21.15.97:0/3282555261 conn(0x7fce180fe970 msgr2=0x7fce181078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:21.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.507+0000 7fce20400640 1 -- 172.21.15.97:0/3282555261 shutdown_connections 2024-03-11T19:56:21.509 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.508+0000 7fce20400640 1 -- 172.21.15.97:0/3282555261 wait complete. 2024-03-11T19:56:21.527 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-11T19:56:21.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-11T19:56:21.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.736+0000 7f12a0600640 1 -- 172.21.15.97:0/4282536819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12981033c0 msgr2=0x7f12981037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:21.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.736+0000 7f12a0600640 1 --2- 172.21.15.97:0/4282536819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12981033c0 0x7f12981037e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f128c009860 tx=0x7f128c02bb70 comp rx=0 tx=0).stop 2024-03-11T19:56:21.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.737+0000 7f12a0600640 1 -- 172.21.15.97:0/4282536819 shutdown_connections 2024-03-11T19:56:21.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.737+0000 7f12a0600640 1 --2- 172.21.15.97:0/4282536819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1298104f70 0x7f129810b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.737+0000 7f12a0600640 1 --2- 172.21.15.97:0/4282536819 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12981045b0 0x7f1298104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.737+0000 7f12a0600640 1 --2- 172.21.15.97:0/4282536819 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12981033c0 0x7f12981037e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.737+0000 7f12a0600640 1 -- 172.21.15.97:0/4282536819 >> 172.21.15.97:0/4282536819 conn(0x7f12980fe970 msgr2=0x7f1298100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:21.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.737+0000 7f12a0600640 1 -- 172.21.15.97:0/4282536819 shutdown_connections 2024-03-11T19:56:21.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.737+0000 7f12a0600640 1 -- 172.21.15.97:0/4282536819 wait complete. 2024-03-11T19:56:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.738+0000 7f12a0600640 1 Processor -- start 2024-03-11T19:56:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.738+0000 7f12a0600640 1 -- start start 2024-03-11T19:56:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.738+0000 7f12a0600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12981033c0 0x7f1298199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.738+0000 7f12a0600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12981045b0 0x7f12981999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.738+0000 7f12a0600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1298104f70 0x7f12981a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.738+0000 7f12a0600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12981a0f70 con 0x7f1298104f70 2024-03-11T19:56:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.738+0000 7f12a0600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12981a10e0 con 0x7f12981033c0 2024-03-11T19:56:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.738+0000 7f12a0600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12981a12a0 con 0x7f12981045b0 2024-03-11T19:56:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.738+0000 7f129e600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1298104f70 0x7f12981a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.738+0000 7f129e600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1298104f70 0x7f12981a0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59878/0 (socket says 172.21.15.97:59878) 2024-03-11T19:56:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.738+0000 7f129e600640 1 -- 172.21.15.97:0/1381584549 learned_addr learned my addr 172.21.15.97:0/1381584549 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.738+0000 7f129d200640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12981045b0 0x7f12981999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:21.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.738+0000 7f129dc00640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12981033c0 0x7f1298199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.739+0000 7f129e600640 1 -- 172.21.15.97:0/1381584549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12981045b0 msgr2=0x7f12981999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.739+0000 7f129e600640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12981045b0 0x7f12981999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.739+0000 7f129e600640 1 -- 172.21.15.97:0/1381584549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12981033c0 msgr2=0x7f1298199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.739+0000 7f129e600640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12981033c0 0x7f1298199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.739+0000 7f129e600640 1 -- 172.21.15.97:0/1381584549 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1284009560 con 0x7f1298104f70 2024-03-11T19:56:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.739+0000 7f129d200640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12981045b0 0x7f12981999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.739+0000 7f129dc00640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12981033c0 0x7f1298199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.739+0000 7f129e600640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1298104f70 0x7f12981a0a30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f129000eb10 tx=0x7f129000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.739+0000 7f1296000640 1 -- 172.21.15.97:0/1381584549 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290004b20 con 0x7f1298104f70 2024-03-11T19:56:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.739+0000 7f12a0600640 1 -- 172.21.15.97:0/1381584549 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128c009490 con 0x7f1298104f70 2024-03-11T19:56:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.739+0000 7f1296000640 1 -- 172.21.15.97:0/1381584549 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1290004c80 con 0x7f1298104f70 2024-03-11T19:56:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.739+0000 7f1296000640 1 -- 172.21.15.97:0/1381584549 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290010670 con 0x7f1298104f70 2024-03-11T19:56:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.740+0000 7f12a0600640 1 -- 172.21.15.97:0/1381584549 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12981a1950 con 0x7f1298104f70 2024-03-11T19:56:21.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.741+0000 7f12a0600640 1 -- 172.21.15.97:0/1381584549 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129810a6e0 con 0x7f1298104f70 2024-03-11T19:56:21.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.743+0000 7f1296000640 1 -- 172.21.15.97:0/1381584549 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1290005020 con 0x7f1298104f70 2024-03-11T19:56:21.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.744+0000 7f1296000640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12800a1ac0 0x7f12800a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:21.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.744+0000 7f1296000640 1 -- 172.21.15.97:0/1381584549 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1290014070 con 0x7f1298104f70 2024-03-11T19:56:21.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.744+0000 7f129dc00640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12800a1ac0 0x7f12800a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:21.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.745+0000 7f129dc00640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12800a1ac0 0x7f12800a3f80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f128c009830 tx=0x7f128c034c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:21.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.748+0000 7f1296000640 1 -- 172.21.15.97:0/1381584549 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1290096b90 con 0x7f1298104f70 2024-03-11T19:56:21.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:21 smithi097 ceph-mon[31898]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:21.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:21 smithi097 ceph-mon[31898]: pgmap v337: 129 pgs: 129 active+clean; 584 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-03-11T19:56:21.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:21 smithi097 ceph-mon[31898]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:21.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:21 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1715310700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:56:21.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:21 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3282555261' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:21.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.940+0000 7f12a0600640 1 -- 172.21.15.97:0/1381584549 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12980615c0 con 0x7f12800a1ac0 2024-03-11T19:56:21.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.955+0000 7f1296000640 1 -- 172.21.15.97:0/1381584549 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340324 (secure 0 0 0) 0x7f12980615c0 con 0x7f12800a1ac0 2024-03-11T19:56:21.961 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:56:21.964 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-11T19:56:21.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.960+0000 7f12a0600640 1 -- 172.21.15.97:0/1381584549 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12800a1ac0 msgr2=0x7f12800a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:21.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.960+0000 7f12a0600640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12800a1ac0 0x7f12800a3f80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f128c009830 tx=0x7f128c034c00 comp rx=0 tx=0).stop 2024-03-11T19:56:21.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.960+0000 7f12a0600640 1 -- 172.21.15.97:0/1381584549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1298104f70 msgr2=0x7f12981a0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:21.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.960+0000 7f12a0600640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1298104f70 0x7f12981a0a30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f129000eb10 tx=0x7f129000c2d0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.960+0000 7f12a0600640 1 -- 172.21.15.97:0/1381584549 shutdown_connections 2024-03-11T19:56:21.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.960+0000 7f12a0600640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12800a1ac0 0x7f12800a3f80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.960+0000 7f12a0600640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1298104f70 0x7f12981a0a30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.960+0000 7f12a0600640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12981045b0 0x7f12981999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.960+0000 7f12a0600640 1 --2- 172.21.15.97:0/1381584549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12981033c0 0x7f1298199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:21.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.960+0000 7f12a0600640 1 -- 172.21.15.97:0/1381584549 >> 172.21.15.97:0/1381584549 conn(0x7f12980fe970 msgr2=0x7f12981078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:21.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.960+0000 7f12a0600640 1 -- 172.21.15.97:0/1381584549 shutdown_connections 2024-03-11T19:56:21.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:21.960+0000 7f12a0600640 1 -- 172.21.15.97:0/1381584549 wait complete. 2024-03-11T19:56:21.981 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-03-11T19:56:20.508518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598352,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":309,"num_write_kb":2982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":252,"ondisk_log_size":252,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383484,"kb_used_data":13512,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558340,"statfs":{"total":1151940427776,"available":1151547740160,"internally_reserved":0,"allocated":13836288,"data_stored":7521051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3344,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006745"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206805+0000","last_change":"2024-03-11T19:55:45.725019+0000","last_active":"2024-03-11T19:55:51.206805+0000","last_peered":"2024-03-11T19:55:51.206805+0000","last_clean":"2024-03-11T19:55:51.206805+0000","last_became_active":"2024-03-11T19:55:45.724602+0000","last_became_peered":"2024-03-11T19:55:45.724602+0000","last_unstale":"2024-03-11T19:55:51.206805+0000","last_undegraded":"2024-03-11T19:55:51.206805+0000","last_fullsized":"2024-03-11T19:55:51.206805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:16:18.218381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206638+0000","last_change":"2024-03-11T19:55:45.724589+0000","last_active":"2024-03-11T19:55:51.206638+0000","last_peered":"2024-03-11T19:55:51.206638+0000","last_clean":"2024-03-11T19:55:51.206638+0000","last_became_active":"2024-03-11T19:55:45.724109+0000","last_became_peered":"2024-03-11T19:55:45.724109+0000","last_unstale":"2024-03-11T19:55:51.206638+0000","last_undegraded":"2024-03-11T19:55:51.206638+0000","last_fullsized":"2024-03-11T19:55:51.206638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:58:12.510437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206347+0000","last_change":"2024-03-11T19:55:45.718831+0000","last_active":"2024-03-11T19:55:51.206347+0000","last_peered":"2024-03-11T19:55:51.206347+0000","last_clean":"2024-03-11T19:55:51.206347+0000","last_became_active":"2024-03-11T19:55:45.717652+0000","last_became_peered":"2024-03-11T19:55:45.717652+0000","last_unstale":"2024-03-11T19:55:51.206347+0000","last_undegraded":"2024-03-11T19:55:51.206347+0000","last_fullsized":"2024-03-11T19:55:51.206347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:16.471754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205658+0000","last_change":"2024-03-11T19:55:45.701640+0000","last_active":"2024-03-11T19:55:51.205658+0000","last_peered":"2024-03-11T19:55:51.205658+0000","last_clean":"2024-03-11T19:55:51.205658+0000","last_became_active":"2024-03-11T19:55:45.699207+0000","last_became_peered":"2024-03-11T19:55:45.699207+0000","last_unstale":"2024-03-11T19:55:51.205658+0000","last_undegraded":"2024-03-11T19:55:51.205658+0000","last_fullsized":"2024-03-11T19:55:51.205658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:01.900429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205411+0000","last_change":"2024-03-11T19:55:45.732040+0000","last_active":"2024-03-11T19:55:51.205411+0000","last_peered":"2024-03-11T19:55:51.205411+0000","last_clean":"2024-03-11T19:55:51.205411+0000","last_became_active":"2024-03-11T19:55:45.731647+0000","last_became_peered":"2024-03-11T19:55:45.731647+0000","last_unstale":"2024-03-11T19:55:51.205411+0000","last_undegraded":"2024-03-11T19:55:51.205411+0000","last_fullsized":"2024-03-11T19:55:51.205411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:51:57.345329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094385+0000","last_change":"2024-03-11T19:55:45.707986+0000","last_active":"2024-03-11T19:55:51.094385+0000","last_peered":"2024-03-11T19:55:51.094385+0000","last_clean":"2024-03-11T19:55:51.094385+0000","last_became_active":"2024-03-11T19:55:45.707641+0000","last_became_peered":"2024-03-11T19:55:45.707641+0000","last_unstale":"2024-03-11T19:55:51.094385+0000","last_undegraded":"2024-03-11T19:55:51.094385+0000","last_fullsized":"2024-03-11T19:55:51.094385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:15:50.635234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206161+0000","last_change":"2024-03-11T19:55:45.713756+0000","last_active":"2024-03-11T19:55:51.206161+0000","last_peered":"2024-03-11T19:55:51.206161+0000","last_clean":"2024-03-11T19:55:51.206161+0000","last_became_active":"2024-03-11T19:55:45.713333+0000","last_became_peered":"2024-03-11T19:55:45.713333+0000","last_unstale":"2024-03-11T19:55:51.206161+0000","last_undegraded":"2024-03-11T19:55:51.206161+0000","last_fullsized":"2024-03-11T19:55:51.206161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:49:58.048921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205952+0000","last_change":"2024-03-11T19:55:45.729587+0000","last_active":"2024-03-11T19:55:51.205952+0000","last_peered":"2024-03-11T19:55:51.205952+0000","last_clean":"2024-03-11T19:55:51.205952+0000","last_became_active":"2024-03-11T19:55:45.729037+0000","last_became_peered":"2024-03-11T19:55:45.729037+0000","last_unstale":"2024-03-11T19:55:51.205952+0000","last_undegraded":"2024-03-11T19:55:51.205952+0000","last_fullsized":"2024-03-11T19:55:51.205952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:55:43.445192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205943+0000","last_change":"2024-03-11T19:55:45.709793+0000","last_active":"2024-03-11T19:55:51.205943+0000","last_peered":"2024-03-11T19:55:51.205943+0000","last_clean":"2024-03-11T19:55:51.205943+0000","last_became_active":"2024-03-11T19:55:45.708631+0000","last_became_peered":"2024-03-11T19:55:45.708631+0000","last_unstale":"2024-03-11T19:55:51.205943+0000","last_undegraded":"2024-03-11T19:55:51.205943+0000","last_fullsized":"2024-03-11T19:55:51.205943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:02.144066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094886+0000","last_change":"2024-03-11T19:55:45.704227+0000","last_active":"2024-03-11T19:55:51.094886+0000","last_peered":"2024-03-11T19:55:51.094886+0000","last_clean":"2024-03-11T19:55:51.094886+0000","last_became_active":"2024-03-11T19:55:45.703885+0000","last_became_peered":"2024-03-11T19:55:45.703885+0000","last_unstale":"2024-03-11T19:55:51.094886+0000","last_undegraded":"2024-03-11T19:55:51.094886+0000","last_fullsized":"2024-03-11T19:55:51.094886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:36:21.401988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206822+0000","last_change":"2024-03-11T19:55:45.708950+0000","last_active":"2024-03-11T19:55:51.206822+0000","last_peered":"2024-03-11T19:55:51.206822+0000","last_clean":"2024-03-11T19:55:51.206822+0000","last_became_active":"2024-03-11T19:55:45.708544+0000","last_became_peered":"2024-03-11T19:55:45.708544+0000","last_unstale":"2024-03-11T19:55:51.206822+0000","last_undegraded":"2024-03-11T19:55:51.206822+0000","last_fullsized":"2024-03-11T19:55:51.206822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:06:34.244449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205548+0000","last_change":"2024-03-11T19:55:45.703923+0000","last_active":"2024-03-11T19:55:51.205548+0000","last_peered":"2024-03-11T19:55:51.205548+0000","last_clean":"2024-03-11T19:55:51.205548+0000","last_became_active":"2024-03-11T19:55:45.703595+0000","last_became_peered":"2024-03-11T19:55:45.703595+0000","last_unstale":"2024-03-11T19:55:51.205548+0000","last_undegraded":"2024-03-11T19:55:51.205548+0000","last_fullsized":"2024-03-11T19:55:51.205548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:00:19.683907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.098569+0000","last_change":"2024-03-11T19:55:45.707499+0000","last_active":"2024-03-11T19:55:51.098569+0000","last_peered":"2024-03-11T19:55:51.098569+0000","last_clean":"2024-03-11T19:55:51.098569+0000","last_became_active":"2024-03-11T19:55:45.706344+0000","last_became_peered":"2024-03-11T19:55:45.706344+0000","last_unstale":"2024-03-11T19:55:51.098569+0000","last_undegraded":"2024-03-11T19:55:51.098569+0000","last_fullsized":"2024-03-11T19:55:51.098569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:38:24.082332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205979+0000","last_change":"2024-03-11T19:55:45.722182+0000","last_active":"2024-03-11T19:55:51.205979+0000","last_peered":"2024-03-11T19:55:51.205979+0000","last_clean":"2024-03-11T19:55:51.205979+0000","last_became_active":"2024-03-11T19:55:45.721783+0000","last_became_peered":"2024-03-11T19:55:45.721783+0000","last_unstale":"2024-03-11T19:55:51.205979+0000","last_undegraded":"2024-03-11T19:55:51.205979+0000","last_fullsized":"2024-03-11T19:55:51.205979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:41:08.559002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205879+0000","last_change":"2024-03-11T19:55:45.718881+0000","last_active":"2024-03-11T19:55:51.205879+0000","last_peered":"2024-03-11T19:55:51.205879+0000","last_clean":"2024-03-11T19:55:51.205879+0000","last_became_active":"2024-03-11T19:55:45.718458+0000","last_became_peered":"2024-03-11T19:55:45.718458+0000","last_unstale":"2024-03-11T19:55:51.205879+0000","last_undegraded":"2024-03-11T19:55:51.205879+0000","last_fullsized":"2024-03-11T19:55:51.205879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:22.075892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.204711+0000","last_change":"2024-03-11T19:55:45.726163+0000","last_active":"2024-03-11T19:55:51.204711+0000","last_peered":"2024-03-11T19:55:51.204711+0000","last_clean":"2024-03-11T19:55:51.204711+0000","last_became_active":"2024-03-11T19:55:45.725772+0000","last_became_peered":"2024-03-11T19:55:45.725772+0000","last_unstale":"2024-03-11T19:55:51.204711+0000","last_undegraded":"2024-03-11T19:55:51.204711+0000","last_fullsized":"2024-03-11T19:55:51.204711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:36:19.057920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206587+0000","last_change":"2024-03-11T19:55:45.724787+0000","last_active":"2024-03-11T19:55:51.206587+0000","last_peered":"2024-03-11T19:55:51.206587+0000","last_clean":"2024-03-11T19:55:51.206587+0000","last_became_active":"2024-03-11T19:55:45.724437+0000","last_became_peered":"2024-03-11T19:55:45.724437+0000","last_unstale":"2024-03-11T19:55:51.206587+0000","last_undegraded":"2024-03-11T19:55:51.206587+0000","last_fullsized":"2024-03-11T19:55:51.206587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:55:53.811679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.210628+0000","last_change":"2024-03-11T19:55:45.714934+0000","last_active":"2024-03-11T19:55:51.210628+0000","last_peered":"2024-03-11T19:55:51.210628+0000","last_clean":"2024-03-11T19:55:51.210628+0000","last_became_active":"2024-03-11T19:55:45.709595+0000","last_became_peered":"2024-03-11T19:55:45.709595+0000","last_unstale":"2024-03-11T19:55:51.210628+0000","last_undegraded":"2024-03-11T19:55:51.210628+0000","last_fullsized":"2024-03-11T19:55:51.210628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:48.319073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094743+0000","last_change":"2024-03-11T19:55:45.725747+0000","last_active":"2024-03-11T19:55:51.094743+0000","last_peered":"2024-03-11T19:55:51.094743+0000","last_clean":"2024-03-11T19:55:51.094743+0000","last_became_active":"2024-03-11T19:55:45.725239+0000","last_became_peered":"2024-03-11T19:55:45.725239+0000","last_unstale":"2024-03-11T19:55:51.094743+0000","last_undegraded":"2024-03-11T19:55:51.094743+0000","last_fullsized":"2024-03-11T19:55:51.094743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:51:24.046681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094568+0000","last_change":"2024-03-11T19:55:50.109913+0000","last_active":"2024-03-11T19:55:51.094568+0000","last_peered":"2024-03-11T19:55:51.094568+0000","last_clean":"2024-03-11T19:55:51.094568+0000","last_became_active":"2024-03-11T19:55:50.109471+0000","last_became_peered":"2024-03-11T19:55:50.109471+0000","last_unstale":"2024-03-11T19:55:51.094568+0000","last_undegraded":"2024-03-11T19:55:51.094568+0000","last_fullsized":"2024-03-11T19:55:51.094568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:40.258022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.211124+0000","last_change":"2024-03-11T19:55:44.680181+0000","last_active":"2024-03-11T19:55:51.211124+0000","last_peered":"2024-03-11T19:55:51.211124+0000","last_clean":"2024-03-11T19:55:51.211124+0000","last_became_active":"2024-03-11T19:55:44.679807+0000","last_became_peered":"2024-03-11T19:55:44.679807+0000","last_unstale":"2024-03-11T19:55:51.211124+0000","last_undegraded":"2024-03-11T19:55:51.211124+0000","last_fullsized":"2024-03-11T19:55:51.211124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:23:04.532836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.210078+0000","last_change":"2024-03-11T19:55:45.723547+0000","last_active":"2024-03-11T19:55:51.210078+0000","last_peered":"2024-03-11T19:55:51.210078+0000","last_clean":"2024-03-11T19:55:51.210078+0000","last_became_active":"2024-03-11T19:55:45.722516+0000","last_became_peered":"2024-03-11T19:55:45.722516+0000","last_unstale":"2024-03-11T19:55:51.210078+0000","last_undegraded":"2024-03-11T19:55:51.210078+0000","last_fullsized":"2024-03-11T19:55:51.210078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:40.271814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205127+0000","last_change":"2024-03-11T19:55:50.102141+0000","last_active":"2024-03-11T19:55:51.205127+0000","last_peered":"2024-03-11T19:55:51.205127+0000","last_clean":"2024-03-11T19:55:51.205127+0000","last_became_active":"2024-03-11T19:55:50.101667+0000","last_became_peered":"2024-03-11T19:55:50.101667+0000","last_unstale":"2024-03-11T19:55:51.205127+0000","last_undegraded":"2024-03-11T19:55:51.205127+0000","last_fullsized":"2024-03-11T19:55:51.205127+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:54:11.453059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.210399+0000","last_change":"2024-03-11T19:55:44.684623+0000","last_active":"2024-03-11T19:55:51.210399+0000","last_peered":"2024-03-11T19:55:51.210399+0000","last_clean":"2024-03-11T19:55:51.210399+0000","last_became_active":"2024-03-11T19:55:44.683611+0000","last_became_peered":"2024-03-11T19:55:44.683611+0000","last_unstale":"2024-03-11T19:55:51.210399+0000","last_undegraded":"2024-03-11T19:55:51.210399+0000","last_fullsized":"2024-03-11T19:55:51.210399+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:22:50.954364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206424+0000","last_change":"2024-03-11T19:55:45.723619+0000","last_active":"2024-03-11T19:55:51.206424+0000","last_peered":"2024-03-11T19:55:51.206424+0000","last_clean":"2024-03-11T19:55:51.206424+0000","last_became_active":"2024-03-11T19:55:45.722685+0000","last_became_peered":"2024-03-11T19:55:45.722685+0000","last_unstale":"2024-03-11T19:55:51.206424+0000","last_undegraded":"2024-03-11T19:55:51.206424+0000","last_fullsized":"2024-03-11T19:55:51.206424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:47:13.747776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205203+0000","last_change":"2024-03-11T19:55:50.102898+0000","last_active":"2024-03-11T19:55:51.205203+0000","last_peered":"2024-03-11T19:55:51.205203+0000","last_clean":"2024-03-11T19:55:51.205203+0000","last_became_active":"2024-03-11T19:55:50.102491+0000","last_became_peered":"2024-03-11T19:55:50.102491+0000","last_unstale":"2024-03-11T19:55:51.205203+0000","last_undegraded":"2024-03-11T19:55:51.205203+0000","last_fullsized":"2024-03-11T19:55:51.205203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:41:25.389542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:17.661009+0000","last_change":"2024-03-11T19:55:44.673547+0000","last_active":"2024-03-11T19:56:17.661009+0000","last_peered":"2024-03-11T19:56:17.661009+0000","last_clean":"2024-03-11T19:56:17.661009+0000","last_became_active":"2024-03-11T19:55:44.673211+0000","last_became_peered":"2024-03-11T19:55:44.673211+0000","last_unstale":"2024-03-11T19:56:17.661009+0000","last_undegraded":"2024-03-11T19:56:17.661009+0000","last_fullsized":"2024-03-11T19:56:17.661009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:38:09.736615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205415+0000","last_change":"2024-03-11T19:55:45.715157+0000","last_active":"2024-03-11T19:55:51.205415+0000","last_peered":"2024-03-11T19:55:51.205415+0000","last_clean":"2024-03-11T19:55:51.205415+0000","last_became_active":"2024-03-11T19:55:45.714536+0000","last_became_peered":"2024-03-11T19:55:45.714536+0000","last_unstale":"2024-03-11T19:55:51.205415+0000","last_undegraded":"2024-03-11T19:55:51.205415+0000","last_fullsized":"2024-03-11T19:55:51.205415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:02.416736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.204978+0000","last_change":"2024-03-11T19:55:50.102157+0000","last_active":"2024-03-11T19:55:51.204978+0000","last_peered":"2024-03-11T19:55:51.204978+0000","last_clean":"2024-03-11T19:55:51.204978+0000","last_became_active":"2024-03-11T19:55:50.101715+0000","last_became_peered":"2024-03-11T19:55:50.101715+0000","last_unstale":"2024-03-11T19:55:51.204978+0000","last_undegraded":"2024-03-11T19:55:51.204978+0000","last_fullsized":"2024-03-11T19:55:51.204978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:12:18.780708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206402+0000","last_change":"2024-03-11T19:55:44.669876+0000","last_active":"2024-03-11T19:55:51.206402+0000","last_peered":"2024-03-11T19:55:51.206402+0000","last_clean":"2024-03-11T19:55:51.206402+0000","last_became_active":"2024-03-11T19:55:44.669539+0000","last_became_peered":"2024-03-11T19:55:44.669539+0000","last_unstale":"2024-03-11T19:55:51.206402+0000","last_undegraded":"2024-03-11T19:55:51.206402+0000","last_fullsized":"2024-03-11T19:55:51.206402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:29:45.676685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205300+0000","last_change":"2024-03-11T19:55:45.709973+0000","last_active":"2024-03-11T19:55:51.205300+0000","last_peered":"2024-03-11T19:55:51.205300+0000","last_clean":"2024-03-11T19:55:51.205300+0000","last_became_active":"2024-03-11T19:55:45.709645+0000","last_became_peered":"2024-03-11T19:55:45.709645+0000","last_unstale":"2024-03-11T19:55:51.205300+0000","last_undegraded":"2024-03-11T19:55:51.205300+0000","last_fullsized":"2024-03-11T19:55:51.205300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:16.516982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094915+0000","last_change":"2024-03-11T19:55:50.094093+0000","last_active":"2024-03-11T19:55:51.094915+0000","last_peered":"2024-03-11T19:55:51.094915+0000","last_clean":"2024-03-11T19:55:51.094915+0000","last_became_active":"2024-03-11T19:55:50.093653+0000","last_became_peered":"2024-03-11T19:55:50.093653+0000","last_unstale":"2024-03-11T19:55:51.094915+0000","last_undegraded":"2024-03-11T19:55:51.094915+0000","last_fullsized":"2024-03-11T19:55:51.094915+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:28:41.668001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.644996+0000","last_change":"2024-03-11T19:55:44.671547+0000","last_active":"2024-03-11T19:56:15.644996+0000","last_peered":"2024-03-11T19:56:15.644996+0000","last_clean":"2024-03-11T19:56:15.644996+0000","last_became_active":"2024-03-11T19:55:44.671211+0000","last_became_peered":"2024-03-11T19:55:44.671211+0000","last_unstale":"2024-03-11T19:56:15.644996+0000","last_undegraded":"2024-03-11T19:56:15.644996+0000","last_fullsized":"2024-03-11T19:56:15.644996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:18:46.261250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206035+0000","last_change":"2024-03-11T19:55:45.707968+0000","last_active":"2024-03-11T19:55:51.206035+0000","last_peered":"2024-03-11T19:55:51.206035+0000","last_clean":"2024-03-11T19:55:51.206035+0000","last_became_active":"2024-03-11T19:55:45.706998+0000","last_became_peered":"2024-03-11T19:55:45.706998+0000","last_unstale":"2024-03-11T19:55:51.206035+0000","last_undegraded":"2024-03-11T19:55:51.206035+0000","last_fullsized":"2024-03-11T19:55:51.206035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:49:35.711123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094598+0000","last_change":"2024-03-11T19:55:50.104310+0000","last_active":"2024-03-11T19:55:51.094598+0000","last_peered":"2024-03-11T19:55:51.094598+0000","last_clean":"2024-03-11T19:55:51.094598+0000","last_became_active":"2024-03-11T19:55:50.103831+0000","last_became_peered":"2024-03-11T19:55:50.103831+0000","last_unstale":"2024-03-11T19:55:51.094598+0000","last_undegraded":"2024-03-11T19:55:51.094598+0000","last_fullsized":"2024-03-11T19:55:51.094598+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:33.213987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:16.658467+0000","last_change":"2024-03-11T19:55:44.677615+0000","last_active":"2024-03-11T19:56:16.658467+0000","last_peered":"2024-03-11T19:56:16.658467+0000","last_clean":"2024-03-11T19:56:16.658467+0000","last_became_active":"2024-03-11T19:55:44.677153+0000","last_became_peered":"2024-03-11T19:55:44.677153+0000","last_unstale":"2024-03-11T19:56:16.658467+0000","last_undegraded":"2024-03-11T19:56:16.658467+0000","last_fullsized":"2024-03-11T19:56:16.658467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:52:59.153222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205117+0000","last_change":"2024-03-11T19:55:45.731827+0000","last_active":"2024-03-11T19:55:51.205117+0000","last_peered":"2024-03-11T19:55:51.205117+0000","last_clean":"2024-03-11T19:55:51.205117+0000","last_became_active":"2024-03-11T19:55:45.731460+0000","last_became_peered":"2024-03-11T19:55:45.731460+0000","last_unstale":"2024-03-11T19:55:51.205117+0000","last_undegraded":"2024-03-11T19:55:51.205117+0000","last_fullsized":"2024-03-11T19:55:51.205117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:12.277198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206703+0000","last_change":"2024-03-11T19:55:50.109812+0000","last_active":"2024-03-11T19:55:51.206703+0000","last_peered":"2024-03-11T19:55:51.206703+0000","last_clean":"2024-03-11T19:55:51.206703+0000","last_became_active":"2024-03-11T19:55:50.109420+0000","last_became_peered":"2024-03-11T19:55:50.109420+0000","last_unstale":"2024-03-11T19:55:51.206703+0000","last_undegraded":"2024-03-11T19:55:51.206703+0000","last_fullsized":"2024-03-11T19:55:51.206703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:08:30.105941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.098100+0000","last_change":"2024-03-11T19:55:44.683173+0000","last_active":"2024-03-11T19:55:51.098100+0000","last_peered":"2024-03-11T19:55:51.098100+0000","last_clean":"2024-03-11T19:55:51.098100+0000","last_became_active":"2024-03-11T19:55:44.682778+0000","last_became_peered":"2024-03-11T19:55:44.682778+0000","last_unstale":"2024-03-11T19:55:51.098100+0000","last_undegraded":"2024-03-11T19:55:51.098100+0000","last_fullsized":"2024-03-11T19:55:51.098100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:15.119864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206856+0000","last_change":"2024-03-11T19:55:45.721833+0000","last_active":"2024-03-11T19:55:51.206856+0000","last_peered":"2024-03-11T19:55:51.206856+0000","last_clean":"2024-03-11T19:55:51.206856+0000","last_became_active":"2024-03-11T19:55:45.721490+0000","last_became_peered":"2024-03-11T19:55:45.721490+0000","last_unstale":"2024-03-11T19:55:51.206856+0000","last_undegraded":"2024-03-11T19:55:51.206856+0000","last_fullsized":"2024-03-11T19:55:51.206856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:09.824912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206517+0000","last_change":"2024-03-11T19:55:50.108424+0000","last_active":"2024-03-11T19:55:51.206517+0000","last_peered":"2024-03-11T19:55:51.206517+0000","last_clean":"2024-03-11T19:55:51.206517+0000","last_became_active":"2024-03-11T19:55:50.107072+0000","last_became_peered":"2024-03-11T19:55:50.107072+0000","last_unstale":"2024-03-11T19:55:51.206517+0000","last_undegraded":"2024-03-11T19:55:51.206517+0000","last_fullsized":"2024-03-11T19:55:51.206517+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:31.751597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.204613+0000","last_change":"2024-03-11T19:55:44.671570+0000","last_active":"2024-03-11T19:55:51.204613+0000","last_peered":"2024-03-11T19:55:51.204613+0000","last_clean":"2024-03-11T19:55:51.204613+0000","last_became_active":"2024-03-11T19:55:44.671007+0000","last_became_peered":"2024-03-11T19:55:44.671007+0000","last_unstale":"2024-03-11T19:55:51.204613+0000","last_undegraded":"2024-03-11T19:55:51.204613+0000","last_fullsized":"2024-03-11T19:55:51.204613+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:23.865011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.207329+0000","last_change":"2024-03-11T19:55:45.733048+0000","last_active":"2024-03-11T19:55:51.207329+0000","last_peered":"2024-03-11T19:55:51.207329+0000","last_clean":"2024-03-11T19:55:51.207329+0000","last_became_active":"2024-03-11T19:55:45.731656+0000","last_became_peered":"2024-03-11T19:55:45.731656+0000","last_unstale":"2024-03-11T19:55:51.207329+0000","last_undegraded":"2024-03-11T19:55:51.207329+0000","last_fullsized":"2024-03-11T19:55:51.207329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:28:56.433327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205928+0000","last_change":"2024-03-11T19:55:50.105532+0000","last_active":"2024-03-11T19:55:51.205928+0000","last_peered":"2024-03-11T19:55:51.205928+0000","last_clean":"2024-03-11T19:55:51.205928+0000","last_became_active":"2024-03-11T19:55:50.102200+0000","last_became_peered":"2024-03-11T19:55:50.102200+0000","last_unstale":"2024-03-11T19:55:51.205928+0000","last_undegraded":"2024-03-11T19:55:51.205928+0000","last_fullsized":"2024-03-11T19:55:51.205928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:37.060472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.645329+0000","last_change":"2024-03-11T19:55:44.678224+0000","last_active":"2024-03-11T19:56:15.645329+0000","last_peered":"2024-03-11T19:56:15.645329+0000","last_clean":"2024-03-11T19:56:15.645329+0000","last_became_active":"2024-03-11T19:55:44.677936+0000","last_became_peered":"2024-03-11T19:55:44.677936+0000","last_unstale":"2024-03-11T19:56:15.645329+0000","last_undegraded":"2024-03-11T19:56:15.645329+0000","last_fullsized":"2024-03-11T19:56:15.645329+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:20:42.486501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205578+0000","last_change":"2024-03-11T19:55:45.709917+0000","last_active":"2024-03-11T19:55:51.205578+0000","last_peered":"2024-03-11T19:55:51.205578+0000","last_clean":"2024-03-11T19:55:51.205578+0000","last_became_active":"2024-03-11T19:55:45.709318+0000","last_became_peered":"2024-03-11T19:55:45.709318+0000","last_unstale":"2024-03-11T19:55:51.205578+0000","last_undegraded":"2024-03-11T19:55:51.205578+0000","last_fullsized":"2024-03-11T19:55:51.205578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:08.298158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206938+0000","last_change":"2024-03-11T19:55:50.103384+0000","last_active":"2024-03-11T19:55:51.206938+0000","last_peered":"2024-03-11T19:55:51.206938+0000","last_clean":"2024-03-11T19:55:51.206938+0000","last_became_active":"2024-03-11T19:55:50.100641+0000","last_became_peered":"2024-03-11T19:55:50.100641+0000","last_unstale":"2024-03-11T19:55:51.206938+0000","last_undegraded":"2024-03-11T19:55:51.206938+0000","last_fullsized":"2024-03-11T19:55:51.206938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:58.524369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.643034+0000","last_change":"2024-03-11T19:55:44.676004+0000","last_active":"2024-03-11T19:56:15.643034+0000","last_peered":"2024-03-11T19:56:15.643034+0000","last_clean":"2024-03-11T19:56:15.643034+0000","last_became_active":"2024-03-11T19:55:44.675640+0000","last_became_peered":"2024-03-11T19:55:44.675640+0000","last_unstale":"2024-03-11T19:56:15.643034+0000","last_undegraded":"2024-03-11T19:56:15.643034+0000","last_fullsized":"2024-03-11T19:56:15.643034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:15.096780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.207155+0000","last_change":"2024-03-11T19:55:45.723937+0000","last_active":"2024-03-11T19:55:51.207155+0000","last_peered":"2024-03-11T19:55:51.207155+0000","last_clean":"2024-03-11T19:55:51.207155+0000","last_became_active":"2024-03-11T19:55:45.723301+0000","last_became_peered":"2024-03-11T19:55:45.723301+0000","last_unstale":"2024-03-11T19:55:51.207155+0000","last_undegraded":"2024-03-11T19:55:51.207155+0000","last_fullsized":"2024-03-11T19:55:51.207155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:10:35.821109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205522+0000","last_change":"2024-03-11T19:55:50.096136+0000","last_active":"2024-03-11T19:55:51.205522+0000","last_peered":"2024-03-11T19:55:51.205522+0000","last_clean":"2024-03-11T19:55:51.205522+0000","last_became_active":"2024-03-11T19:55:50.095670+0000","last_became_peered":"2024-03-11T19:55:50.095670+0000","last_unstale":"2024-03-11T19:55:51.205522+0000","last_undegraded":"2024-03-11T19:55:51.205522+0000","last_fullsized":"2024-03-11T19:55:51.205522+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:40.943273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205177+0000","last_change":"2024-03-11T19:55:44.681547+0000","last_active":"2024-03-11T19:55:51.205177+0000","last_peered":"2024-03-11T19:55:51.205177+0000","last_clean":"2024-03-11T19:55:51.205177+0000","last_became_active":"2024-03-11T19:55:44.681237+0000","last_became_peered":"2024-03-11T19:55:44.681237+0000","last_unstale":"2024-03-11T19:55:51.205177+0000","last_undegraded":"2024-03-11T19:55:51.205177+0000","last_fullsized":"2024-03-11T19:55:51.205177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:08:09.509915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205616+0000","last_change":"2024-03-11T19:55:45.705025+0000","last_active":"2024-03-11T19:55:51.205616+0000","last_peered":"2024-03-11T19:55:51.205616+0000","last_clean":"2024-03-11T19:55:51.205616+0000","last_became_active":"2024-03-11T19:55:45.702680+0000","last_became_peered":"2024-03-11T19:55:45.702680+0000","last_unstale":"2024-03-11T19:55:51.205616+0000","last_undegraded":"2024-03-11T19:55:51.205616+0000","last_fullsized":"2024-03-11T19:55:51.205616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:28.245002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.207889+0000","last_change":"2024-03-11T19:55:50.099648+0000","last_active":"2024-03-11T19:55:51.207889+0000","last_peered":"2024-03-11T19:55:51.207889+0000","last_clean":"2024-03-11T19:55:51.207889+0000","last_became_active":"2024-03-11T19:55:50.098516+0000","last_became_peered":"2024-03-11T19:55:50.098516+0000","last_unstale":"2024-03-11T19:55:51.207889+0000","last_undegraded":"2024-03-11T19:55:51.207889+0000","last_fullsized":"2024-03-11T19:55:51.207889+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:19.644046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.643538+0000","last_change":"2024-03-11T19:55:44.672460+0000","last_active":"2024-03-11T19:56:15.643538+0000","last_peered":"2024-03-11T19:56:15.643538+0000","last_clean":"2024-03-11T19:56:15.643538+0000","last_became_active":"2024-03-11T19:55:44.672123+0000","last_became_peered":"2024-03-11T19:55:44.672123+0000","last_unstale":"2024-03-11T19:56:15.643538+0000","last_undegraded":"2024-03-11T19:56:15.643538+0000","last_fullsized":"2024-03-11T19:56:15.643538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:43.940638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205933+0000","last_change":"2024-03-11T19:55:45.702933+0000","last_active":"2024-03-11T19:55:51.205933+0000","last_peered":"2024-03-11T19:55:51.205933+0000","last_clean":"2024-03-11T19:55:51.205933+0000","last_became_active":"2024-03-11T19:55:45.702585+0000","last_became_peered":"2024-03-11T19:55:45.702585+0000","last_unstale":"2024-03-11T19:55:51.205933+0000","last_undegraded":"2024-03-11T19:55:51.205933+0000","last_fullsized":"2024-03-11T19:55:51.205933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:54:38.762156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205840+0000","last_change":"2024-03-11T19:55:50.102571+0000","last_active":"2024-03-11T19:55:51.205840+0000","last_peered":"2024-03-11T19:55:51.205840+0000","last_clean":"2024-03-11T19:55:51.205840+0000","last_became_active":"2024-03-11T19:55:50.102099+0000","last_became_peered":"2024-03-11T19:55:50.102099+0000","last_unstale":"2024-03-11T19:55:51.205840+0000","last_undegraded":"2024-03-11T19:55:51.205840+0000","last_fullsized":"2024-03-11T19:55:51.205840+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:17.631378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:16.655904+0000","last_change":"2024-03-11T19:55:44.671027+0000","last_active":"2024-03-11T19:56:16.655904+0000","last_peered":"2024-03-11T19:56:16.655904+0000","last_clean":"2024-03-11T19:56:16.655904+0000","last_became_active":"2024-03-11T19:55:44.669696+0000","last_became_peered":"2024-03-11T19:55:44.669696+0000","last_unstale":"2024-03-11T19:56:16.655904+0000","last_undegraded":"2024-03-11T19:56:16.655904+0000","last_fullsized":"2024-03-11T19:56:16.655904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:48:20.169638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.098501+0000","last_change":"2024-03-11T19:55:45.729119+0000","last_active":"2024-03-11T19:55:51.098501+0000","last_peered":"2024-03-11T19:55:51.098501+0000","last_clean":"2024-03-11T19:55:51.098501+0000","last_became_active":"2024-03-11T19:55:45.727442+0000","last_became_peered":"2024-03-11T19:55:45.727442+0000","last_unstale":"2024-03-11T19:55:51.098501+0000","last_undegraded":"2024-03-11T19:55:51.098501+0000","last_fullsized":"2024-03-11T19:55:51.098501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:14:51.483199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205678+0000","last_change":"2024-03-11T19:55:50.104326+0000","last_active":"2024-03-11T19:55:51.205678+0000","last_peered":"2024-03-11T19:55:51.205678+0000","last_clean":"2024-03-11T19:55:51.205678+0000","last_became_active":"2024-03-11T19:55:50.103881+0000","last_became_peered":"2024-03-11T19:55:50.103881+0000","last_unstale":"2024-03-11T19:55:51.205678+0000","last_undegraded":"2024-03-11T19:55:51.205678+0000","last_fullsized":"2024-03-11T19:55:51.205678+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:12:38.819591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.648465+0000","last_change":"2024-03-11T19:55:44.684900+0000","last_active":"2024-03-11T19:56:15.648465+0000","last_peered":"2024-03-11T19:56:15.648465+0000","last_clean":"2024-03-11T19:56:15.648465+0000","last_became_active":"2024-03-11T19:55:44.684590+0000","last_became_peered":"2024-03-11T19:55:44.684590+0000","last_unstale":"2024-03-11T19:56:15.648465+0000","last_undegraded":"2024-03-11T19:56:15.648465+0000","last_fullsized":"2024-03-11T19:56:15.648465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:07.744366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206521+0000","last_change":"2024-03-11T19:55:45.704787+0000","last_active":"2024-03-11T19:55:51.206521+0000","last_peered":"2024-03-11T19:55:51.206521+0000","last_clean":"2024-03-11T19:55:51.206521+0000","last_became_active":"2024-03-11T19:55:45.704405+0000","last_became_peered":"2024-03-11T19:55:45.704405+0000","last_unstale":"2024-03-11T19:55:51.206521+0000","last_undegraded":"2024-03-11T19:55:51.206521+0000","last_fullsized":"2024-03-11T19:55:51.206521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:58:05.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.211359+0000","last_change":"2024-03-11T19:55:50.104793+0000","last_active":"2024-03-11T19:55:51.211359+0000","last_peered":"2024-03-11T19:55:51.211359+0000","last_clean":"2024-03-11T19:55:51.211359+0000","last_became_active":"2024-03-11T19:55:50.104415+0000","last_became_peered":"2024-03-11T19:55:50.104415+0000","last_unstale":"2024-03-11T19:55:51.211359+0000","last_undegraded":"2024-03-11T19:55:51.211359+0000","last_fullsized":"2024-03-11T19:55:51.211359+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:53.307313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:16.652559+0000","last_change":"2024-03-11T19:55:44.669403+0000","last_active":"2024-03-11T19:56:16.652559+0000","last_peered":"2024-03-11T19:56:16.652559+0000","last_clean":"2024-03-11T19:56:16.652559+0000","last_became_active":"2024-03-11T19:55:44.668908+0000","last_became_peered":"2024-03-11T19:55:44.668908+0000","last_unstale":"2024-03-11T19:56:16.652559+0000","last_undegraded":"2024-03-11T19:56:16.652559+0000","last_fullsized":"2024-03-11T19:56:16.652559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:56:36.831506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205604+0000","last_change":"2024-03-11T19:55:45.700029+0000","last_active":"2024-03-11T19:55:51.205604+0000","last_peered":"2024-03-11T19:55:51.205604+0000","last_clean":"2024-03-11T19:55:51.205604+0000","last_became_active":"2024-03-11T19:55:45.697296+0000","last_became_peered":"2024-03-11T19:55:45.697296+0000","last_unstale":"2024-03-11T19:55:51.205604+0000","last_undegraded":"2024-03-11T19:55:51.205604+0000","last_fullsized":"2024-03-11T19:55:51.205604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:59:35.025845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.100065+0000","last_change":"2024-03-11T19:55:50.108645+0000","last_active":"2024-03-11T19:55:51.100065+0000","last_peered":"2024-03-11T19:55:51.100065+0000","last_clean":"2024-03-11T19:55:51.100065+0000","last_became_active":"2024-03-11T19:55:50.108182+0000","last_became_peered":"2024-03-11T19:55:50.108182+0000","last_unstale":"2024-03-11T19:55:51.100065+0000","last_undegraded":"2024-03-11T19:55:51.100065+0000","last_fullsized":"2024-03-11T19:55:51.100065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:00.756986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'16","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:16.660527+0000","last_change":"2024-03-11T19:55:44.677491+0000","last_active":"2024-03-11T19:56:16.660527+0000","last_peered":"2024-03-11T19:56:16.660527+0000","last_clean":"2024-03-11T19:56:16.660527+0000","last_became_active":"2024-03-11T19:55:44.677176+0000","last_became_peered":"2024-03-11T19:55:44.677176+0000","last_unstale":"2024-03-11T19:56:16.660527+0000","last_undegraded":"2024-03-11T19:56:16.660527+0000","last_fullsized":"2024-03-11T19:56:16.660527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:51.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205028+0000","last_change":"2024-03-11T19:55:45.703310+0000","last_active":"2024-03-11T19:55:51.205028+0000","last_peered":"2024-03-11T19:55:51.205028+0000","last_clean":"2024-03-11T19:55:51.205028+0000","last_became_active":"2024-03-11T19:55:45.702933+0000","last_became_peered":"2024-03-11T19:55:45.702933+0000","last_unstale":"2024-03-11T19:55:51.205028+0000","last_undegraded":"2024-03-11T19:55:51.205028+0000","last_fullsized":"2024-03-11T19:55:51.205028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:07:58.787949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205103+0000","last_change":"2024-03-11T19:55:50.103849+0000","last_active":"2024-03-11T19:55:51.205103+0000","last_peered":"2024-03-11T19:55:51.205103+0000","last_clean":"2024-03-11T19:55:51.205103+0000","last_became_active":"2024-03-11T19:55:50.103360+0000","last_became_peered":"2024-03-11T19:55:50.103360+0000","last_unstale":"2024-03-11T19:55:51.205103+0000","last_undegraded":"2024-03-11T19:55:51.205103+0000","last_fullsized":"2024-03-11T19:55:51.205103+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:38:05.151041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205907+0000","last_change":"2024-03-11T19:55:44.669839+0000","last_active":"2024-03-11T19:55:51.205907+0000","last_peered":"2024-03-11T19:55:51.205907+0000","last_clean":"2024-03-11T19:55:51.205907+0000","last_became_active":"2024-03-11T19:55:44.669494+0000","last_became_peered":"2024-03-11T19:55:44.669494+0000","last_unstale":"2024-03-11T19:55:51.205907+0000","last_undegraded":"2024-03-11T19:55:51.205907+0000","last_fullsized":"2024-03-11T19:55:51.205907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:00.313847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206362+0000","last_change":"2024-03-11T19:55:45.708929+0000","last_active":"2024-03-11T19:55:51.206362+0000","last_peered":"2024-03-11T19:55:51.206362+0000","last_clean":"2024-03-11T19:55:51.206362+0000","last_became_active":"2024-03-11T19:55:45.708141+0000","last_became_peered":"2024-03-11T19:55:45.708141+0000","last_unstale":"2024-03-11T19:55:51.206362+0000","last_undegraded":"2024-03-11T19:55:51.206362+0000","last_fullsized":"2024-03-11T19:55:51.206362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:01:19.756610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206284+0000","last_change":"2024-03-11T19:55:50.143916+0000","last_active":"2024-03-11T19:55:51.206284+0000","last_peered":"2024-03-11T19:55:51.206284+0000","last_clean":"2024-03-11T19:55:51.206284+0000","last_became_active":"2024-03-11T19:55:50.143341+0000","last_became_peered":"2024-03-11T19:55:50.143341+0000","last_unstale":"2024-03-11T19:55:51.206284+0000","last_undegraded":"2024-03-11T19:55:51.206284+0000","last_fullsized":"2024-03-11T19:55:51.206284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:40.637529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.207974+0000","last_change":"2024-03-11T19:55:44.688468+0000","last_active":"2024-03-11T19:55:51.207974+0000","last_peered":"2024-03-11T19:55:51.207974+0000","last_clean":"2024-03-11T19:55:51.207974+0000","last_became_active":"2024-03-11T19:55:44.688065+0000","last_became_peered":"2024-03-11T19:55:44.688065+0000","last_unstale":"2024-03-11T19:55:51.207974+0000","last_undegraded":"2024-03-11T19:55:51.207974+0000","last_fullsized":"2024-03-11T19:55:51.207974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:29:23.392749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094750+0000","last_change":"2024-03-11T19:55:45.728970+0000","last_active":"2024-03-11T19:55:51.094750+0000","last_peered":"2024-03-11T19:55:51.094750+0000","last_clean":"2024-03-11T19:55:51.094750+0000","last_became_active":"2024-03-11T19:55:45.728516+0000","last_became_peered":"2024-03-11T19:55:45.728516+0000","last_unstale":"2024-03-11T19:55:51.094750+0000","last_undegraded":"2024-03-11T19:55:51.094750+0000","last_fullsized":"2024-03-11T19:55:51.094750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:19.242610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.099289+0000","last_change":"2024-03-11T19:55:50.111884+0000","last_active":"2024-03-11T19:55:51.099289+0000","last_peered":"2024-03-11T19:55:51.099289+0000","last_clean":"2024-03-11T19:55:51.099289+0000","last_became_active":"2024-03-11T19:55:50.110802+0000","last_became_peered":"2024-03-11T19:55:50.110802+0000","last_unstale":"2024-03-11T19:55:51.099289+0000","last_undegraded":"2024-03-11T19:55:51.099289+0000","last_fullsized":"2024-03-11T19:55:51.099289+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:35.866368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206422+0000","last_change":"2024-03-11T19:55:44.677156+0000","last_active":"2024-03-11T19:55:51.206422+0000","last_peered":"2024-03-11T19:55:51.206422+0000","last_clean":"2024-03-11T19:55:51.206422+0000","last_became_active":"2024-03-11T19:55:44.676656+0000","last_became_peered":"2024-03-11T19:55:44.676656+0000","last_unstale":"2024-03-11T19:55:51.206422+0000","last_undegraded":"2024-03-11T19:55:51.206422+0000","last_fullsized":"2024-03-11T19:55:51.206422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:40:27.521669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094284+0000","last_change":"2024-03-11T19:55:45.715545+0000","last_active":"2024-03-11T19:55:51.094284+0000","last_peered":"2024-03-11T19:55:51.094284+0000","last_clean":"2024-03-11T19:55:51.094284+0000","last_became_active":"2024-03-11T19:55:45.715042+0000","last_became_peered":"2024-03-11T19:55:45.715042+0000","last_unstale":"2024-03-11T19:55:51.094284+0000","last_undegraded":"2024-03-11T19:55:51.094284+0000","last_fullsized":"2024-03-11T19:55:51.094284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:06:33.362235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.099946+0000","last_change":"2024-03-11T19:55:50.145603+0000","last_active":"2024-03-11T19:55:51.099946+0000","last_peered":"2024-03-11T19:55:51.099946+0000","last_clean":"2024-03-11T19:55:51.099946+0000","last_became_active":"2024-03-11T19:55:50.143689+0000","last_became_peered":"2024-03-11T19:55:50.143689+0000","last_unstale":"2024-03-11T19:55:51.099946+0000","last_undegraded":"2024-03-11T19:55:51.099946+0000","last_fullsized":"2024-03-11T19:55:51.099946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:57.289983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205253+0000","last_change":"2024-03-11T19:55:45.702011+0000","last_active":"2024-03-11T19:55:51.205253+0000","last_peered":"2024-03-11T19:55:51.205253+0000","last_clean":"2024-03-11T19:55:51.205253+0000","last_became_active":"2024-03-11T19:55:45.701673+0000","last_became_peered":"2024-03-11T19:55:45.701673+0000","last_unstale":"2024-03-11T19:55:51.205253+0000","last_undegraded":"2024-03-11T19:55:51.205253+0000","last_fullsized":"2024-03-11T19:55:51.205253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:17.028276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.097926+0000","last_change":"2024-03-11T19:55:44.689121+0000","last_active":"2024-03-11T19:55:51.097926+0000","last_peered":"2024-03-11T19:55:51.097926+0000","last_clean":"2024-03-11T19:55:51.097926+0000","last_became_active":"2024-03-11T19:55:44.688453+0000","last_became_peered":"2024-03-11T19:55:44.688453+0000","last_unstale":"2024-03-11T19:55:51.097926+0000","last_undegraded":"2024-03-11T19:55:51.097926+0000","last_fullsized":"2024-03-11T19:55:51.097926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:31.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206786+0000","last_change":"2024-03-11T19:55:50.098694+0000","last_active":"2024-03-11T19:55:51.206786+0000","last_peered":"2024-03-11T19:55:51.206786+0000","last_clean":"2024-03-11T19:55:51.206786+0000","last_became_active":"2024-03-11T19:55:50.098121+0000","last_became_peered":"2024-03-11T19:55:50.098121+0000","last_unstale":"2024-03-11T19:55:51.206786+0000","last_undegraded":"2024-03-11T19:55:51.206786+0000","last_fullsized":"2024-03-11T19:55:51.206786+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:29:11.046703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.209910+0000","last_change":"2024-03-11T19:55:45.723755+0000","last_active":"2024-03-11T19:55:51.209910+0000","last_peered":"2024-03-11T19:55:51.209910+0000","last_clean":"2024-03-11T19:55:51.209910+0000","last_became_active":"2024-03-11T19:55:45.723078+0000","last_became_peered":"2024-03-11T19:55:45.723078+0000","last_unstale":"2024-03-11T19:55:51.209910+0000","last_undegraded":"2024-03-11T19:55:51.209910+0000","last_fullsized":"2024-03-11T19:55:51.209910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:40:13.739875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.647667+0000","last_change":"2024-03-11T19:55:44.672644+0000","last_active":"2024-03-11T19:56:15.647667+0000","last_peered":"2024-03-11T19:56:15.647667+0000","last_clean":"2024-03-11T19:56:15.647667+0000","last_became_active":"2024-03-11T19:55:44.671645+0000","last_became_peered":"2024-03-11T19:55:44.671645+0000","last_unstale":"2024-03-11T19:56:15.647667+0000","last_undegraded":"2024-03-11T19:56:15.647667+0000","last_fullsized":"2024-03-11T19:56:15.647667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:25:46.637768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":358,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205407+0000","last_change":"2024-03-11T19:54:20.915891+0000","last_active":"2024-03-11T19:55:51.205407+0000","last_peered":"2024-03-11T19:55:51.205407+0000","last_clean":"2024-03-11T19:55:51.205407+0000","last_became_active":"2024-03-11T19:54:20.915450+0000","last_became_peered":"2024-03-11T19:54:20.915450+0000","last_unstale":"2024-03-11T19:55:51.205407+0000","last_undegraded":"2024-03-11T19:55:51.205407+0000","last_fullsized":"2024-03-11T19:55:51.205407+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_clean_scrub_stamp":"2024-03-11T19:50:29.052858+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:47.966076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206136+0000","last_change":"2024-03-11T19:55:50.144053+0000","last_active":"2024-03-11T19:55:51.206136+0000","last_peered":"2024-03-11T19:55:51.206136+0000","last_clean":"2024-03-11T19:55:51.206136+0000","last_became_active":"2024-03-11T19:55:50.143509+0000","last_became_peered":"2024-03-11T19:55:50.143509+0000","last_unstale":"2024-03-11T19:55:51.206136+0000","last_undegraded":"2024-03-11T19:55:51.206136+0000","last_fullsized":"2024-03-11T19:55:51.206136+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:57:15.448812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.097770+0000","last_change":"2024-03-11T19:55:45.712621+0000","last_active":"2024-03-11T19:55:51.097770+0000","last_peered":"2024-03-11T19:55:51.097770+0000","last_clean":"2024-03-11T19:55:51.097770+0000","last_became_active":"2024-03-11T19:55:45.712161+0000","last_became_peered":"2024-03-11T19:55:45.712161+0000","last_unstale":"2024-03-11T19:55:51.097770+0000","last_undegraded":"2024-03-11T19:55:51.097770+0000","last_fullsized":"2024-03-11T19:55:51.097770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:34:52.911944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:16.654456+0000","last_change":"2024-03-11T19:55:44.674606+0000","last_active":"2024-03-11T19:56:16.654456+0000","last_peered":"2024-03-11T19:56:16.654456+0000","last_clean":"2024-03-11T19:56:16.654456+0000","last_became_active":"2024-03-11T19:55:44.674355+0000","last_became_peered":"2024-03-11T19:55:44.674355+0000","last_unstale":"2024-03-11T19:56:16.654456+0000","last_undegraded":"2024-03-11T19:56:16.654456+0000","last_fullsized":"2024-03-11T19:56:16.654456+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:31:01.662582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206367+0000","last_change":"2024-03-11T19:55:50.101468+0000","last_active":"2024-03-11T19:55:51.206367+0000","last_peered":"2024-03-11T19:55:51.206367+0000","last_clean":"2024-03-11T19:55:51.206367+0000","last_became_active":"2024-03-11T19:55:50.101040+0000","last_became_peered":"2024-03-11T19:55:50.101040+0000","last_unstale":"2024-03-11T19:55:51.206367+0000","last_undegraded":"2024-03-11T19:55:51.206367+0000","last_fullsized":"2024-03-11T19:55:51.206367+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:55.354176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:17.661927+0000","last_change":"2024-03-11T19:55:44.681467+0000","last_active":"2024-03-11T19:56:17.661927+0000","last_peered":"2024-03-11T19:56:17.661927+0000","last_clean":"2024-03-11T19:56:17.661927+0000","last_became_active":"2024-03-11T19:55:44.681176+0000","last_became_peered":"2024-03-11T19:55:44.681176+0000","last_unstale":"2024-03-11T19:56:17.661927+0000","last_undegraded":"2024-03-11T19:56:17.661927+0000","last_fullsized":"2024-03-11T19:56:17.661927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:27:38.019116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.099373+0000","last_change":"2024-03-11T19:55:45.709798+0000","last_active":"2024-03-11T19:55:51.099373+0000","last_peered":"2024-03-11T19:55:51.099373+0000","last_clean":"2024-03-11T19:55:51.099373+0000","last_became_active":"2024-03-11T19:55:45.709277+0000","last_became_peered":"2024-03-11T19:55:45.709277+0000","last_unstale":"2024-03-11T19:55:51.099373+0000","last_undegraded":"2024-03-11T19:55:51.099373+0000","last_fullsized":"2024-03-11T19:55:51.099373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:58.852316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206542+0000","last_change":"2024-03-11T19:55:50.146601+0000","last_active":"2024-03-11T19:55:51.206542+0000","last_peered":"2024-03-11T19:55:51.206542+0000","last_clean":"2024-03-11T19:55:51.206542+0000","last_became_active":"2024-03-11T19:55:50.144584+0000","last_became_peered":"2024-03-11T19:55:50.144584+0000","last_unstale":"2024-03-11T19:55:51.206542+0000","last_undegraded":"2024-03-11T19:55:51.206542+0000","last_fullsized":"2024-03-11T19:55:51.206542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:00.365713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.210550+0000","last_change":"2024-03-11T19:55:44.689109+0000","last_active":"2024-03-11T19:55:51.210550+0000","last_peered":"2024-03-11T19:55:51.210550+0000","last_clean":"2024-03-11T19:55:51.210550+0000","last_became_active":"2024-03-11T19:55:44.687323+0000","last_became_peered":"2024-03-11T19:55:44.687323+0000","last_unstale":"2024-03-11T19:55:51.210550+0000","last_undegraded":"2024-03-11T19:55:51.210550+0000","last_fullsized":"2024-03-11T19:55:51.210550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:24.613901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.209058+0000","last_change":"2024-03-11T19:55:45.726033+0000","last_active":"2024-03-11T19:55:51.209058+0000","last_peered":"2024-03-11T19:55:51.209058+0000","last_clean":"2024-03-11T19:55:51.209058+0000","last_became_active":"2024-03-11T19:55:45.725353+0000","last_became_peered":"2024-03-11T19:55:45.725353+0000","last_unstale":"2024-03-11T19:55:51.209058+0000","last_undegraded":"2024-03-11T19:55:51.209058+0000","last_fullsized":"2024-03-11T19:55:51.209058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:36:24.341870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094277+0000","last_change":"2024-03-11T19:55:50.089496+0000","last_active":"2024-03-11T19:55:51.094277+0000","last_peered":"2024-03-11T19:55:51.094277+0000","last_clean":"2024-03-11T19:55:51.094277+0000","last_became_active":"2024-03-11T19:55:50.089072+0000","last_became_peered":"2024-03-11T19:55:50.089072+0000","last_unstale":"2024-03-11T19:55:51.094277+0000","last_undegraded":"2024-03-11T19:55:51.094277+0000","last_fullsized":"2024-03-11T19:55:51.094277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:27:32.539333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.645423+0000","last_change":"2024-03-11T19:55:44.673147+0000","last_active":"2024-03-11T19:56:15.645423+0000","last_peered":"2024-03-11T19:56:15.645423+0000","last_clean":"2024-03-11T19:56:15.645423+0000","last_became_active":"2024-03-11T19:55:44.672787+0000","last_became_peered":"2024-03-11T19:55:44.672787+0000","last_unstale":"2024-03-11T19:56:15.645423+0000","last_undegraded":"2024-03-11T19:56:15.645423+0000","last_fullsized":"2024-03-11T19:56:15.645423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:11:23.864419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206757+0000","last_change":"2024-03-11T19:55:45.732376+0000","last_active":"2024-03-11T19:55:51.206757+0000","last_peered":"2024-03-11T19:55:51.206757+0000","last_clean":"2024-03-11T19:55:51.206757+0000","last_became_active":"2024-03-11T19:55:45.731956+0000","last_became_peered":"2024-03-11T19:55:45.731956+0000","last_unstale":"2024-03-11T19:55:51.206757+0000","last_undegraded":"2024-03-11T19:55:51.206757+0000","last_fullsized":"2024-03-11T19:55:51.206757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:43:48.423678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206190+0000","last_change":"2024-03-11T19:55:50.144080+0000","last_active":"2024-03-11T19:55:51.206190+0000","last_peered":"2024-03-11T19:55:51.206190+0000","last_clean":"2024-03-11T19:55:51.206190+0000","last_became_active":"2024-03-11T19:55:50.143585+0000","last_became_peered":"2024-03-11T19:55:50.143585+0000","last_unstale":"2024-03-11T19:55:51.206190+0000","last_undegraded":"2024-03-11T19:55:51.206190+0000","last_fullsized":"2024-03-11T19:55:51.206190+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:47:51.388823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.643980+0000","last_change":"2024-03-11T19:55:44.677354+0000","last_active":"2024-03-11T19:56:15.643980+0000","last_peered":"2024-03-11T19:56:15.643980+0000","last_clean":"2024-03-11T19:56:15.643980+0000","last_became_active":"2024-03-11T19:55:44.677013+0000","last_became_peered":"2024-03-11T19:55:44.677013+0000","last_unstale":"2024-03-11T19:56:15.643980+0000","last_undegraded":"2024-03-11T19:56:15.643980+0000","last_fullsized":"2024-03-11T19:56:15.643980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:50:48.855320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205541+0000","last_change":"2024-03-11T19:55:45.702278+0000","last_active":"2024-03-11T19:55:51.205541+0000","last_peered":"2024-03-11T19:55:51.205541+0000","last_clean":"2024-03-11T19:55:51.205541+0000","last_became_active":"2024-03-11T19:55:45.701843+0000","last_became_peered":"2024-03-11T19:55:45.701843+0000","last_unstale":"2024-03-11T19:55:51.205541+0000","last_undegraded":"2024-03-11T19:55:51.205541+0000","last_fullsized":"2024-03-11T19:55:51.205541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:55:28.351093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.211060+0000","last_change":"2024-03-11T19:55:50.106083+0000","last_active":"2024-03-11T19:55:51.211060+0000","last_peered":"2024-03-11T19:55:51.211060+0000","last_clean":"2024-03-11T19:55:51.211060+0000","last_became_active":"2024-03-11T19:55:50.105591+0000","last_became_peered":"2024-03-11T19:55:50.105591+0000","last_unstale":"2024-03-11T19:55:51.211060+0000","last_undegraded":"2024-03-11T19:55:51.211060+0000","last_fullsized":"2024-03-11T19:55:51.211060+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:34:58.713672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:18.641056+0000","last_change":"2024-03-11T19:55:44.688128+0000","last_active":"2024-03-11T19:56:18.641056+0000","last_peered":"2024-03-11T19:56:18.641056+0000","last_clean":"2024-03-11T19:56:18.641056+0000","last_became_active":"2024-03-11T19:55:44.684165+0000","last_became_peered":"2024-03-11T19:55:44.684165+0000","last_unstale":"2024-03-11T19:56:18.641056+0000","last_undegraded":"2024-03-11T19:56:18.641056+0000","last_fullsized":"2024-03-11T19:56:18.641056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:37:19.415960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205606+0000","last_change":"2024-03-11T19:55:45.718968+0000","last_active":"2024-03-11T19:55:51.205606+0000","last_peered":"2024-03-11T19:55:51.205606+0000","last_clean":"2024-03-11T19:55:51.205606+0000","last_became_active":"2024-03-11T19:55:45.718625+0000","last_became_peered":"2024-03-11T19:55:45.718625+0000","last_unstale":"2024-03-11T19:55:51.205606+0000","last_undegraded":"2024-03-11T19:55:51.205606+0000","last_fullsized":"2024-03-11T19:55:51.205606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:27.197227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205344+0000","last_change":"2024-03-11T19:55:50.097617+0000","last_active":"2024-03-11T19:55:51.205344+0000","last_peered":"2024-03-11T19:55:51.205344+0000","last_clean":"2024-03-11T19:55:51.205344+0000","last_became_active":"2024-03-11T19:55:50.097137+0000","last_became_peered":"2024-03-11T19:55:50.097137+0000","last_unstale":"2024-03-11T19:55:51.205344+0000","last_undegraded":"2024-03-11T19:55:51.205344+0000","last_fullsized":"2024-03-11T19:55:51.205344+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:19:20.995137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:16.642473+0000","last_change":"2024-03-11T19:55:44.669944+0000","last_active":"2024-03-11T19:56:16.642473+0000","last_peered":"2024-03-11T19:56:16.642473+0000","last_clean":"2024-03-11T19:56:16.642473+0000","last_became_active":"2024-03-11T19:55:44.669309+0000","last_became_peered":"2024-03-11T19:55:44.669309+0000","last_unstale":"2024-03-11T19:56:16.642473+0000","last_undegraded":"2024-03-11T19:56:16.642473+0000","last_fullsized":"2024-03-11T19:56:16.642473+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:06:41.489579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.208243+0000","last_change":"2024-03-11T19:55:45.731348+0000","last_active":"2024-03-11T19:55:51.208243+0000","last_peered":"2024-03-11T19:55:51.208243+0000","last_clean":"2024-03-11T19:55:51.208243+0000","last_became_active":"2024-03-11T19:55:45.730957+0000","last_became_peered":"2024-03-11T19:55:45.730957+0000","last_unstale":"2024-03-11T19:55:51.208243+0000","last_undegraded":"2024-03-11T19:55:51.208243+0000","last_fullsized":"2024-03-11T19:55:51.208243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:06:50.422511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206209+0000","last_change":"2024-03-11T19:55:50.144050+0000","last_active":"2024-03-11T19:55:51.206209+0000","last_peered":"2024-03-11T19:55:51.206209+0000","last_clean":"2024-03-11T19:55:51.206209+0000","last_became_active":"2024-03-11T19:55:50.143591+0000","last_became_peered":"2024-03-11T19:55:50.143591+0000","last_unstale":"2024-03-11T19:55:51.206209+0000","last_undegraded":"2024-03-11T19:55:51.206209+0000","last_fullsized":"2024-03-11T19:55:51.206209+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:02.006665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.644492+0000","last_change":"2024-03-11T19:55:44.671973+0000","last_active":"2024-03-11T19:56:15.644492+0000","last_peered":"2024-03-11T19:56:15.644492+0000","last_clean":"2024-03-11T19:56:15.644492+0000","last_became_active":"2024-03-11T19:55:44.671661+0000","last_became_peered":"2024-03-11T19:55:44.671661+0000","last_unstale":"2024-03-11T19:56:15.644492+0000","last_undegraded":"2024-03-11T19:56:15.644492+0000","last_fullsized":"2024-03-11T19:56:15.644492+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:35.747386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206466+0000","last_change":"2024-03-11T19:55:45.705545+0000","last_active":"2024-03-11T19:55:51.206466+0000","last_peered":"2024-03-11T19:55:51.206466+0000","last_clean":"2024-03-11T19:55:51.206466+0000","last_became_active":"2024-03-11T19:55:45.705083+0000","last_became_peered":"2024-03-11T19:55:45.705083+0000","last_unstale":"2024-03-11T19:55:51.206466+0000","last_undegraded":"2024-03-11T19:55:51.206466+0000","last_fullsized":"2024-03-11T19:55:51.206466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:02:58.219026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.204914+0000","last_change":"2024-03-11T19:55:50.097951+0000","last_active":"2024-03-11T19:55:51.204914+0000","last_peered":"2024-03-11T19:55:51.204914+0000","last_clean":"2024-03-11T19:55:51.204914+0000","last_became_active":"2024-03-11T19:55:50.097446+0000","last_became_peered":"2024-03-11T19:55:50.097446+0000","last_unstale":"2024-03-11T19:55:51.204914+0000","last_undegraded":"2024-03-11T19:55:51.204914+0000","last_fullsized":"2024-03-11T19:55:51.204914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:12.105801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:17.662364+0000","last_change":"2024-03-11T19:55:44.673136+0000","last_active":"2024-03-11T19:56:17.662364+0000","last_peered":"2024-03-11T19:56:17.662364+0000","last_clean":"2024-03-11T19:56:17.662364+0000","last_became_active":"2024-03-11T19:55:44.672467+0000","last_became_peered":"2024-03-11T19:55:44.672467+0000","last_unstale":"2024-03-11T19:56:17.662364+0000","last_undegraded":"2024-03-11T19:56:17.662364+0000","last_fullsized":"2024-03-11T19:56:17.662364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:46:05.931253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206896+0000","last_change":"2024-03-11T19:55:45.710561+0000","last_active":"2024-03-11T19:55:51.206896+0000","last_peered":"2024-03-11T19:55:51.206896+0000","last_clean":"2024-03-11T19:55:51.206896+0000","last_became_active":"2024-03-11T19:55:45.710164+0000","last_became_peered":"2024-03-11T19:55:45.710164+0000","last_unstale":"2024-03-11T19:55:51.206896+0000","last_undegraded":"2024-03-11T19:55:51.206896+0000","last_fullsized":"2024-03-11T19:55:51.206896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:00:33.977595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206167+0000","last_change":"2024-03-11T19:55:50.113390+0000","last_active":"2024-03-11T19:55:51.206167+0000","last_peered":"2024-03-11T19:55:51.206167+0000","last_clean":"2024-03-11T19:55:51.206167+0000","last_became_active":"2024-03-11T19:55:50.112656+0000","last_became_peered":"2024-03-11T19:55:50.112656+0000","last_unstale":"2024-03-11T19:55:51.206167+0000","last_undegraded":"2024-03-11T19:55:51.206167+0000","last_fullsized":"2024-03-11T19:55:51.206167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:11.952585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205406+0000","last_change":"2024-03-11T19:55:44.684907+0000","last_active":"2024-03-11T19:55:51.205406+0000","last_peered":"2024-03-11T19:55:51.205406+0000","last_clean":"2024-03-11T19:55:51.205406+0000","last_became_active":"2024-03-11T19:55:44.684600+0000","last_became_peered":"2024-03-11T19:55:44.684600+0000","last_unstale":"2024-03-11T19:55:51.205406+0000","last_undegraded":"2024-03-11T19:55:51.205406+0000","last_fullsized":"2024-03-11T19:55:51.205406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:12:33.666756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206565+0000","last_change":"2024-03-11T19:55:45.723431+0000","last_active":"2024-03-11T19:55:51.206565+0000","last_peered":"2024-03-11T19:55:51.206565+0000","last_clean":"2024-03-11T19:55:51.206565+0000","last_became_active":"2024-03-11T19:55:45.723135+0000","last_became_peered":"2024-03-11T19:55:45.723135+0000","last_unstale":"2024-03-11T19:55:51.206565+0000","last_undegraded":"2024-03-11T19:55:51.206565+0000","last_fullsized":"2024-03-11T19:55:51.206565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:12:35.654094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.204643+0000","last_change":"2024-03-11T19:55:50.092876+0000","last_active":"2024-03-11T19:55:51.204643+0000","last_peered":"2024-03-11T19:55:51.204643+0000","last_clean":"2024-03-11T19:55:51.204643+0000","last_became_active":"2024-03-11T19:55:50.092478+0000","last_became_peered":"2024-03-11T19:55:50.092478+0000","last_unstale":"2024-03-11T19:55:51.204643+0000","last_undegraded":"2024-03-11T19:55:51.204643+0000","last_fullsized":"2024-03-11T19:55:51.204643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:39.248325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.099826+0000","last_change":"2024-03-11T19:55:44.688932+0000","last_active":"2024-03-11T19:55:51.099826+0000","last_peered":"2024-03-11T19:55:51.099826+0000","last_clean":"2024-03-11T19:55:51.099826+0000","last_became_active":"2024-03-11T19:55:44.687997+0000","last_became_peered":"2024-03-11T19:55:44.687997+0000","last_unstale":"2024-03-11T19:55:51.099826+0000","last_undegraded":"2024-03-11T19:55:51.099826+0000","last_fullsized":"2024-03-11T19:55:51.099826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:51:38.444939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206849+0000","last_change":"2024-03-11T19:55:45.702924+0000","last_active":"2024-03-11T19:55:51.206849+0000","last_peered":"2024-03-11T19:55:51.206849+0000","last_clean":"2024-03-11T19:55:51.206849+0000","last_became_active":"2024-03-11T19:55:45.702530+0000","last_became_peered":"2024-03-11T19:55:45.702530+0000","last_unstale":"2024-03-11T19:55:51.206849+0000","last_undegraded":"2024-03-11T19:55:51.206849+0000","last_fullsized":"2024-03-11T19:55:51.206849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:14.679964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205728+0000","last_change":"2024-03-11T19:55:45.706123+0000","last_active":"2024-03-11T19:55:51.205728+0000","last_peered":"2024-03-11T19:55:51.205728+0000","last_clean":"2024-03-11T19:55:51.205728+0000","last_became_active":"2024-03-11T19:55:45.705706+0000","last_became_peered":"2024-03-11T19:55:45.705706+0000","last_unstale":"2024-03-11T19:55:51.205728+0000","last_undegraded":"2024-03-11T19:55:51.205728+0000","last_fullsized":"2024-03-11T19:55:51.205728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:42:58.989823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094469+0000","last_change":"2024-03-11T19:55:45.703188+0000","last_active":"2024-03-11T19:55:51.094469+0000","last_peered":"2024-03-11T19:55:51.094469+0000","last_clean":"2024-03-11T19:55:51.094469+0000","last_became_active":"2024-03-11T19:55:45.702803+0000","last_became_peered":"2024-03-11T19:55:45.702803+0000","last_unstale":"2024-03-11T19:55:51.094469+0000","last_undegraded":"2024-03-11T19:55:51.094469+0000","last_fullsized":"2024-03-11T19:55:51.094469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:20.922963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094112+0000","last_change":"2024-03-11T19:55:45.728856+0000","last_active":"2024-03-11T19:55:51.094112+0000","last_peered":"2024-03-11T19:55:51.094112+0000","last_clean":"2024-03-11T19:55:51.094112+0000","last_became_active":"2024-03-11T19:55:45.728526+0000","last_became_peered":"2024-03-11T19:55:45.728526+0000","last_unstale":"2024-03-11T19:55:51.094112+0000","last_undegraded":"2024-03-11T19:55:51.094112+0000","last_fullsized":"2024-03-11T19:55:51.094112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:25:30.836073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206776+0000","last_change":"2024-03-11T19:55:45.725075+0000","last_active":"2024-03-11T19:55:51.206776+0000","last_peered":"2024-03-11T19:55:51.206776+0000","last_clean":"2024-03-11T19:55:51.206776+0000","last_became_active":"2024-03-11T19:55:45.724627+0000","last_became_peered":"2024-03-11T19:55:45.724627+0000","last_unstale":"2024-03-11T19:55:51.206776+0000","last_undegraded":"2024-03-11T19:55:51.206776+0000","last_fullsized":"2024-03-11T19:55:51.206776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:01.039660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.099167+0000","last_change":"2024-03-11T19:55:45.718879+0000","last_active":"2024-03-11T19:55:51.099167+0000","last_peered":"2024-03-11T19:55:51.099167+0000","last_clean":"2024-03-11T19:55:51.099167+0000","last_became_active":"2024-03-11T19:55:45.718535+0000","last_became_peered":"2024-03-11T19:55:45.718535+0000","last_unstale":"2024-03-11T19:55:51.099167+0000","last_undegraded":"2024-03-11T19:55:51.099167+0000","last_fullsized":"2024-03-11T19:55:51.099167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:48.236368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206528+0000","last_change":"2024-03-11T19:55:45.703432+0000","last_active":"2024-03-11T19:55:51.206528+0000","last_peered":"2024-03-11T19:55:51.206528+0000","last_clean":"2024-03-11T19:55:51.206528+0000","last_became_active":"2024-03-11T19:55:45.702669+0000","last_became_peered":"2024-03-11T19:55:45.702669+0000","last_unstale":"2024-03-11T19:55:51.206528+0000","last_undegraded":"2024-03-11T19:55:51.206528+0000","last_fullsized":"2024-03-11T19:55:51.206528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:02.496981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094804+0000","last_change":"2024-03-11T19:55:45.708733+0000","last_active":"2024-03-11T19:55:51.094804+0000","last_peered":"2024-03-11T19:55:51.094804+0000","last_clean":"2024-03-11T19:55:51.094804+0000","last_became_active":"2024-03-11T19:55:45.708369+0000","last_became_peered":"2024-03-11T19:55:45.708369+0000","last_unstale":"2024-03-11T19:55:51.094804+0000","last_undegraded":"2024-03-11T19:55:51.094804+0000","last_fullsized":"2024-03-11T19:55:51.094804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:36.619802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.209557+0000","last_change":"2024-03-11T19:55:45.725698+0000","last_active":"2024-03-11T19:55:51.209557+0000","last_peered":"2024-03-11T19:55:51.209557+0000","last_clean":"2024-03-11T19:55:51.209557+0000","last_became_active":"2024-03-11T19:55:45.724872+0000","last_became_peered":"2024-03-11T19:55:45.724872+0000","last_unstale":"2024-03-11T19:55:51.209557+0000","last_undegraded":"2024-03-11T19:55:51.209557+0000","last_fullsized":"2024-03-11T19:55:51.209557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.982969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205251+0000","last_change":"2024-03-11T19:55:45.726923+0000","last_active":"2024-03-11T19:55:51.205251+0000","last_peered":"2024-03-11T19:55:51.205251+0000","last_clean":"2024-03-11T19:55:51.205251+0000","last_became_active":"2024-03-11T19:55:45.726589+0000","last_became_peered":"2024-03-11T19:55:45.726589+0000","last_unstale":"2024-03-11T19:55:51.205251+0000","last_undegraded":"2024-03-11T19:55:51.205251+0000","last_fullsized":"2024-03-11T19:55:51.205251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:29:15.197359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.208841+0000","last_change":"2024-03-11T19:55:45.723311+0000","last_active":"2024-03-11T19:55:51.208841+0000","last_peered":"2024-03-11T19:55:51.208841+0000","last_clean":"2024-03-11T19:55:51.208841+0000","last_became_active":"2024-03-11T19:55:45.722727+0000","last_became_peered":"2024-03-11T19:55:45.722727+0000","last_unstale":"2024-03-11T19:55:51.208841+0000","last_undegraded":"2024-03-11T19:55:51.208841+0000","last_fullsized":"2024-03-11T19:55:51.208841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:19:34.819665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205888+0000","last_change":"2024-03-11T19:55:45.729141+0000","last_active":"2024-03-11T19:55:51.205888+0000","last_peered":"2024-03-11T19:55:51.205888+0000","last_clean":"2024-03-11T19:55:51.205888+0000","last_became_active":"2024-03-11T19:55:45.728781+0000","last_became_peered":"2024-03-11T19:55:45.728781+0000","last_unstale":"2024-03-11T19:55:51.205888+0000","last_undegraded":"2024-03-11T19:55:51.205888+0000","last_fullsized":"2024-03-11T19:55:51.205888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:02.939181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.098730+0000","last_change":"2024-03-11T19:55:45.723892+0000","last_active":"2024-03-11T19:55:51.098730+0000","last_peered":"2024-03-11T19:55:51.098730+0000","last_clean":"2024-03-11T19:55:51.098730+0000","last_became_active":"2024-03-11T19:55:45.723544+0000","last_became_peered":"2024-03-11T19:55:45.723544+0000","last_unstale":"2024-03-11T19:55:51.098730+0000","last_undegraded":"2024-03-11T19:55:51.098730+0000","last_fullsized":"2024-03-11T19:55:51.098730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:09:08.720854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206470+0000","last_change":"2024-03-11T19:55:45.708494+0000","last_active":"2024-03-11T19:55:51.206470+0000","last_peered":"2024-03-11T19:55:51.206470+0000","last_clean":"2024-03-11T19:55:51.206470+0000","last_became_active":"2024-03-11T19:55:45.708114+0000","last_became_peered":"2024-03-11T19:55:45.708114+0000","last_unstale":"2024-03-11T19:55:51.206470+0000","last_undegraded":"2024-03-11T19:55:51.206470+0000","last_fullsized":"2024-03-11T19:55:51.206470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:28.809268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7984,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":25955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383701,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":478407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645337,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32760,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712392,"statfs":{"total":95995035648,"available":95961489408,"internally_reserved":0,"allocated":1622016,"data_stored":1073003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":477999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070503,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":477953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332140,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36892,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708260,"statfs":{"total":95995035648,"available":95957258240,"internally_reserved":0,"allocated":1593344,"data_stored":1069375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495663,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":481639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":479784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":482407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":477645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1589248,"data_stored":1067765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:56:21.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-11T19:56:21.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:21 smithi114 ceph-mon[36376]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:21.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:21 smithi114 ceph-mon[36376]: pgmap v337: 129 pgs: 129 active+clean; 584 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-03-11T19:56:21.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:21 smithi114 ceph-mon[36376]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:21.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:21 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1715310700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:56:21.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:21 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3282555261' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:22.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:21 smithi161 ceph-mon[36332]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:22.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:21 smithi161 ceph-mon[36332]: pgmap v337: 129 pgs: 129 active+clean; 584 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-03-11T19:56:22.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:21 smithi161 ceph-mon[36332]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:56:22.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:21 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1715310700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:56:22.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:21 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3282555261' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:22.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.247+0000 7fec7b200640 1 -- 172.21.15.97:0/171733481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec74109370 msgr2=0x7fec7410fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:22.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.247+0000 7fec7b200640 1 --2- 172.21.15.97:0/171733481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec74109370 0x7fec7410fc00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fec6c009be0 tx=0x7fec6c02bef0 comp rx=0 tx=0).stop 2024-03-11T19:56:22.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.247+0000 7fec7b200640 1 -- 172.21.15.97:0/171733481 shutdown_connections 2024-03-11T19:56:22.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.247+0000 7fec7b200640 1 --2- 172.21.15.97:0/171733481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec74109370 0x7fec7410fc00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:22.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.247+0000 7fec7b200640 1 --2- 172.21.15.97:0/171733481 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec741089b0 0x7fec74108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:22.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.247+0000 7fec7b200640 1 --2- 172.21.15.97:0/171733481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec741077c0 0x7fec74107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:22.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.247+0000 7fec7b200640 1 -- 172.21.15.97:0/171733481 >> 172.21.15.97:0/171733481 conn(0x7fec74076790 msgr2=0x7fec74078bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:22.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.248+0000 7fec7b200640 1 -- 172.21.15.97:0/171733481 shutdown_connections 2024-03-11T19:56:22.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.248+0000 7fec7b200640 1 -- 172.21.15.97:0/171733481 wait complete. 2024-03-11T19:56:22.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.249+0000 7fec7b200640 1 Processor -- start 2024-03-11T19:56:22.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.249+0000 7fec7b200640 1 -- start start 2024-03-11T19:56:22.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.249+0000 7fec7b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec741077c0 0x7fec7419d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:22.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.249+0000 7fec7b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec741089b0 0x7fec7419dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:22.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.249+0000 7fec7b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec74109370 0x7fec741a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:22.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.249+0000 7fec7b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec741a52b0 con 0x7fec741077c0 2024-03-11T19:56:22.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.249+0000 7fec7b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec741a5420 con 0x7fec741089b0 2024-03-11T19:56:22.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.249+0000 7fec7b200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec741a55e0 con 0x7fec74109370 2024-03-11T19:56:22.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.249+0000 7fec73e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec741077c0 0x7fec7419d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:22.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.249+0000 7fec73e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec741077c0 0x7fec7419d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:59904/0 (socket says 172.21.15.97:59904) 2024-03-11T19:56:22.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.249+0000 7fec73e00640 1 -- 172.21.15.97:0/1522668321 learned_addr learned my addr 172.21.15.97:0/1522668321 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:22.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.249+0000 7fec79200640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec74109370 0x7fec741a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:22.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.249+0000 7fec6b400640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec741089b0 0x7fec7419dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:22.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.250+0000 7fec73e00640 1 -- 172.21.15.97:0/1522668321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec74109370 msgr2=0x7fec741a4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:22.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.250+0000 7fec73e00640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec74109370 0x7fec741a4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:22.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.250+0000 7fec73e00640 1 -- 172.21.15.97:0/1522668321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec741089b0 msgr2=0x7fec7419dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:22.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.250+0000 7fec73e00640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec741089b0 0x7fec7419dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:22.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.250+0000 7fec73e00640 1 -- 172.21.15.97:0/1522668321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec64009490 con 0x7fec741077c0 2024-03-11T19:56:22.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.250+0000 7fec73e00640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec741077c0 0x7fec7419d7c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fec6400e9c0 tx=0x7fec6400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:22.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.250+0000 7fec71600640 1 -- 172.21.15.97:0/1522668321 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6400cd80 con 0x7fec741077c0 2024-03-11T19:56:22.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.250+0000 7fec71600640 1 -- 172.21.15.97:0/1522668321 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec64010910 con 0x7fec741077c0 2024-03-11T19:56:22.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.250+0000 7fec71600640 1 -- 172.21.15.97:0/1522668321 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec640188e0 con 0x7fec741077c0 2024-03-11T19:56:22.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.251+0000 7fec7b200640 1 -- 172.21.15.97:0/1522668321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec6c009840 con 0x7fec741077c0 2024-03-11T19:56:22.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.251+0000 7fec7b200640 1 -- 172.21.15.97:0/1522668321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec741a5c90 con 0x7fec741077c0 2024-03-11T19:56:22.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.252+0000 7fec7b200640 1 -- 172.21.15.97:0/1522668321 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7410ea80 con 0x7fec741077c0 2024-03-11T19:56:22.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.254+0000 7fec71600640 1 -- 172.21.15.97:0/1522668321 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fec64010a80 con 0x7fec741077c0 2024-03-11T19:56:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.255+0000 7fec71600640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fec580a1b40 0x7fec580a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.255+0000 7fec71600640 1 -- 172.21.15.97:0/1522668321 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fec64014070 con 0x7fec741077c0 2024-03-11T19:56:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.255+0000 7fec6b400640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fec580a1b40 0x7fec580a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.256+0000 7fec6b400640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fec580a1b40 0x7fec580a4000 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fec7419ecd0 tx=0x7fec5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:22.260 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.259+0000 7fec71600640 1 -- 172.21.15.97:0/1522668321 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fec64096b00 con 0x7fec741077c0 2024-03-11T19:56:22.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.450+0000 7fec7b200640 1 -- 172.21.15.97:0/1522668321 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fec7402ceb0 con 0x7fec580a1b40 2024-03-11T19:56:22.471 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.470+0000 7fec71600640 1 -- 172.21.15.97:0/1522668321 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340324 (secure 0 0 0) 0x7fec7402ceb0 con 0x7fec580a1b40 2024-03-11T19:56:22.480 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:56:22.482 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-11T19:56:22.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.477+0000 7fec7b200640 1 -- 172.21.15.97:0/1522668321 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fec580a1b40 msgr2=0x7fec580a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:22.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.477+0000 7fec7b200640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fec580a1b40 0x7fec580a4000 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fec7419ecd0 tx=0x7fec5c009260 comp rx=0 tx=0).stop 2024-03-11T19:56:22.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.477+0000 7fec7b200640 1 -- 172.21.15.97:0/1522668321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec741077c0 msgr2=0x7fec7419d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:22.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.477+0000 7fec7b200640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec741077c0 0x7fec7419d7c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fec6400e9c0 tx=0x7fec6400edc0 comp rx=0 tx=0).stop 2024-03-11T19:56:22.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.477+0000 7fec7b200640 1 -- 172.21.15.97:0/1522668321 shutdown_connections 2024-03-11T19:56:22.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.477+0000 7fec7b200640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec74109370 0x7fec741a4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:22.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.477+0000 7fec7b200640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec741089b0 0x7fec7419dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:22.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.477+0000 7fec7b200640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fec580a1b40 0x7fec580a4000 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:22.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.477+0000 7fec7b200640 1 --2- 172.21.15.97:0/1522668321 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec741077c0 0x7fec7419d7c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:22.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.477+0000 7fec7b200640 1 -- 172.21.15.97:0/1522668321 >> 172.21.15.97:0/1522668321 conn(0x7fec74076790 msgr2=0x7fec7410bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:22.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.478+0000 7fec7b200640 1 -- 172.21.15.97:0/1522668321 shutdown_connections 2024-03-11T19:56:22.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:22.478+0000 7fec7b200640 1 -- 172.21.15.97:0/1522668321 wait complete. 2024-03-11T19:56:22.498 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-03-11T19:56:20.508518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598352,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":309,"num_write_kb":2982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":252,"ondisk_log_size":252,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383484,"kb_used_data":13512,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558340,"statfs":{"total":1151940427776,"available":1151547740160,"internally_reserved":0,"allocated":13836288,"data_stored":7521051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3344,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006745"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206805+0000","last_change":"2024-03-11T19:55:45.725019+0000","last_active":"2024-03-11T19:55:51.206805+0000","last_peered":"2024-03-11T19:55:51.206805+0000","last_clean":"2024-03-11T19:55:51.206805+0000","last_became_active":"2024-03-11T19:55:45.724602+0000","last_became_peered":"2024-03-11T19:55:45.724602+0000","last_unstale":"2024-03-11T19:55:51.206805+0000","last_undegraded":"2024-03-11T19:55:51.206805+0000","last_fullsized":"2024-03-11T19:55:51.206805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:16:18.218381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206638+0000","last_change":"2024-03-11T19:55:45.724589+0000","last_active":"2024-03-11T19:55:51.206638+0000","last_peered":"2024-03-11T19:55:51.206638+0000","last_clean":"2024-03-11T19:55:51.206638+0000","last_became_active":"2024-03-11T19:55:45.724109+0000","last_became_peered":"2024-03-11T19:55:45.724109+0000","last_unstale":"2024-03-11T19:55:51.206638+0000","last_undegraded":"2024-03-11T19:55:51.206638+0000","last_fullsized":"2024-03-11T19:55:51.206638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:58:12.510437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206347+0000","last_change":"2024-03-11T19:55:45.718831+0000","last_active":"2024-03-11T19:55:51.206347+0000","last_peered":"2024-03-11T19:55:51.206347+0000","last_clean":"2024-03-11T19:55:51.206347+0000","last_became_active":"2024-03-11T19:55:45.717652+0000","last_became_peered":"2024-03-11T19:55:45.717652+0000","last_unstale":"2024-03-11T19:55:51.206347+0000","last_undegraded":"2024-03-11T19:55:51.206347+0000","last_fullsized":"2024-03-11T19:55:51.206347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:16.471754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205658+0000","last_change":"2024-03-11T19:55:45.701640+0000","last_active":"2024-03-11T19:55:51.205658+0000","last_peered":"2024-03-11T19:55:51.205658+0000","last_clean":"2024-03-11T19:55:51.205658+0000","last_became_active":"2024-03-11T19:55:45.699207+0000","last_became_peered":"2024-03-11T19:55:45.699207+0000","last_unstale":"2024-03-11T19:55:51.205658+0000","last_undegraded":"2024-03-11T19:55:51.205658+0000","last_fullsized":"2024-03-11T19:55:51.205658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:01.900429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205411+0000","last_change":"2024-03-11T19:55:45.732040+0000","last_active":"2024-03-11T19:55:51.205411+0000","last_peered":"2024-03-11T19:55:51.205411+0000","last_clean":"2024-03-11T19:55:51.205411+0000","last_became_active":"2024-03-11T19:55:45.731647+0000","last_became_peered":"2024-03-11T19:55:45.731647+0000","last_unstale":"2024-03-11T19:55:51.205411+0000","last_undegraded":"2024-03-11T19:55:51.205411+0000","last_fullsized":"2024-03-11T19:55:51.205411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:51:57.345329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094385+0000","last_change":"2024-03-11T19:55:45.707986+0000","last_active":"2024-03-11T19:55:51.094385+0000","last_peered":"2024-03-11T19:55:51.094385+0000","last_clean":"2024-03-11T19:55:51.094385+0000","last_became_active":"2024-03-11T19:55:45.707641+0000","last_became_peered":"2024-03-11T19:55:45.707641+0000","last_unstale":"2024-03-11T19:55:51.094385+0000","last_undegraded":"2024-03-11T19:55:51.094385+0000","last_fullsized":"2024-03-11T19:55:51.094385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:15:50.635234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206161+0000","last_change":"2024-03-11T19:55:45.713756+0000","last_active":"2024-03-11T19:55:51.206161+0000","last_peered":"2024-03-11T19:55:51.206161+0000","last_clean":"2024-03-11T19:55:51.206161+0000","last_became_active":"2024-03-11T19:55:45.713333+0000","last_became_peered":"2024-03-11T19:55:45.713333+0000","last_unstale":"2024-03-11T19:55:51.206161+0000","last_undegraded":"2024-03-11T19:55:51.206161+0000","last_fullsized":"2024-03-11T19:55:51.206161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:49:58.048921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205952+0000","last_change":"2024-03-11T19:55:45.729587+0000","last_active":"2024-03-11T19:55:51.205952+0000","last_peered":"2024-03-11T19:55:51.205952+0000","last_clean":"2024-03-11T19:55:51.205952+0000","last_became_active":"2024-03-11T19:55:45.729037+0000","last_became_peered":"2024-03-11T19:55:45.729037+0000","last_unstale":"2024-03-11T19:55:51.205952+0000","last_undegraded":"2024-03-11T19:55:51.205952+0000","last_fullsized":"2024-03-11T19:55:51.205952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:55:43.445192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205943+0000","last_change":"2024-03-11T19:55:45.709793+0000","last_active":"2024-03-11T19:55:51.205943+0000","last_peered":"2024-03-11T19:55:51.205943+0000","last_clean":"2024-03-11T19:55:51.205943+0000","last_became_active":"2024-03-11T19:55:45.708631+0000","last_became_peered":"2024-03-11T19:55:45.708631+0000","last_unstale":"2024-03-11T19:55:51.205943+0000","last_undegraded":"2024-03-11T19:55:51.205943+0000","last_fullsized":"2024-03-11T19:55:51.205943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:02.144066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094886+0000","last_change":"2024-03-11T19:55:45.704227+0000","last_active":"2024-03-11T19:55:51.094886+0000","last_peered":"2024-03-11T19:55:51.094886+0000","last_clean":"2024-03-11T19:55:51.094886+0000","last_became_active":"2024-03-11T19:55:45.703885+0000","last_became_peered":"2024-03-11T19:55:45.703885+0000","last_unstale":"2024-03-11T19:55:51.094886+0000","last_undegraded":"2024-03-11T19:55:51.094886+0000","last_fullsized":"2024-03-11T19:55:51.094886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:36:21.401988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206822+0000","last_change":"2024-03-11T19:55:45.708950+0000","last_active":"2024-03-11T19:55:51.206822+0000","last_peered":"2024-03-11T19:55:51.206822+0000","last_clean":"2024-03-11T19:55:51.206822+0000","last_became_active":"2024-03-11T19:55:45.708544+0000","last_became_peered":"2024-03-11T19:55:45.708544+0000","last_unstale":"2024-03-11T19:55:51.206822+0000","last_undegraded":"2024-03-11T19:55:51.206822+0000","last_fullsized":"2024-03-11T19:55:51.206822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:06:34.244449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205548+0000","last_change":"2024-03-11T19:55:45.703923+0000","last_active":"2024-03-11T19:55:51.205548+0000","last_peered":"2024-03-11T19:55:51.205548+0000","last_clean":"2024-03-11T19:55:51.205548+0000","last_became_active":"2024-03-11T19:55:45.703595+0000","last_became_peered":"2024-03-11T19:55:45.703595+0000","last_unstale":"2024-03-11T19:55:51.205548+0000","last_undegraded":"2024-03-11T19:55:51.205548+0000","last_fullsized":"2024-03-11T19:55:51.205548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:00:19.683907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.098569+0000","last_change":"2024-03-11T19:55:45.707499+0000","last_active":"2024-03-11T19:55:51.098569+0000","last_peered":"2024-03-11T19:55:51.098569+0000","last_clean":"2024-03-11T19:55:51.098569+0000","last_became_active":"2024-03-11T19:55:45.706344+0000","last_became_peered":"2024-03-11T19:55:45.706344+0000","last_unstale":"2024-03-11T19:55:51.098569+0000","last_undegraded":"2024-03-11T19:55:51.098569+0000","last_fullsized":"2024-03-11T19:55:51.098569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:38:24.082332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205979+0000","last_change":"2024-03-11T19:55:45.722182+0000","last_active":"2024-03-11T19:55:51.205979+0000","last_peered":"2024-03-11T19:55:51.205979+0000","last_clean":"2024-03-11T19:55:51.205979+0000","last_became_active":"2024-03-11T19:55:45.721783+0000","last_became_peered":"2024-03-11T19:55:45.721783+0000","last_unstale":"2024-03-11T19:55:51.205979+0000","last_undegraded":"2024-03-11T19:55:51.205979+0000","last_fullsized":"2024-03-11T19:55:51.205979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:41:08.559002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205879+0000","last_change":"2024-03-11T19:55:45.718881+0000","last_active":"2024-03-11T19:55:51.205879+0000","last_peered":"2024-03-11T19:55:51.205879+0000","last_clean":"2024-03-11T19:55:51.205879+0000","last_became_active":"2024-03-11T19:55:45.718458+0000","last_became_peered":"2024-03-11T19:55:45.718458+0000","last_unstale":"2024-03-11T19:55:51.205879+0000","last_undegraded":"2024-03-11T19:55:51.205879+0000","last_fullsized":"2024-03-11T19:55:51.205879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:22.075892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.204711+0000","last_change":"2024-03-11T19:55:45.726163+0000","last_active":"2024-03-11T19:55:51.204711+0000","last_peered":"2024-03-11T19:55:51.204711+0000","last_clean":"2024-03-11T19:55:51.204711+0000","last_became_active":"2024-03-11T19:55:45.725772+0000","last_became_peered":"2024-03-11T19:55:45.725772+0000","last_unstale":"2024-03-11T19:55:51.204711+0000","last_undegraded":"2024-03-11T19:55:51.204711+0000","last_fullsized":"2024-03-11T19:55:51.204711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:36:19.057920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206587+0000","last_change":"2024-03-11T19:55:45.724787+0000","last_active":"2024-03-11T19:55:51.206587+0000","last_peered":"2024-03-11T19:55:51.206587+0000","last_clean":"2024-03-11T19:55:51.206587+0000","last_became_active":"2024-03-11T19:55:45.724437+0000","last_became_peered":"2024-03-11T19:55:45.724437+0000","last_unstale":"2024-03-11T19:55:51.206587+0000","last_undegraded":"2024-03-11T19:55:51.206587+0000","last_fullsized":"2024-03-11T19:55:51.206587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:55:53.811679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.210628+0000","last_change":"2024-03-11T19:55:45.714934+0000","last_active":"2024-03-11T19:55:51.210628+0000","last_peered":"2024-03-11T19:55:51.210628+0000","last_clean":"2024-03-11T19:55:51.210628+0000","last_became_active":"2024-03-11T19:55:45.709595+0000","last_became_peered":"2024-03-11T19:55:45.709595+0000","last_unstale":"2024-03-11T19:55:51.210628+0000","last_undegraded":"2024-03-11T19:55:51.210628+0000","last_fullsized":"2024-03-11T19:55:51.210628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:48.319073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094743+0000","last_change":"2024-03-11T19:55:45.725747+0000","last_active":"2024-03-11T19:55:51.094743+0000","last_peered":"2024-03-11T19:55:51.094743+0000","last_clean":"2024-03-11T19:55:51.094743+0000","last_became_active":"2024-03-11T19:55:45.725239+0000","last_became_peered":"2024-03-11T19:55:45.725239+0000","last_unstale":"2024-03-11T19:55:51.094743+0000","last_undegraded":"2024-03-11T19:55:51.094743+0000","last_fullsized":"2024-03-11T19:55:51.094743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:51:24.046681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094568+0000","last_change":"2024-03-11T19:55:50.109913+0000","last_active":"2024-03-11T19:55:51.094568+0000","last_peered":"2024-03-11T19:55:51.094568+0000","last_clean":"2024-03-11T19:55:51.094568+0000","last_became_active":"2024-03-11T19:55:50.109471+0000","last_became_peered":"2024-03-11T19:55:50.109471+0000","last_unstale":"2024-03-11T19:55:51.094568+0000","last_undegraded":"2024-03-11T19:55:51.094568+0000","last_fullsized":"2024-03-11T19:55:51.094568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:40.258022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.211124+0000","last_change":"2024-03-11T19:55:44.680181+0000","last_active":"2024-03-11T19:55:51.211124+0000","last_peered":"2024-03-11T19:55:51.211124+0000","last_clean":"2024-03-11T19:55:51.211124+0000","last_became_active":"2024-03-11T19:55:44.679807+0000","last_became_peered":"2024-03-11T19:55:44.679807+0000","last_unstale":"2024-03-11T19:55:51.211124+0000","last_undegraded":"2024-03-11T19:55:51.211124+0000","last_fullsized":"2024-03-11T19:55:51.211124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:23:04.532836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.210078+0000","last_change":"2024-03-11T19:55:45.723547+0000","last_active":"2024-03-11T19:55:51.210078+0000","last_peered":"2024-03-11T19:55:51.210078+0000","last_clean":"2024-03-11T19:55:51.210078+0000","last_became_active":"2024-03-11T19:55:45.722516+0000","last_became_peered":"2024-03-11T19:55:45.722516+0000","last_unstale":"2024-03-11T19:55:51.210078+0000","last_undegraded":"2024-03-11T19:55:51.210078+0000","last_fullsized":"2024-03-11T19:55:51.210078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:40.271814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205127+0000","last_change":"2024-03-11T19:55:50.102141+0000","last_active":"2024-03-11T19:55:51.205127+0000","last_peered":"2024-03-11T19:55:51.205127+0000","last_clean":"2024-03-11T19:55:51.205127+0000","last_became_active":"2024-03-11T19:55:50.101667+0000","last_became_peered":"2024-03-11T19:55:50.101667+0000","last_unstale":"2024-03-11T19:55:51.205127+0000","last_undegraded":"2024-03-11T19:55:51.205127+0000","last_fullsized":"2024-03-11T19:55:51.205127+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:54:11.453059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.210399+0000","last_change":"2024-03-11T19:55:44.684623+0000","last_active":"2024-03-11T19:55:51.210399+0000","last_peered":"2024-03-11T19:55:51.210399+0000","last_clean":"2024-03-11T19:55:51.210399+0000","last_became_active":"2024-03-11T19:55:44.683611+0000","last_became_peered":"2024-03-11T19:55:44.683611+0000","last_unstale":"2024-03-11T19:55:51.210399+0000","last_undegraded":"2024-03-11T19:55:51.210399+0000","last_fullsized":"2024-03-11T19:55:51.210399+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:22:50.954364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206424+0000","last_change":"2024-03-11T19:55:45.723619+0000","last_active":"2024-03-11T19:55:51.206424+0000","last_peered":"2024-03-11T19:55:51.206424+0000","last_clean":"2024-03-11T19:55:51.206424+0000","last_became_active":"2024-03-11T19:55:45.722685+0000","last_became_peered":"2024-03-11T19:55:45.722685+0000","last_unstale":"2024-03-11T19:55:51.206424+0000","last_undegraded":"2024-03-11T19:55:51.206424+0000","last_fullsized":"2024-03-11T19:55:51.206424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:47:13.747776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205203+0000","last_change":"2024-03-11T19:55:50.102898+0000","last_active":"2024-03-11T19:55:51.205203+0000","last_peered":"2024-03-11T19:55:51.205203+0000","last_clean":"2024-03-11T19:55:51.205203+0000","last_became_active":"2024-03-11T19:55:50.102491+0000","last_became_peered":"2024-03-11T19:55:50.102491+0000","last_unstale":"2024-03-11T19:55:51.205203+0000","last_undegraded":"2024-03-11T19:55:51.205203+0000","last_fullsized":"2024-03-11T19:55:51.205203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:41:25.389542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:17.661009+0000","last_change":"2024-03-11T19:55:44.673547+0000","last_active":"2024-03-11T19:56:17.661009+0000","last_peered":"2024-03-11T19:56:17.661009+0000","last_clean":"2024-03-11T19:56:17.661009+0000","last_became_active":"2024-03-11T19:55:44.673211+0000","last_became_peered":"2024-03-11T19:55:44.673211+0000","last_unstale":"2024-03-11T19:56:17.661009+0000","last_undegraded":"2024-03-11T19:56:17.661009+0000","last_fullsized":"2024-03-11T19:56:17.661009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:38:09.736615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205415+0000","last_change":"2024-03-11T19:55:45.715157+0000","last_active":"2024-03-11T19:55:51.205415+0000","last_peered":"2024-03-11T19:55:51.205415+0000","last_clean":"2024-03-11T19:55:51.205415+0000","last_became_active":"2024-03-11T19:55:45.714536+0000","last_became_peered":"2024-03-11T19:55:45.714536+0000","last_unstale":"2024-03-11T19:55:51.205415+0000","last_undegraded":"2024-03-11T19:55:51.205415+0000","last_fullsized":"2024-03-11T19:55:51.205415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:02.416736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.204978+0000","last_change":"2024-03-11T19:55:50.102157+0000","last_active":"2024-03-11T19:55:51.204978+0000","last_peered":"2024-03-11T19:55:51.204978+0000","last_clean":"2024-03-11T19:55:51.204978+0000","last_became_active":"2024-03-11T19:55:50.101715+0000","last_became_peered":"2024-03-11T19:55:50.101715+0000","last_unstale":"2024-03-11T19:55:51.204978+0000","last_undegraded":"2024-03-11T19:55:51.204978+0000","last_fullsized":"2024-03-11T19:55:51.204978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:12:18.780708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206402+0000","last_change":"2024-03-11T19:55:44.669876+0000","last_active":"2024-03-11T19:55:51.206402+0000","last_peered":"2024-03-11T19:55:51.206402+0000","last_clean":"2024-03-11T19:55:51.206402+0000","last_became_active":"2024-03-11T19:55:44.669539+0000","last_became_peered":"2024-03-11T19:55:44.669539+0000","last_unstale":"2024-03-11T19:55:51.206402+0000","last_undegraded":"2024-03-11T19:55:51.206402+0000","last_fullsized":"2024-03-11T19:55:51.206402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:29:45.676685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205300+0000","last_change":"2024-03-11T19:55:45.709973+0000","last_active":"2024-03-11T19:55:51.205300+0000","last_peered":"2024-03-11T19:55:51.205300+0000","last_clean":"2024-03-11T19:55:51.205300+0000","last_became_active":"2024-03-11T19:55:45.709645+0000","last_became_peered":"2024-03-11T19:55:45.709645+0000","last_unstale":"2024-03-11T19:55:51.205300+0000","last_undegraded":"2024-03-11T19:55:51.205300+0000","last_fullsized":"2024-03-11T19:55:51.205300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:16.516982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094915+0000","last_change":"2024-03-11T19:55:50.094093+0000","last_active":"2024-03-11T19:55:51.094915+0000","last_peered":"2024-03-11T19:55:51.094915+0000","last_clean":"2024-03-11T19:55:51.094915+0000","last_became_active":"2024-03-11T19:55:50.093653+0000","last_became_peered":"2024-03-11T19:55:50.093653+0000","last_unstale":"2024-03-11T19:55:51.094915+0000","last_undegraded":"2024-03-11T19:55:51.094915+0000","last_fullsized":"2024-03-11T19:55:51.094915+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:28:41.668001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.644996+0000","last_change":"2024-03-11T19:55:44.671547+0000","last_active":"2024-03-11T19:56:15.644996+0000","last_peered":"2024-03-11T19:56:15.644996+0000","last_clean":"2024-03-11T19:56:15.644996+0000","last_became_active":"2024-03-11T19:55:44.671211+0000","last_became_peered":"2024-03-11T19:55:44.671211+0000","last_unstale":"2024-03-11T19:56:15.644996+0000","last_undegraded":"2024-03-11T19:56:15.644996+0000","last_fullsized":"2024-03-11T19:56:15.644996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:18:46.261250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206035+0000","last_change":"2024-03-11T19:55:45.707968+0000","last_active":"2024-03-11T19:55:51.206035+0000","last_peered":"2024-03-11T19:55:51.206035+0000","last_clean":"2024-03-11T19:55:51.206035+0000","last_became_active":"2024-03-11T19:55:45.706998+0000","last_became_peered":"2024-03-11T19:55:45.706998+0000","last_unstale":"2024-03-11T19:55:51.206035+0000","last_undegraded":"2024-03-11T19:55:51.206035+0000","last_fullsized":"2024-03-11T19:55:51.206035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:49:35.711123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094598+0000","last_change":"2024-03-11T19:55:50.104310+0000","last_active":"2024-03-11T19:55:51.094598+0000","last_peered":"2024-03-11T19:55:51.094598+0000","last_clean":"2024-03-11T19:55:51.094598+0000","last_became_active":"2024-03-11T19:55:50.103831+0000","last_became_peered":"2024-03-11T19:55:50.103831+0000","last_unstale":"2024-03-11T19:55:51.094598+0000","last_undegraded":"2024-03-11T19:55:51.094598+0000","last_fullsized":"2024-03-11T19:55:51.094598+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:33.213987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:16.658467+0000","last_change":"2024-03-11T19:55:44.677615+0000","last_active":"2024-03-11T19:56:16.658467+0000","last_peered":"2024-03-11T19:56:16.658467+0000","last_clean":"2024-03-11T19:56:16.658467+0000","last_became_active":"2024-03-11T19:55:44.677153+0000","last_became_peered":"2024-03-11T19:55:44.677153+0000","last_unstale":"2024-03-11T19:56:16.658467+0000","last_undegraded":"2024-03-11T19:56:16.658467+0000","last_fullsized":"2024-03-11T19:56:16.658467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:52:59.153222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205117+0000","last_change":"2024-03-11T19:55:45.731827+0000","last_active":"2024-03-11T19:55:51.205117+0000","last_peered":"2024-03-11T19:55:51.205117+0000","last_clean":"2024-03-11T19:55:51.205117+0000","last_became_active":"2024-03-11T19:55:45.731460+0000","last_became_peered":"2024-03-11T19:55:45.731460+0000","last_unstale":"2024-03-11T19:55:51.205117+0000","last_undegraded":"2024-03-11T19:55:51.205117+0000","last_fullsized":"2024-03-11T19:55:51.205117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:12.277198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206703+0000","last_change":"2024-03-11T19:55:50.109812+0000","last_active":"2024-03-11T19:55:51.206703+0000","last_peered":"2024-03-11T19:55:51.206703+0000","last_clean":"2024-03-11T19:55:51.206703+0000","last_became_active":"2024-03-11T19:55:50.109420+0000","last_became_peered":"2024-03-11T19:55:50.109420+0000","last_unstale":"2024-03-11T19:55:51.206703+0000","last_undegraded":"2024-03-11T19:55:51.206703+0000","last_fullsized":"2024-03-11T19:55:51.206703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:08:30.105941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.098100+0000","last_change":"2024-03-11T19:55:44.683173+0000","last_active":"2024-03-11T19:55:51.098100+0000","last_peered":"2024-03-11T19:55:51.098100+0000","last_clean":"2024-03-11T19:55:51.098100+0000","last_became_active":"2024-03-11T19:55:44.682778+0000","last_became_peered":"2024-03-11T19:55:44.682778+0000","last_unstale":"2024-03-11T19:55:51.098100+0000","last_undegraded":"2024-03-11T19:55:51.098100+0000","last_fullsized":"2024-03-11T19:55:51.098100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:15.119864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206856+0000","last_change":"2024-03-11T19:55:45.721833+0000","last_active":"2024-03-11T19:55:51.206856+0000","last_peered":"2024-03-11T19:55:51.206856+0000","last_clean":"2024-03-11T19:55:51.206856+0000","last_became_active":"2024-03-11T19:55:45.721490+0000","last_became_peered":"2024-03-11T19:55:45.721490+0000","last_unstale":"2024-03-11T19:55:51.206856+0000","last_undegraded":"2024-03-11T19:55:51.206856+0000","last_fullsized":"2024-03-11T19:55:51.206856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:09.824912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206517+0000","last_change":"2024-03-11T19:55:50.108424+0000","last_active":"2024-03-11T19:55:51.206517+0000","last_peered":"2024-03-11T19:55:51.206517+0000","last_clean":"2024-03-11T19:55:51.206517+0000","last_became_active":"2024-03-11T19:55:50.107072+0000","last_became_peered":"2024-03-11T19:55:50.107072+0000","last_unstale":"2024-03-11T19:55:51.206517+0000","last_undegraded":"2024-03-11T19:55:51.206517+0000","last_fullsized":"2024-03-11T19:55:51.206517+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:31.751597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.204613+0000","last_change":"2024-03-11T19:55:44.671570+0000","last_active":"2024-03-11T19:55:51.204613+0000","last_peered":"2024-03-11T19:55:51.204613+0000","last_clean":"2024-03-11T19:55:51.204613+0000","last_became_active":"2024-03-11T19:55:44.671007+0000","last_became_peered":"2024-03-11T19:55:44.671007+0000","last_unstale":"2024-03-11T19:55:51.204613+0000","last_undegraded":"2024-03-11T19:55:51.204613+0000","last_fullsized":"2024-03-11T19:55:51.204613+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:23.865011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.207329+0000","last_change":"2024-03-11T19:55:45.733048+0000","last_active":"2024-03-11T19:55:51.207329+0000","last_peered":"2024-03-11T19:55:51.207329+0000","last_clean":"2024-03-11T19:55:51.207329+0000","last_became_active":"2024-03-11T19:55:45.731656+0000","last_became_peered":"2024-03-11T19:55:45.731656+0000","last_unstale":"2024-03-11T19:55:51.207329+0000","last_undegraded":"2024-03-11T19:55:51.207329+0000","last_fullsized":"2024-03-11T19:55:51.207329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:28:56.433327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205928+0000","last_change":"2024-03-11T19:55:50.105532+0000","last_active":"2024-03-11T19:55:51.205928+0000","last_peered":"2024-03-11T19:55:51.205928+0000","last_clean":"2024-03-11T19:55:51.205928+0000","last_became_active":"2024-03-11T19:55:50.102200+0000","last_became_peered":"2024-03-11T19:55:50.102200+0000","last_unstale":"2024-03-11T19:55:51.205928+0000","last_undegraded":"2024-03-11T19:55:51.205928+0000","last_fullsized":"2024-03-11T19:55:51.205928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:37.060472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.645329+0000","last_change":"2024-03-11T19:55:44.678224+0000","last_active":"2024-03-11T19:56:15.645329+0000","last_peered":"2024-03-11T19:56:15.645329+0000","last_clean":"2024-03-11T19:56:15.645329+0000","last_became_active":"2024-03-11T19:55:44.677936+0000","last_became_peered":"2024-03-11T19:55:44.677936+0000","last_unstale":"2024-03-11T19:56:15.645329+0000","last_undegraded":"2024-03-11T19:56:15.645329+0000","last_fullsized":"2024-03-11T19:56:15.645329+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:20:42.486501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205578+0000","last_change":"2024-03-11T19:55:45.709917+0000","last_active":"2024-03-11T19:55:51.205578+0000","last_peered":"2024-03-11T19:55:51.205578+0000","last_clean":"2024-03-11T19:55:51.205578+0000","last_became_active":"2024-03-11T19:55:45.709318+0000","last_became_peered":"2024-03-11T19:55:45.709318+0000","last_unstale":"2024-03-11T19:55:51.205578+0000","last_undegraded":"2024-03-11T19:55:51.205578+0000","last_fullsized":"2024-03-11T19:55:51.205578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:08.298158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206938+0000","last_change":"2024-03-11T19:55:50.103384+0000","last_active":"2024-03-11T19:55:51.206938+0000","last_peered":"2024-03-11T19:55:51.206938+0000","last_clean":"2024-03-11T19:55:51.206938+0000","last_became_active":"2024-03-11T19:55:50.100641+0000","last_became_peered":"2024-03-11T19:55:50.100641+0000","last_unstale":"2024-03-11T19:55:51.206938+0000","last_undegraded":"2024-03-11T19:55:51.206938+0000","last_fullsized":"2024-03-11T19:55:51.206938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:58.524369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.643034+0000","last_change":"2024-03-11T19:55:44.676004+0000","last_active":"2024-03-11T19:56:15.643034+0000","last_peered":"2024-03-11T19:56:15.643034+0000","last_clean":"2024-03-11T19:56:15.643034+0000","last_became_active":"2024-03-11T19:55:44.675640+0000","last_became_peered":"2024-03-11T19:55:44.675640+0000","last_unstale":"2024-03-11T19:56:15.643034+0000","last_undegraded":"2024-03-11T19:56:15.643034+0000","last_fullsized":"2024-03-11T19:56:15.643034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:15.096780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.207155+0000","last_change":"2024-03-11T19:55:45.723937+0000","last_active":"2024-03-11T19:55:51.207155+0000","last_peered":"2024-03-11T19:55:51.207155+0000","last_clean":"2024-03-11T19:55:51.207155+0000","last_became_active":"2024-03-11T19:55:45.723301+0000","last_became_peered":"2024-03-11T19:55:45.723301+0000","last_unstale":"2024-03-11T19:55:51.207155+0000","last_undegraded":"2024-03-11T19:55:51.207155+0000","last_fullsized":"2024-03-11T19:55:51.207155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:10:35.821109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205522+0000","last_change":"2024-03-11T19:55:50.096136+0000","last_active":"2024-03-11T19:55:51.205522+0000","last_peered":"2024-03-11T19:55:51.205522+0000","last_clean":"2024-03-11T19:55:51.205522+0000","last_became_active":"2024-03-11T19:55:50.095670+0000","last_became_peered":"2024-03-11T19:55:50.095670+0000","last_unstale":"2024-03-11T19:55:51.205522+0000","last_undegraded":"2024-03-11T19:55:51.205522+0000","last_fullsized":"2024-03-11T19:55:51.205522+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:40.943273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205177+0000","last_change":"2024-03-11T19:55:44.681547+0000","last_active":"2024-03-11T19:55:51.205177+0000","last_peered":"2024-03-11T19:55:51.205177+0000","last_clean":"2024-03-11T19:55:51.205177+0000","last_became_active":"2024-03-11T19:55:44.681237+0000","last_became_peered":"2024-03-11T19:55:44.681237+0000","last_unstale":"2024-03-11T19:55:51.205177+0000","last_undegraded":"2024-03-11T19:55:51.205177+0000","last_fullsized":"2024-03-11T19:55:51.205177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:08:09.509915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205616+0000","last_change":"2024-03-11T19:55:45.705025+0000","last_active":"2024-03-11T19:55:51.205616+0000","last_peered":"2024-03-11T19:55:51.205616+0000","last_clean":"2024-03-11T19:55:51.205616+0000","last_became_active":"2024-03-11T19:55:45.702680+0000","last_became_peered":"2024-03-11T19:55:45.702680+0000","last_unstale":"2024-03-11T19:55:51.205616+0000","last_undegraded":"2024-03-11T19:55:51.205616+0000","last_fullsized":"2024-03-11T19:55:51.205616+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:28.245002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.207889+0000","last_change":"2024-03-11T19:55:50.099648+0000","last_active":"2024-03-11T19:55:51.207889+0000","last_peered":"2024-03-11T19:55:51.207889+0000","last_clean":"2024-03-11T19:55:51.207889+0000","last_became_active":"2024-03-11T19:55:50.098516+0000","last_became_peered":"2024-03-11T19:55:50.098516+0000","last_unstale":"2024-03-11T19:55:51.207889+0000","last_undegraded":"2024-03-11T19:55:51.207889+0000","last_fullsized":"2024-03-11T19:55:51.207889+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:19.644046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.643538+0000","last_change":"2024-03-11T19:55:44.672460+0000","last_active":"2024-03-11T19:56:15.643538+0000","last_peered":"2024-03-11T19:56:15.643538+0000","last_clean":"2024-03-11T19:56:15.643538+0000","last_became_active":"2024-03-11T19:55:44.672123+0000","last_became_peered":"2024-03-11T19:55:44.672123+0000","last_unstale":"2024-03-11T19:56:15.643538+0000","last_undegraded":"2024-03-11T19:56:15.643538+0000","last_fullsized":"2024-03-11T19:56:15.643538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:43.940638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205933+0000","last_change":"2024-03-11T19:55:45.702933+0000","last_active":"2024-03-11T19:55:51.205933+0000","last_peered":"2024-03-11T19:55:51.205933+0000","last_clean":"2024-03-11T19:55:51.205933+0000","last_became_active":"2024-03-11T19:55:45.702585+0000","last_became_peered":"2024-03-11T19:55:45.702585+0000","last_unstale":"2024-03-11T19:55:51.205933+0000","last_undegraded":"2024-03-11T19:55:51.205933+0000","last_fullsized":"2024-03-11T19:55:51.205933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:54:38.762156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205840+0000","last_change":"2024-03-11T19:55:50.102571+0000","last_active":"2024-03-11T19:55:51.205840+0000","last_peered":"2024-03-11T19:55:51.205840+0000","last_clean":"2024-03-11T19:55:51.205840+0000","last_became_active":"2024-03-11T19:55:50.102099+0000","last_became_peered":"2024-03-11T19:55:50.102099+0000","last_unstale":"2024-03-11T19:55:51.205840+0000","last_undegraded":"2024-03-11T19:55:51.205840+0000","last_fullsized":"2024-03-11T19:55:51.205840+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:17.631378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:16.655904+0000","last_change":"2024-03-11T19:55:44.671027+0000","last_active":"2024-03-11T19:56:16.655904+0000","last_peered":"2024-03-11T19:56:16.655904+0000","last_clean":"2024-03-11T19:56:16.655904+0000","last_became_active":"2024-03-11T19:55:44.669696+0000","last_became_peered":"2024-03-11T19:55:44.669696+0000","last_unstale":"2024-03-11T19:56:16.655904+0000","last_undegraded":"2024-03-11T19:56:16.655904+0000","last_fullsized":"2024-03-11T19:56:16.655904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:48:20.169638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.098501+0000","last_change":"2024-03-11T19:55:45.729119+0000","last_active":"2024-03-11T19:55:51.098501+0000","last_peered":"2024-03-11T19:55:51.098501+0000","last_clean":"2024-03-11T19:55:51.098501+0000","last_became_active":"2024-03-11T19:55:45.727442+0000","last_became_peered":"2024-03-11T19:55:45.727442+0000","last_unstale":"2024-03-11T19:55:51.098501+0000","last_undegraded":"2024-03-11T19:55:51.098501+0000","last_fullsized":"2024-03-11T19:55:51.098501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:14:51.483199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205678+0000","last_change":"2024-03-11T19:55:50.104326+0000","last_active":"2024-03-11T19:55:51.205678+0000","last_peered":"2024-03-11T19:55:51.205678+0000","last_clean":"2024-03-11T19:55:51.205678+0000","last_became_active":"2024-03-11T19:55:50.103881+0000","last_became_peered":"2024-03-11T19:55:50.103881+0000","last_unstale":"2024-03-11T19:55:51.205678+0000","last_undegraded":"2024-03-11T19:55:51.205678+0000","last_fullsized":"2024-03-11T19:55:51.205678+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:12:38.819591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.648465+0000","last_change":"2024-03-11T19:55:44.684900+0000","last_active":"2024-03-11T19:56:15.648465+0000","last_peered":"2024-03-11T19:56:15.648465+0000","last_clean":"2024-03-11T19:56:15.648465+0000","last_became_active":"2024-03-11T19:55:44.684590+0000","last_became_peered":"2024-03-11T19:55:44.684590+0000","last_unstale":"2024-03-11T19:56:15.648465+0000","last_undegraded":"2024-03-11T19:56:15.648465+0000","last_fullsized":"2024-03-11T19:56:15.648465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:07.744366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206521+0000","last_change":"2024-03-11T19:55:45.704787+0000","last_active":"2024-03-11T19:55:51.206521+0000","last_peered":"2024-03-11T19:55:51.206521+0000","last_clean":"2024-03-11T19:55:51.206521+0000","last_became_active":"2024-03-11T19:55:45.704405+0000","last_became_peered":"2024-03-11T19:55:45.704405+0000","last_unstale":"2024-03-11T19:55:51.206521+0000","last_undegraded":"2024-03-11T19:55:51.206521+0000","last_fullsized":"2024-03-11T19:55:51.206521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:58:05.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.211359+0000","last_change":"2024-03-11T19:55:50.104793+0000","last_active":"2024-03-11T19:55:51.211359+0000","last_peered":"2024-03-11T19:55:51.211359+0000","last_clean":"2024-03-11T19:55:51.211359+0000","last_became_active":"2024-03-11T19:55:50.104415+0000","last_became_peered":"2024-03-11T19:55:50.104415+0000","last_unstale":"2024-03-11T19:55:51.211359+0000","last_undegraded":"2024-03-11T19:55:51.211359+0000","last_fullsized":"2024-03-11T19:55:51.211359+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:53.307313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:16.652559+0000","last_change":"2024-03-11T19:55:44.669403+0000","last_active":"2024-03-11T19:56:16.652559+0000","last_peered":"2024-03-11T19:56:16.652559+0000","last_clean":"2024-03-11T19:56:16.652559+0000","last_became_active":"2024-03-11T19:55:44.668908+0000","last_became_peered":"2024-03-11T19:55:44.668908+0000","last_unstale":"2024-03-11T19:56:16.652559+0000","last_undegraded":"2024-03-11T19:56:16.652559+0000","last_fullsized":"2024-03-11T19:56:16.652559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:56:36.831506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205604+0000","last_change":"2024-03-11T19:55:45.700029+0000","last_active":"2024-03-11T19:55:51.205604+0000","last_peered":"2024-03-11T19:55:51.205604+0000","last_clean":"2024-03-11T19:55:51.205604+0000","last_became_active":"2024-03-11T19:55:45.697296+0000","last_became_peered":"2024-03-11T19:55:45.697296+0000","last_unstale":"2024-03-11T19:55:51.205604+0000","last_undegraded":"2024-03-11T19:55:51.205604+0000","last_fullsized":"2024-03-11T19:55:51.205604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:59:35.025845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.100065+0000","last_change":"2024-03-11T19:55:50.108645+0000","last_active":"2024-03-11T19:55:51.100065+0000","last_peered":"2024-03-11T19:55:51.100065+0000","last_clean":"2024-03-11T19:55:51.100065+0000","last_became_active":"2024-03-11T19:55:50.108182+0000","last_became_peered":"2024-03-11T19:55:50.108182+0000","last_unstale":"2024-03-11T19:55:51.100065+0000","last_undegraded":"2024-03-11T19:55:51.100065+0000","last_fullsized":"2024-03-11T19:55:51.100065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:00.756986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'16","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:16.660527+0000","last_change":"2024-03-11T19:55:44.677491+0000","last_active":"2024-03-11T19:56:16.660527+0000","last_peered":"2024-03-11T19:56:16.660527+0000","last_clean":"2024-03-11T19:56:16.660527+0000","last_became_active":"2024-03-11T19:55:44.677176+0000","last_became_peered":"2024-03-11T19:55:44.677176+0000","last_unstale":"2024-03-11T19:56:16.660527+0000","last_undegraded":"2024-03-11T19:56:16.660527+0000","last_fullsized":"2024-03-11T19:56:16.660527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:51.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205028+0000","last_change":"2024-03-11T19:55:45.703310+0000","last_active":"2024-03-11T19:55:51.205028+0000","last_peered":"2024-03-11T19:55:51.205028+0000","last_clean":"2024-03-11T19:55:51.205028+0000","last_became_active":"2024-03-11T19:55:45.702933+0000","last_became_peered":"2024-03-11T19:55:45.702933+0000","last_unstale":"2024-03-11T19:55:51.205028+0000","last_undegraded":"2024-03-11T19:55:51.205028+0000","last_fullsized":"2024-03-11T19:55:51.205028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:07:58.787949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205103+0000","last_change":"2024-03-11T19:55:50.103849+0000","last_active":"2024-03-11T19:55:51.205103+0000","last_peered":"2024-03-11T19:55:51.205103+0000","last_clean":"2024-03-11T19:55:51.205103+0000","last_became_active":"2024-03-11T19:55:50.103360+0000","last_became_peered":"2024-03-11T19:55:50.103360+0000","last_unstale":"2024-03-11T19:55:51.205103+0000","last_undegraded":"2024-03-11T19:55:51.205103+0000","last_fullsized":"2024-03-11T19:55:51.205103+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:38:05.151041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205907+0000","last_change":"2024-03-11T19:55:44.669839+0000","last_active":"2024-03-11T19:55:51.205907+0000","last_peered":"2024-03-11T19:55:51.205907+0000","last_clean":"2024-03-11T19:55:51.205907+0000","last_became_active":"2024-03-11T19:55:44.669494+0000","last_became_peered":"2024-03-11T19:55:44.669494+0000","last_unstale":"2024-03-11T19:55:51.205907+0000","last_undegraded":"2024-03-11T19:55:51.205907+0000","last_fullsized":"2024-03-11T19:55:51.205907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:00.313847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206362+0000","last_change":"2024-03-11T19:55:45.708929+0000","last_active":"2024-03-11T19:55:51.206362+0000","last_peered":"2024-03-11T19:55:51.206362+0000","last_clean":"2024-03-11T19:55:51.206362+0000","last_became_active":"2024-03-11T19:55:45.708141+0000","last_became_peered":"2024-03-11T19:55:45.708141+0000","last_unstale":"2024-03-11T19:55:51.206362+0000","last_undegraded":"2024-03-11T19:55:51.206362+0000","last_fullsized":"2024-03-11T19:55:51.206362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:01:19.756610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206284+0000","last_change":"2024-03-11T19:55:50.143916+0000","last_active":"2024-03-11T19:55:51.206284+0000","last_peered":"2024-03-11T19:55:51.206284+0000","last_clean":"2024-03-11T19:55:51.206284+0000","last_became_active":"2024-03-11T19:55:50.143341+0000","last_became_peered":"2024-03-11T19:55:50.143341+0000","last_unstale":"2024-03-11T19:55:51.206284+0000","last_undegraded":"2024-03-11T19:55:51.206284+0000","last_fullsized":"2024-03-11T19:55:51.206284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:40.637529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.207974+0000","last_change":"2024-03-11T19:55:44.688468+0000","last_active":"2024-03-11T19:55:51.207974+0000","last_peered":"2024-03-11T19:55:51.207974+0000","last_clean":"2024-03-11T19:55:51.207974+0000","last_became_active":"2024-03-11T19:55:44.688065+0000","last_became_peered":"2024-03-11T19:55:44.688065+0000","last_unstale":"2024-03-11T19:55:51.207974+0000","last_undegraded":"2024-03-11T19:55:51.207974+0000","last_fullsized":"2024-03-11T19:55:51.207974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:29:23.392749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094750+0000","last_change":"2024-03-11T19:55:45.728970+0000","last_active":"2024-03-11T19:55:51.094750+0000","last_peered":"2024-03-11T19:55:51.094750+0000","last_clean":"2024-03-11T19:55:51.094750+0000","last_became_active":"2024-03-11T19:55:45.728516+0000","last_became_peered":"2024-03-11T19:55:45.728516+0000","last_unstale":"2024-03-11T19:55:51.094750+0000","last_undegraded":"2024-03-11T19:55:51.094750+0000","last_fullsized":"2024-03-11T19:55:51.094750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:19.242610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.099289+0000","last_change":"2024-03-11T19:55:50.111884+0000","last_active":"2024-03-11T19:55:51.099289+0000","last_peered":"2024-03-11T19:55:51.099289+0000","last_clean":"2024-03-11T19:55:51.099289+0000","last_became_active":"2024-03-11T19:55:50.110802+0000","last_became_peered":"2024-03-11T19:55:50.110802+0000","last_unstale":"2024-03-11T19:55:51.099289+0000","last_undegraded":"2024-03-11T19:55:51.099289+0000","last_fullsized":"2024-03-11T19:55:51.099289+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:35.866368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206422+0000","last_change":"2024-03-11T19:55:44.677156+0000","last_active":"2024-03-11T19:55:51.206422+0000","last_peered":"2024-03-11T19:55:51.206422+0000","last_clean":"2024-03-11T19:55:51.206422+0000","last_became_active":"2024-03-11T19:55:44.676656+0000","last_became_peered":"2024-03-11T19:55:44.676656+0000","last_unstale":"2024-03-11T19:55:51.206422+0000","last_undegraded":"2024-03-11T19:55:51.206422+0000","last_fullsized":"2024-03-11T19:55:51.206422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:40:27.521669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094284+0000","last_change":"2024-03-11T19:55:45.715545+0000","last_active":"2024-03-11T19:55:51.094284+0000","last_peered":"2024-03-11T19:55:51.094284+0000","last_clean":"2024-03-11T19:55:51.094284+0000","last_became_active":"2024-03-11T19:55:45.715042+0000","last_became_peered":"2024-03-11T19:55:45.715042+0000","last_unstale":"2024-03-11T19:55:51.094284+0000","last_undegraded":"2024-03-11T19:55:51.094284+0000","last_fullsized":"2024-03-11T19:55:51.094284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:06:33.362235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.099946+0000","last_change":"2024-03-11T19:55:50.145603+0000","last_active":"2024-03-11T19:55:51.099946+0000","last_peered":"2024-03-11T19:55:51.099946+0000","last_clean":"2024-03-11T19:55:51.099946+0000","last_became_active":"2024-03-11T19:55:50.143689+0000","last_became_peered":"2024-03-11T19:55:50.143689+0000","last_unstale":"2024-03-11T19:55:51.099946+0000","last_undegraded":"2024-03-11T19:55:51.099946+0000","last_fullsized":"2024-03-11T19:55:51.099946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:57.289983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205253+0000","last_change":"2024-03-11T19:55:45.702011+0000","last_active":"2024-03-11T19:55:51.205253+0000","last_peered":"2024-03-11T19:55:51.205253+0000","last_clean":"2024-03-11T19:55:51.205253+0000","last_became_active":"2024-03-11T19:55:45.701673+0000","last_became_peered":"2024-03-11T19:55:45.701673+0000","last_unstale":"2024-03-11T19:55:51.205253+0000","last_undegraded":"2024-03-11T19:55:51.205253+0000","last_fullsized":"2024-03-11T19:55:51.205253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:17.028276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.097926+0000","last_change":"2024-03-11T19:55:44.689121+0000","last_active":"2024-03-11T19:55:51.097926+0000","last_peered":"2024-03-11T19:55:51.097926+0000","last_clean":"2024-03-11T19:55:51.097926+0000","last_became_active":"2024-03-11T19:55:44.688453+0000","last_became_peered":"2024-03-11T19:55:44.688453+0000","last_unstale":"2024-03-11T19:55:51.097926+0000","last_undegraded":"2024-03-11T19:55:51.097926+0000","last_fullsized":"2024-03-11T19:55:51.097926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:31.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206786+0000","last_change":"2024-03-11T19:55:50.098694+0000","last_active":"2024-03-11T19:55:51.206786+0000","last_peered":"2024-03-11T19:55:51.206786+0000","last_clean":"2024-03-11T19:55:51.206786+0000","last_became_active":"2024-03-11T19:55:50.098121+0000","last_became_peered":"2024-03-11T19:55:50.098121+0000","last_unstale":"2024-03-11T19:55:51.206786+0000","last_undegraded":"2024-03-11T19:55:51.206786+0000","last_fullsized":"2024-03-11T19:55:51.206786+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:29:11.046703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.209910+0000","last_change":"2024-03-11T19:55:45.723755+0000","last_active":"2024-03-11T19:55:51.209910+0000","last_peered":"2024-03-11T19:55:51.209910+0000","last_clean":"2024-03-11T19:55:51.209910+0000","last_became_active":"2024-03-11T19:55:45.723078+0000","last_became_peered":"2024-03-11T19:55:45.723078+0000","last_unstale":"2024-03-11T19:55:51.209910+0000","last_undegraded":"2024-03-11T19:55:51.209910+0000","last_fullsized":"2024-03-11T19:55:51.209910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:40:13.739875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.647667+0000","last_change":"2024-03-11T19:55:44.672644+0000","last_active":"2024-03-11T19:56:15.647667+0000","last_peered":"2024-03-11T19:56:15.647667+0000","last_clean":"2024-03-11T19:56:15.647667+0000","last_became_active":"2024-03-11T19:55:44.671645+0000","last_became_peered":"2024-03-11T19:55:44.671645+0000","last_unstale":"2024-03-11T19:56:15.647667+0000","last_undegraded":"2024-03-11T19:56:15.647667+0000","last_fullsized":"2024-03-11T19:56:15.647667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:25:46.637768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":358,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205407+0000","last_change":"2024-03-11T19:54:20.915891+0000","last_active":"2024-03-11T19:55:51.205407+0000","last_peered":"2024-03-11T19:55:51.205407+0000","last_clean":"2024-03-11T19:55:51.205407+0000","last_became_active":"2024-03-11T19:54:20.915450+0000","last_became_peered":"2024-03-11T19:54:20.915450+0000","last_unstale":"2024-03-11T19:55:51.205407+0000","last_undegraded":"2024-03-11T19:55:51.205407+0000","last_fullsized":"2024-03-11T19:55:51.205407+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_clean_scrub_stamp":"2024-03-11T19:50:29.052858+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:47.966076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206136+0000","last_change":"2024-03-11T19:55:50.144053+0000","last_active":"2024-03-11T19:55:51.206136+0000","last_peered":"2024-03-11T19:55:51.206136+0000","last_clean":"2024-03-11T19:55:51.206136+0000","last_became_active":"2024-03-11T19:55:50.143509+0000","last_became_peered":"2024-03-11T19:55:50.143509+0000","last_unstale":"2024-03-11T19:55:51.206136+0000","last_undegraded":"2024-03-11T19:55:51.206136+0000","last_fullsized":"2024-03-11T19:55:51.206136+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:57:15.448812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.097770+0000","last_change":"2024-03-11T19:55:45.712621+0000","last_active":"2024-03-11T19:55:51.097770+0000","last_peered":"2024-03-11T19:55:51.097770+0000","last_clean":"2024-03-11T19:55:51.097770+0000","last_became_active":"2024-03-11T19:55:45.712161+0000","last_became_peered":"2024-03-11T19:55:45.712161+0000","last_unstale":"2024-03-11T19:55:51.097770+0000","last_undegraded":"2024-03-11T19:55:51.097770+0000","last_fullsized":"2024-03-11T19:55:51.097770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:34:52.911944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:16.654456+0000","last_change":"2024-03-11T19:55:44.674606+0000","last_active":"2024-03-11T19:56:16.654456+0000","last_peered":"2024-03-11T19:56:16.654456+0000","last_clean":"2024-03-11T19:56:16.654456+0000","last_became_active":"2024-03-11T19:55:44.674355+0000","last_became_peered":"2024-03-11T19:55:44.674355+0000","last_unstale":"2024-03-11T19:56:16.654456+0000","last_undegraded":"2024-03-11T19:56:16.654456+0000","last_fullsized":"2024-03-11T19:56:16.654456+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:31:01.662582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206367+0000","last_change":"2024-03-11T19:55:50.101468+0000","last_active":"2024-03-11T19:55:51.206367+0000","last_peered":"2024-03-11T19:55:51.206367+0000","last_clean":"2024-03-11T19:55:51.206367+0000","last_became_active":"2024-03-11T19:55:50.101040+0000","last_became_peered":"2024-03-11T19:55:50.101040+0000","last_unstale":"2024-03-11T19:55:51.206367+0000","last_undegraded":"2024-03-11T19:55:51.206367+0000","last_fullsized":"2024-03-11T19:55:51.206367+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:55.354176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:17.661927+0000","last_change":"2024-03-11T19:55:44.681467+0000","last_active":"2024-03-11T19:56:17.661927+0000","last_peered":"2024-03-11T19:56:17.661927+0000","last_clean":"2024-03-11T19:56:17.661927+0000","last_became_active":"2024-03-11T19:55:44.681176+0000","last_became_peered":"2024-03-11T19:55:44.681176+0000","last_unstale":"2024-03-11T19:56:17.661927+0000","last_undegraded":"2024-03-11T19:56:17.661927+0000","last_fullsized":"2024-03-11T19:56:17.661927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:27:38.019116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.099373+0000","last_change":"2024-03-11T19:55:45.709798+0000","last_active":"2024-03-11T19:55:51.099373+0000","last_peered":"2024-03-11T19:55:51.099373+0000","last_clean":"2024-03-11T19:55:51.099373+0000","last_became_active":"2024-03-11T19:55:45.709277+0000","last_became_peered":"2024-03-11T19:55:45.709277+0000","last_unstale":"2024-03-11T19:55:51.099373+0000","last_undegraded":"2024-03-11T19:55:51.099373+0000","last_fullsized":"2024-03-11T19:55:51.099373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:58.852316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206542+0000","last_change":"2024-03-11T19:55:50.146601+0000","last_active":"2024-03-11T19:55:51.206542+0000","last_peered":"2024-03-11T19:55:51.206542+0000","last_clean":"2024-03-11T19:55:51.206542+0000","last_became_active":"2024-03-11T19:55:50.144584+0000","last_became_peered":"2024-03-11T19:55:50.144584+0000","last_unstale":"2024-03-11T19:55:51.206542+0000","last_undegraded":"2024-03-11T19:55:51.206542+0000","last_fullsized":"2024-03-11T19:55:51.206542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:00.365713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.210550+0000","last_change":"2024-03-11T19:55:44.689109+0000","last_active":"2024-03-11T19:55:51.210550+0000","last_peered":"2024-03-11T19:55:51.210550+0000","last_clean":"2024-03-11T19:55:51.210550+0000","last_became_active":"2024-03-11T19:55:44.687323+0000","last_became_peered":"2024-03-11T19:55:44.687323+0000","last_unstale":"2024-03-11T19:55:51.210550+0000","last_undegraded":"2024-03-11T19:55:51.210550+0000","last_fullsized":"2024-03-11T19:55:51.210550+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:24.613901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.209058+0000","last_change":"2024-03-11T19:55:45.726033+0000","last_active":"2024-03-11T19:55:51.209058+0000","last_peered":"2024-03-11T19:55:51.209058+0000","last_clean":"2024-03-11T19:55:51.209058+0000","last_became_active":"2024-03-11T19:55:45.725353+0000","last_became_peered":"2024-03-11T19:55:45.725353+0000","last_unstale":"2024-03-11T19:55:51.209058+0000","last_undegraded":"2024-03-11T19:55:51.209058+0000","last_fullsized":"2024-03-11T19:55:51.209058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:36:24.341870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094277+0000","last_change":"2024-03-11T19:55:50.089496+0000","last_active":"2024-03-11T19:55:51.094277+0000","last_peered":"2024-03-11T19:55:51.094277+0000","last_clean":"2024-03-11T19:55:51.094277+0000","last_became_active":"2024-03-11T19:55:50.089072+0000","last_became_peered":"2024-03-11T19:55:50.089072+0000","last_unstale":"2024-03-11T19:55:51.094277+0000","last_undegraded":"2024-03-11T19:55:51.094277+0000","last_fullsized":"2024-03-11T19:55:51.094277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:27:32.539333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.645423+0000","last_change":"2024-03-11T19:55:44.673147+0000","last_active":"2024-03-11T19:56:15.645423+0000","last_peered":"2024-03-11T19:56:15.645423+0000","last_clean":"2024-03-11T19:56:15.645423+0000","last_became_active":"2024-03-11T19:55:44.672787+0000","last_became_peered":"2024-03-11T19:55:44.672787+0000","last_unstale":"2024-03-11T19:56:15.645423+0000","last_undegraded":"2024-03-11T19:56:15.645423+0000","last_fullsized":"2024-03-11T19:56:15.645423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:11:23.864419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206757+0000","last_change":"2024-03-11T19:55:45.732376+0000","last_active":"2024-03-11T19:55:51.206757+0000","last_peered":"2024-03-11T19:55:51.206757+0000","last_clean":"2024-03-11T19:55:51.206757+0000","last_became_active":"2024-03-11T19:55:45.731956+0000","last_became_peered":"2024-03-11T19:55:45.731956+0000","last_unstale":"2024-03-11T19:55:51.206757+0000","last_undegraded":"2024-03-11T19:55:51.206757+0000","last_fullsized":"2024-03-11T19:55:51.206757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:43:48.423678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206190+0000","last_change":"2024-03-11T19:55:50.144080+0000","last_active":"2024-03-11T19:55:51.206190+0000","last_peered":"2024-03-11T19:55:51.206190+0000","last_clean":"2024-03-11T19:55:51.206190+0000","last_became_active":"2024-03-11T19:55:50.143585+0000","last_became_peered":"2024-03-11T19:55:50.143585+0000","last_unstale":"2024-03-11T19:55:51.206190+0000","last_undegraded":"2024-03-11T19:55:51.206190+0000","last_fullsized":"2024-03-11T19:55:51.206190+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:47:51.388823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.643980+0000","last_change":"2024-03-11T19:55:44.677354+0000","last_active":"2024-03-11T19:56:15.643980+0000","last_peered":"2024-03-11T19:56:15.643980+0000","last_clean":"2024-03-11T19:56:15.643980+0000","last_became_active":"2024-03-11T19:55:44.677013+0000","last_became_peered":"2024-03-11T19:55:44.677013+0000","last_unstale":"2024-03-11T19:56:15.643980+0000","last_undegraded":"2024-03-11T19:56:15.643980+0000","last_fullsized":"2024-03-11T19:56:15.643980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:50:48.855320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205541+0000","last_change":"2024-03-11T19:55:45.702278+0000","last_active":"2024-03-11T19:55:51.205541+0000","last_peered":"2024-03-11T19:55:51.205541+0000","last_clean":"2024-03-11T19:55:51.205541+0000","last_became_active":"2024-03-11T19:55:45.701843+0000","last_became_peered":"2024-03-11T19:55:45.701843+0000","last_unstale":"2024-03-11T19:55:51.205541+0000","last_undegraded":"2024-03-11T19:55:51.205541+0000","last_fullsized":"2024-03-11T19:55:51.205541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:55:28.351093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.211060+0000","last_change":"2024-03-11T19:55:50.106083+0000","last_active":"2024-03-11T19:55:51.211060+0000","last_peered":"2024-03-11T19:55:51.211060+0000","last_clean":"2024-03-11T19:55:51.211060+0000","last_became_active":"2024-03-11T19:55:50.105591+0000","last_became_peered":"2024-03-11T19:55:50.105591+0000","last_unstale":"2024-03-11T19:55:51.211060+0000","last_undegraded":"2024-03-11T19:55:51.211060+0000","last_fullsized":"2024-03-11T19:55:51.211060+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:34:58.713672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:18.641056+0000","last_change":"2024-03-11T19:55:44.688128+0000","last_active":"2024-03-11T19:56:18.641056+0000","last_peered":"2024-03-11T19:56:18.641056+0000","last_clean":"2024-03-11T19:56:18.641056+0000","last_became_active":"2024-03-11T19:55:44.684165+0000","last_became_peered":"2024-03-11T19:55:44.684165+0000","last_unstale":"2024-03-11T19:56:18.641056+0000","last_undegraded":"2024-03-11T19:56:18.641056+0000","last_fullsized":"2024-03-11T19:56:18.641056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:37:19.415960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205606+0000","last_change":"2024-03-11T19:55:45.718968+0000","last_active":"2024-03-11T19:55:51.205606+0000","last_peered":"2024-03-11T19:55:51.205606+0000","last_clean":"2024-03-11T19:55:51.205606+0000","last_became_active":"2024-03-11T19:55:45.718625+0000","last_became_peered":"2024-03-11T19:55:45.718625+0000","last_unstale":"2024-03-11T19:55:51.205606+0000","last_undegraded":"2024-03-11T19:55:51.205606+0000","last_fullsized":"2024-03-11T19:55:51.205606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:27.197227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205344+0000","last_change":"2024-03-11T19:55:50.097617+0000","last_active":"2024-03-11T19:55:51.205344+0000","last_peered":"2024-03-11T19:55:51.205344+0000","last_clean":"2024-03-11T19:55:51.205344+0000","last_became_active":"2024-03-11T19:55:50.097137+0000","last_became_peered":"2024-03-11T19:55:50.097137+0000","last_unstale":"2024-03-11T19:55:51.205344+0000","last_undegraded":"2024-03-11T19:55:51.205344+0000","last_fullsized":"2024-03-11T19:55:51.205344+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:19:20.995137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:16.642473+0000","last_change":"2024-03-11T19:55:44.669944+0000","last_active":"2024-03-11T19:56:16.642473+0000","last_peered":"2024-03-11T19:56:16.642473+0000","last_clean":"2024-03-11T19:56:16.642473+0000","last_became_active":"2024-03-11T19:55:44.669309+0000","last_became_peered":"2024-03-11T19:55:44.669309+0000","last_unstale":"2024-03-11T19:56:16.642473+0000","last_undegraded":"2024-03-11T19:56:16.642473+0000","last_fullsized":"2024-03-11T19:56:16.642473+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:06:41.489579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.208243+0000","last_change":"2024-03-11T19:55:45.731348+0000","last_active":"2024-03-11T19:55:51.208243+0000","last_peered":"2024-03-11T19:55:51.208243+0000","last_clean":"2024-03-11T19:55:51.208243+0000","last_became_active":"2024-03-11T19:55:45.730957+0000","last_became_peered":"2024-03-11T19:55:45.730957+0000","last_unstale":"2024-03-11T19:55:51.208243+0000","last_undegraded":"2024-03-11T19:55:51.208243+0000","last_fullsized":"2024-03-11T19:55:51.208243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:06:50.422511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206209+0000","last_change":"2024-03-11T19:55:50.144050+0000","last_active":"2024-03-11T19:55:51.206209+0000","last_peered":"2024-03-11T19:55:51.206209+0000","last_clean":"2024-03-11T19:55:51.206209+0000","last_became_active":"2024-03-11T19:55:50.143591+0000","last_became_peered":"2024-03-11T19:55:50.143591+0000","last_unstale":"2024-03-11T19:55:51.206209+0000","last_undegraded":"2024-03-11T19:55:51.206209+0000","last_fullsized":"2024-03-11T19:55:51.206209+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:02.006665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:15.644492+0000","last_change":"2024-03-11T19:55:44.671973+0000","last_active":"2024-03-11T19:56:15.644492+0000","last_peered":"2024-03-11T19:56:15.644492+0000","last_clean":"2024-03-11T19:56:15.644492+0000","last_became_active":"2024-03-11T19:55:44.671661+0000","last_became_peered":"2024-03-11T19:55:44.671661+0000","last_unstale":"2024-03-11T19:56:15.644492+0000","last_undegraded":"2024-03-11T19:56:15.644492+0000","last_fullsized":"2024-03-11T19:56:15.644492+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:35.747386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206466+0000","last_change":"2024-03-11T19:55:45.705545+0000","last_active":"2024-03-11T19:55:51.206466+0000","last_peered":"2024-03-11T19:55:51.206466+0000","last_clean":"2024-03-11T19:55:51.206466+0000","last_became_active":"2024-03-11T19:55:45.705083+0000","last_became_peered":"2024-03-11T19:55:45.705083+0000","last_unstale":"2024-03-11T19:55:51.206466+0000","last_undegraded":"2024-03-11T19:55:51.206466+0000","last_fullsized":"2024-03-11T19:55:51.206466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:02:58.219026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.204914+0000","last_change":"2024-03-11T19:55:50.097951+0000","last_active":"2024-03-11T19:55:51.204914+0000","last_peered":"2024-03-11T19:55:51.204914+0000","last_clean":"2024-03-11T19:55:51.204914+0000","last_became_active":"2024-03-11T19:55:50.097446+0000","last_became_peered":"2024-03-11T19:55:50.097446+0000","last_unstale":"2024-03-11T19:55:51.204914+0000","last_undegraded":"2024-03-11T19:55:51.204914+0000","last_fullsized":"2024-03-11T19:55:51.204914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:12.105801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:56:17.662364+0000","last_change":"2024-03-11T19:55:44.673136+0000","last_active":"2024-03-11T19:56:17.662364+0000","last_peered":"2024-03-11T19:56:17.662364+0000","last_clean":"2024-03-11T19:56:17.662364+0000","last_became_active":"2024-03-11T19:55:44.672467+0000","last_became_peered":"2024-03-11T19:55:44.672467+0000","last_unstale":"2024-03-11T19:56:17.662364+0000","last_undegraded":"2024-03-11T19:56:17.662364+0000","last_fullsized":"2024-03-11T19:56:17.662364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:46:05.931253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206896+0000","last_change":"2024-03-11T19:55:45.710561+0000","last_active":"2024-03-11T19:55:51.206896+0000","last_peered":"2024-03-11T19:55:51.206896+0000","last_clean":"2024-03-11T19:55:51.206896+0000","last_became_active":"2024-03-11T19:55:45.710164+0000","last_became_peered":"2024-03-11T19:55:45.710164+0000","last_unstale":"2024-03-11T19:55:51.206896+0000","last_undegraded":"2024-03-11T19:55:51.206896+0000","last_fullsized":"2024-03-11T19:55:51.206896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:00:33.977595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206167+0000","last_change":"2024-03-11T19:55:50.113390+0000","last_active":"2024-03-11T19:55:51.206167+0000","last_peered":"2024-03-11T19:55:51.206167+0000","last_clean":"2024-03-11T19:55:51.206167+0000","last_became_active":"2024-03-11T19:55:50.112656+0000","last_became_peered":"2024-03-11T19:55:50.112656+0000","last_unstale":"2024-03-11T19:55:51.206167+0000","last_undegraded":"2024-03-11T19:55:51.206167+0000","last_fullsized":"2024-03-11T19:55:51.206167+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:11.952585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205406+0000","last_change":"2024-03-11T19:55:44.684907+0000","last_active":"2024-03-11T19:55:51.205406+0000","last_peered":"2024-03-11T19:55:51.205406+0000","last_clean":"2024-03-11T19:55:51.205406+0000","last_became_active":"2024-03-11T19:55:44.684600+0000","last_became_peered":"2024-03-11T19:55:44.684600+0000","last_unstale":"2024-03-11T19:55:51.205406+0000","last_undegraded":"2024-03-11T19:55:51.205406+0000","last_fullsized":"2024-03-11T19:55:51.205406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:12:33.666756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206565+0000","last_change":"2024-03-11T19:55:45.723431+0000","last_active":"2024-03-11T19:55:51.206565+0000","last_peered":"2024-03-11T19:55:51.206565+0000","last_clean":"2024-03-11T19:55:51.206565+0000","last_became_active":"2024-03-11T19:55:45.723135+0000","last_became_peered":"2024-03-11T19:55:45.723135+0000","last_unstale":"2024-03-11T19:55:51.206565+0000","last_undegraded":"2024-03-11T19:55:51.206565+0000","last_fullsized":"2024-03-11T19:55:51.206565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:12:35.654094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.204643+0000","last_change":"2024-03-11T19:55:50.092876+0000","last_active":"2024-03-11T19:55:51.204643+0000","last_peered":"2024-03-11T19:55:51.204643+0000","last_clean":"2024-03-11T19:55:51.204643+0000","last_became_active":"2024-03-11T19:55:50.092478+0000","last_became_peered":"2024-03-11T19:55:50.092478+0000","last_unstale":"2024-03-11T19:55:51.204643+0000","last_undegraded":"2024-03-11T19:55:51.204643+0000","last_fullsized":"2024-03-11T19:55:51.204643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:39.248325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.099826+0000","last_change":"2024-03-11T19:55:44.688932+0000","last_active":"2024-03-11T19:55:51.099826+0000","last_peered":"2024-03-11T19:55:51.099826+0000","last_clean":"2024-03-11T19:55:51.099826+0000","last_became_active":"2024-03-11T19:55:44.687997+0000","last_became_peered":"2024-03-11T19:55:44.687997+0000","last_unstale":"2024-03-11T19:55:51.099826+0000","last_undegraded":"2024-03-11T19:55:51.099826+0000","last_fullsized":"2024-03-11T19:55:51.099826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:51:38.444939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206849+0000","last_change":"2024-03-11T19:55:45.702924+0000","last_active":"2024-03-11T19:55:51.206849+0000","last_peered":"2024-03-11T19:55:51.206849+0000","last_clean":"2024-03-11T19:55:51.206849+0000","last_became_active":"2024-03-11T19:55:45.702530+0000","last_became_peered":"2024-03-11T19:55:45.702530+0000","last_unstale":"2024-03-11T19:55:51.206849+0000","last_undegraded":"2024-03-11T19:55:51.206849+0000","last_fullsized":"2024-03-11T19:55:51.206849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:14.679964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205728+0000","last_change":"2024-03-11T19:55:45.706123+0000","last_active":"2024-03-11T19:55:51.205728+0000","last_peered":"2024-03-11T19:55:51.205728+0000","last_clean":"2024-03-11T19:55:51.205728+0000","last_became_active":"2024-03-11T19:55:45.705706+0000","last_became_peered":"2024-03-11T19:55:45.705706+0000","last_unstale":"2024-03-11T19:55:51.205728+0000","last_undegraded":"2024-03-11T19:55:51.205728+0000","last_fullsized":"2024-03-11T19:55:51.205728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:42:58.989823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094469+0000","last_change":"2024-03-11T19:55:45.703188+0000","last_active":"2024-03-11T19:55:51.094469+0000","last_peered":"2024-03-11T19:55:51.094469+0000","last_clean":"2024-03-11T19:55:51.094469+0000","last_became_active":"2024-03-11T19:55:45.702803+0000","last_became_peered":"2024-03-11T19:55:45.702803+0000","last_unstale":"2024-03-11T19:55:51.094469+0000","last_undegraded":"2024-03-11T19:55:51.094469+0000","last_fullsized":"2024-03-11T19:55:51.094469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:20.922963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094112+0000","last_change":"2024-03-11T19:55:45.728856+0000","last_active":"2024-03-11T19:55:51.094112+0000","last_peered":"2024-03-11T19:55:51.094112+0000","last_clean":"2024-03-11T19:55:51.094112+0000","last_became_active":"2024-03-11T19:55:45.728526+0000","last_became_peered":"2024-03-11T19:55:45.728526+0000","last_unstale":"2024-03-11T19:55:51.094112+0000","last_undegraded":"2024-03-11T19:55:51.094112+0000","last_fullsized":"2024-03-11T19:55:51.094112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:25:30.836073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206776+0000","last_change":"2024-03-11T19:55:45.725075+0000","last_active":"2024-03-11T19:55:51.206776+0000","last_peered":"2024-03-11T19:55:51.206776+0000","last_clean":"2024-03-11T19:55:51.206776+0000","last_became_active":"2024-03-11T19:55:45.724627+0000","last_became_peered":"2024-03-11T19:55:45.724627+0000","last_unstale":"2024-03-11T19:55:51.206776+0000","last_undegraded":"2024-03-11T19:55:51.206776+0000","last_fullsized":"2024-03-11T19:55:51.206776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:01.039660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.099167+0000","last_change":"2024-03-11T19:55:45.718879+0000","last_active":"2024-03-11T19:55:51.099167+0000","last_peered":"2024-03-11T19:55:51.099167+0000","last_clean":"2024-03-11T19:55:51.099167+0000","last_became_active":"2024-03-11T19:55:45.718535+0000","last_became_peered":"2024-03-11T19:55:45.718535+0000","last_unstale":"2024-03-11T19:55:51.099167+0000","last_undegraded":"2024-03-11T19:55:51.099167+0000","last_fullsized":"2024-03-11T19:55:51.099167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:48.236368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206528+0000","last_change":"2024-03-11T19:55:45.703432+0000","last_active":"2024-03-11T19:55:51.206528+0000","last_peered":"2024-03-11T19:55:51.206528+0000","last_clean":"2024-03-11T19:55:51.206528+0000","last_became_active":"2024-03-11T19:55:45.702669+0000","last_became_peered":"2024-03-11T19:55:45.702669+0000","last_unstale":"2024-03-11T19:55:51.206528+0000","last_undegraded":"2024-03-11T19:55:51.206528+0000","last_fullsized":"2024-03-11T19:55:51.206528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:02.496981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.094804+0000","last_change":"2024-03-11T19:55:45.708733+0000","last_active":"2024-03-11T19:55:51.094804+0000","last_peered":"2024-03-11T19:55:51.094804+0000","last_clean":"2024-03-11T19:55:51.094804+0000","last_became_active":"2024-03-11T19:55:45.708369+0000","last_became_peered":"2024-03-11T19:55:45.708369+0000","last_unstale":"2024-03-11T19:55:51.094804+0000","last_undegraded":"2024-03-11T19:55:51.094804+0000","last_fullsized":"2024-03-11T19:55:51.094804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:36.619802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.209557+0000","last_change":"2024-03-11T19:55:45.725698+0000","last_active":"2024-03-11T19:55:51.209557+0000","last_peered":"2024-03-11T19:55:51.209557+0000","last_clean":"2024-03-11T19:55:51.209557+0000","last_became_active":"2024-03-11T19:55:45.724872+0000","last_became_peered":"2024-03-11T19:55:45.724872+0000","last_unstale":"2024-03-11T19:55:51.209557+0000","last_undegraded":"2024-03-11T19:55:51.209557+0000","last_fullsized":"2024-03-11T19:55:51.209557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.982969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205251+0000","last_change":"2024-03-11T19:55:45.726923+0000","last_active":"2024-03-11T19:55:51.205251+0000","last_peered":"2024-03-11T19:55:51.205251+0000","last_clean":"2024-03-11T19:55:51.205251+0000","last_became_active":"2024-03-11T19:55:45.726589+0000","last_became_peered":"2024-03-11T19:55:45.726589+0000","last_unstale":"2024-03-11T19:55:51.205251+0000","last_undegraded":"2024-03-11T19:55:51.205251+0000","last_fullsized":"2024-03-11T19:55:51.205251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:29:15.197359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.208841+0000","last_change":"2024-03-11T19:55:45.723311+0000","last_active":"2024-03-11T19:55:51.208841+0000","last_peered":"2024-03-11T19:55:51.208841+0000","last_clean":"2024-03-11T19:55:51.208841+0000","last_became_active":"2024-03-11T19:55:45.722727+0000","last_became_peered":"2024-03-11T19:55:45.722727+0000","last_unstale":"2024-03-11T19:55:51.208841+0000","last_undegraded":"2024-03-11T19:55:51.208841+0000","last_fullsized":"2024-03-11T19:55:51.208841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:19:34.819665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.205888+0000","last_change":"2024-03-11T19:55:45.729141+0000","last_active":"2024-03-11T19:55:51.205888+0000","last_peered":"2024-03-11T19:55:51.205888+0000","last_clean":"2024-03-11T19:55:51.205888+0000","last_became_active":"2024-03-11T19:55:45.728781+0000","last_became_peered":"2024-03-11T19:55:45.728781+0000","last_unstale":"2024-03-11T19:55:51.205888+0000","last_undegraded":"2024-03-11T19:55:51.205888+0000","last_fullsized":"2024-03-11T19:55:51.205888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:02.939181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.098730+0000","last_change":"2024-03-11T19:55:45.723892+0000","last_active":"2024-03-11T19:55:51.098730+0000","last_peered":"2024-03-11T19:55:51.098730+0000","last_clean":"2024-03-11T19:55:51.098730+0000","last_became_active":"2024-03-11T19:55:45.723544+0000","last_became_peered":"2024-03-11T19:55:45.723544+0000","last_unstale":"2024-03-11T19:55:51.098730+0000","last_undegraded":"2024-03-11T19:55:51.098730+0000","last_fullsized":"2024-03-11T19:55:51.098730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:09:08.720854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-11T19:55:51.206470+0000","last_change":"2024-03-11T19:55:45.708494+0000","last_active":"2024-03-11T19:55:51.206470+0000","last_peered":"2024-03-11T19:55:51.206470+0000","last_clean":"2024-03-11T19:55:51.206470+0000","last_became_active":"2024-03-11T19:55:45.708114+0000","last_became_peered":"2024-03-11T19:55:45.708114+0000","last_unstale":"2024-03-11T19:55:51.206470+0000","last_undegraded":"2024-03-11T19:55:51.206470+0000","last_fullsized":"2024-03-11T19:55:51.206470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:28.809268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7984,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":25955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383701,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":478407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645337,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32760,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712392,"statfs":{"total":95995035648,"available":95961489408,"internally_reserved":0,"allocated":1622016,"data_stored":1073003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":477999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070503,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":477953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332140,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36892,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708260,"statfs":{"total":95995035648,"available":95957258240,"internally_reserved":0,"allocated":1593344,"data_stored":1069375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495663,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":481639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":479784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":482407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":477645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36900,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708252,"statfs":{"total":95995035648,"available":95957250048,"internally_reserved":0,"allocated":1589248,"data_stored":1067765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:56:22.503 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-11T19:56:22.503 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T19:56:22.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T19:56:22.990 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:23.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:23 smithi097 ceph-mon[31898]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:56:23.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:23 smithi097 ceph-mon[31898]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:56:23.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:23 smithi097 ceph-mon[31898]: pgmap v338: 129 pgs: 129 active+clean; 621 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 7.3 KiB/s wr, 15 op/s 2024-03-11T19:56:23.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:23 smithi114 ceph-mon[36376]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:56:23.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:23 smithi114 ceph-mon[36376]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:56:23.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:23 smithi114 ceph-mon[36376]: pgmap v338: 129 pgs: 129 active+clean; 621 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 7.3 KiB/s wr, 15 op/s 2024-03-11T19:56:24.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:23 smithi161 ceph-mon[36332]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:56:24.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:23 smithi161 ceph-mon[36332]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:56:24.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:23 smithi161 ceph-mon[36332]: pgmap v338: 129 pgs: 129 active+clean; 621 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 7.3 KiB/s wr, 15 op/s 2024-03-11T19:56:25.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.419+0000 7f1146200700 1 -- 172.21.15.97:0/4091790065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1140101570 msgr2=0x7f11401019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:25.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.419+0000 7f1146200700 1 --2- 172.21.15.97:0/4091790065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1140101570 0x7f11401019d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f112800aad0 tx=0x7f112800ade0 comp rx=0 tx=0).stop 2024-03-11T19:56:25.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.419+0000 7f1146200700 1 -- 172.21.15.97:0/4091790065 shutdown_connections 2024-03-11T19:56:25.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.419+0000 7f1146200700 1 --2- 172.21.15.97:0/4091790065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1140101f10 0x7f11401087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:25.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.419+0000 7f1146200700 1 --2- 172.21.15.97:0/4091790065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1140100370 0x7f1140100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:25.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.419+0000 7f1146200700 1 --2- 172.21.15.97:0/4091790065 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1140101570 0x7f11401019d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:25.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.419+0000 7f1146200700 1 -- 172.21.15.97:0/4091790065 >> 172.21.15.97:0/4091790065 conn(0x7f11400fb910 msgr2=0x7f11400fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:25.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.419+0000 7f1146200700 1 -- 172.21.15.97:0/4091790065 shutdown_connections 2024-03-11T19:56:25.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.419+0000 7f1146200700 1 -- 172.21.15.97:0/4091790065 wait complete. 2024-03-11T19:56:25.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.420+0000 7f1146200700 1 Processor -- start 2024-03-11T19:56:25.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.420+0000 7f1146200700 1 -- start start 2024-03-11T19:56:25.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f1146200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1140100370 0x7f1140197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:25.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f1146200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1140101570 0x7f11401980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:25.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f1146200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1140101f10 0x7f114019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:25.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f1146200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114019f6f0 con 0x7f1140101570 2024-03-11T19:56:25.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f1146200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114019f830 con 0x7f1140100370 2024-03-11T19:56:25.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f1146200700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114019f9c0 con 0x7f1140101f10 2024-03-11T19:56:25.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f1136a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1140101570 0x7f11401980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:25.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f1136a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1140101570 0x7f11401980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39988/0 (socket says 172.21.15.97:39988) 2024-03-11T19:56:25.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f1136a00700 1 -- 172.21.15.97:0/278150029 learned_addr learned my addr 172.21.15.97:0/278150029 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:25.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f113fe00700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1140101f10 0x7f114019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:25.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f113f400700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1140100370 0x7f1140197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:25.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f1136a00700 1 -- 172.21.15.97:0/278150029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1140101f10 msgr2=0x7f114019f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:25.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f1136a00700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1140101f10 0x7f114019f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:25.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f1136a00700 1 -- 172.21.15.97:0/278150029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1140100370 msgr2=0x7f1140197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:25.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.421+0000 7f1136a00700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1140100370 0x7f1140197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:25.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.422+0000 7f1136a00700 1 -- 172.21.15.97:0/278150029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112800a700 con 0x7f1140101570 2024-03-11T19:56:25.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.422+0000 7f113f400700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1140100370 0x7f1140197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:25.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.422+0000 7f1136a00700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1140101570 0x7f11401980a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1128005910 tx=0x7f1128004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:25.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.422+0000 7f113cc00700 1 -- 172.21.15.97:0/278150029 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1128005220 con 0x7f1140101570 2024-03-11T19:56:25.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.422+0000 7f113cc00700 1 -- 172.21.15.97:0/278150029 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f112800e450 con 0x7f1140101570 2024-03-11T19:56:25.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.422+0000 7f113cc00700 1 -- 172.21.15.97:0/278150029 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1128016680 con 0x7f1140101570 2024-03-11T19:56:25.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.422+0000 7f1146200700 1 -- 172.21.15.97:0/278150029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114019fc40 con 0x7f1140101570 2024-03-11T19:56:25.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.422+0000 7f1146200700 1 -- 172.21.15.97:0/278150029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11401a0000 con 0x7f1140101570 2024-03-11T19:56:25.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.423+0000 7f1146200700 1 -- 172.21.15.97:0/278150029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f114010b4d0 con 0x7f1140101570 2024-03-11T19:56:25.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.425+0000 7f113cc00700 1 -- 172.21.15.97:0/278150029 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f112800e5c0 con 0x7f1140101570 2024-03-11T19:56:25.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.427+0000 7f113cc00700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f11240a1c80 0x7f11240a4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:25.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.428+0000 7f113cc00700 1 -- 172.21.15.97:0/278150029 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1128012070 con 0x7f1140101570 2024-03-11T19:56:25.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.428+0000 7f113f400700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f11240a1c80 0x7f11240a4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:25.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.428+0000 7f113f400700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f11240a1c80 0x7f11240a4140 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f11401013d0 tx=0x7f1130009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:25.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.432+0000 7f113cc00700 1 -- 172.21.15.97:0/278150029 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1128098200 con 0x7f1140101570 2024-03-11T19:56:25.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.747+0000 7f1146200700 1 -- 172.21.15.97:0/278150029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f114010b6b0 con 0x7f1140101570 2024-03-11T19:56:25.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.752+0000 7f113cc00700 1 -- 172.21.15.97:0/278150029 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f112809b8c0 con 0x7f1140101570 2024-03-11T19:56:25.753 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-03-11T19:56:25.753 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDpYe9lQe2eLBAA1Z+Ddg7Mr70mErggpIgIlw== 2024-03-11T19:56:25.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.755+0000 7f1146200700 1 -- 172.21.15.97:0/278150029 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f11240a1c80 msgr2=0x7f11240a4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:25.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.755+0000 7f1146200700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f11240a1c80 0x7f11240a4140 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f11401013d0 tx=0x7f1130009260 comp rx=0 tx=0).stop 2024-03-11T19:56:25.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.755+0000 7f1146200700 1 -- 172.21.15.97:0/278150029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1140101570 msgr2=0x7f11401980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:25.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.755+0000 7f1146200700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1140101570 0x7f11401980a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1128005910 tx=0x7f1128004d60 comp rx=0 tx=0).stop 2024-03-11T19:56:25.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.755+0000 7f1146200700 1 -- 172.21.15.97:0/278150029 shutdown_connections 2024-03-11T19:56:25.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.755+0000 7f1146200700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f11240a1c80 0x7f11240a4140 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:25.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.755+0000 7f1146200700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1140101f10 0x7f114019f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:25.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.755+0000 7f1146200700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1140100370 0x7f1140197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:25.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.755+0000 7f1146200700 1 --2- 172.21.15.97:0/278150029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1140101570 0x7f11401980a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:25.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.755+0000 7f1146200700 1 -- 172.21.15.97:0/278150029 >> 172.21.15.97:0/278150029 conn(0x7f11400fb910 msgr2=0x7f11401069b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:25.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.755+0000 7f1146200700 1 -- 172.21.15.97:0/278150029 shutdown_connections 2024-03-11T19:56:25.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:25.755+0000 7f1146200700 1 -- 172.21.15.97:0/278150029 wait complete. 2024-03-11T19:56:25.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:25 smithi114 ceph-mon[36376]: pgmap v339: 129 pgs: 129 active+clean; 623 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.0 KiB/s wr, 21 op/s 2024-03-11T19:56:26.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:25 smithi161 ceph-mon[36332]: pgmap v339: 129 pgs: 129 active+clean; 623 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.0 KiB/s wr, 21 op/s 2024-03-11T19:56:26.136 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:25 smithi097 ceph-mon[31898]: pgmap v339: 129 pgs: 129 active+clean; 623 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.0 KiB/s wr, 21 op/s 2024-03-11T19:56:26.553 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:56:26.553 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T19:56:26.553 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T19:56:26.705 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T19:56:26.705 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T19:56:26.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T19:56:26.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:26 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/278150029' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T19:56:26.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:26 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/278150029' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T19:56:27.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:26 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/278150029' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T19:56:27.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:26 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/278150029' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T19:56:27.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:26 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/278150029' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T19:56:27.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:26 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/278150029' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T19:56:27.205 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:28.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:27 smithi161 ceph-mon[36332]: pgmap v340: 129 pgs: 129 active+clean; 623 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 8.0 KiB/s wr, 21 op/s 2024-03-11T19:56:28.169 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:27 smithi097 ceph-mon[31898]: pgmap v340: 129 pgs: 129 active+clean; 623 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 8.0 KiB/s wr, 21 op/s 2024-03-11T19:56:28.240 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:27 smithi114 ceph-mon[36376]: pgmap v340: 129 pgs: 129 active+clean; 623 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 8.0 KiB/s wr, 21 op/s 2024-03-11T19:56:28.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:28 smithi097 ceph-mon[31898]: pgmap v341: 129 pgs: 129 active+clean; 630 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.5 KiB/s wr, 25 op/s 2024-03-11T19:56:29.240 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:28 smithi114 ceph-mon[36376]: pgmap v341: 129 pgs: 129 active+clean; 630 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.5 KiB/s wr, 25 op/s 2024-03-11T19:56:29.286 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:28 smithi161 ceph-mon[36332]: pgmap v341: 129 pgs: 129 active+clean; 630 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.5 KiB/s wr, 25 op/s 2024-03-11T19:56:29.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.477+0000 7f583e200700 1 -- 172.21.15.97:0/3100096526 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5838108e90 msgr2=0x7f583810cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:29.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.477+0000 7f583e200700 1 --2- 172.21.15.97:0/3100096526 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5838108e90 0x7f583810cee0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5824009990 tx=0x7f5824009ca0 comp rx=0 tx=0).stop 2024-03-11T19:56:29.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.477+0000 7f583e200700 1 -- 172.21.15.97:0/3100096526 shutdown_connections 2024-03-11T19:56:29.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.477+0000 7f583e200700 1 --2- 172.21.15.97:0/3100096526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f583810d5c0 0x7f5838115190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:29.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.477+0000 7f583e200700 1 --2- 172.21.15.97:0/3100096526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58381084e0 0x7f58381088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:29.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.477+0000 7f583e200700 1 --2- 172.21.15.97:0/3100096526 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5838108e90 0x7f583810cee0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:29.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.477+0000 7f583e200700 1 -- 172.21.15.97:0/3100096526 >> 172.21.15.97:0/3100096526 conn(0x7f5838075440 msgr2=0x7f5838077860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:29.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.478+0000 7f583e200700 1 -- 172.21.15.97:0/3100096526 shutdown_connections 2024-03-11T19:56:29.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.478+0000 7f583e200700 1 -- 172.21.15.97:0/3100096526 wait complete. 2024-03-11T19:56:29.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.479+0000 7f583e200700 1 Processor -- start 2024-03-11T19:56:29.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.479+0000 7f583e200700 1 -- start start 2024-03-11T19:56:29.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.479+0000 7f583e200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58381084e0 0x7f58381a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:29.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.479+0000 7f583e200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5838108e90 0x7f58381a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:29.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.479+0000 7f583e200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f583810d5c0 0x7f583819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:29.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.479+0000 7f583e200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583819ebf0 con 0x7f58381084e0 2024-03-11T19:56:29.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.479+0000 7f583e200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583819ed30 con 0x7f583810d5c0 2024-03-11T19:56:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.479+0000 7f583e200700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583819eef0 con 0x7f5838108e90 2024-03-11T19:56:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.479+0000 7f5837e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f583810d5c0 0x7f583819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.480+0000 7f5837e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f583810d5c0 0x7f583819e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:60898/0 (socket says 172.21.15.97:60898) 2024-03-11T19:56:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.480+0000 7f5837e00700 1 -- 172.21.15.97:0/2259742626 learned_addr learned my addr 172.21.15.97:0/2259742626 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.480+0000 7f5837400700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58381084e0 0x7f58381a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.480+0000 7f5836a00700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5838108e90 0x7f58381a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.480+0000 7f5837e00700 1 -- 172.21.15.97:0/2259742626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5838108e90 msgr2=0x7f58381a4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.480+0000 7f5837e00700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5838108e90 0x7f58381a4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.480+0000 7f5837e00700 1 -- 172.21.15.97:0/2259742626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58381084e0 msgr2=0x7f58381a43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.480+0000 7f5837e00700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58381084e0 0x7f58381a43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.480+0000 7f5837e00700 1 -- 172.21.15.97:0/2259742626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58240095f0 con 0x7f583810d5c0 2024-03-11T19:56:29.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.480+0000 7f5837e00700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f583810d5c0 0x7f583819e590 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f583000ef30 tx=0x7f583000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:29.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.481+0000 7f582fe00700 1 -- 172.21.15.97:0/2259742626 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830018070 con 0x7f583810d5c0 2024-03-11T19:56:29.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.481+0000 7f583e200700 1 -- 172.21.15.97:0/2259742626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583819f170 con 0x7f583810d5c0 2024-03-11T19:56:29.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.481+0000 7f582fe00700 1 -- 172.21.15.97:0/2259742626 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f583000f040 con 0x7f583810d5c0 2024-03-11T19:56:29.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.481+0000 7f582fe00700 1 -- 172.21.15.97:0/2259742626 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583000ce00 con 0x7f583810d5c0 2024-03-11T19:56:29.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.481+0000 7f583e200700 1 -- 172.21.15.97:0/2259742626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58381ab470 con 0x7f583810d5c0 2024-03-11T19:56:29.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.482+0000 7f583e200700 1 -- 172.21.15.97:0/2259742626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58381108e0 con 0x7f583810d5c0 2024-03-11T19:56:29.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.484+0000 7f582fe00700 1 -- 172.21.15.97:0/2259742626 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f5830007500 con 0x7f583810d5c0 2024-03-11T19:56:29.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.485+0000 7f582fe00700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f581c0a1b50 0x7f581c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:29.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.485+0000 7f582fe00700 1 -- 172.21.15.97:0/2259742626 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f58300cd3c0 con 0x7f583810d5c0 2024-03-11T19:56:29.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.485+0000 7f5837400700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f581c0a1b50 0x7f581c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:29.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.486+0000 7f5837400700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f581c0a1b50 0x7f581c0a4010 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5828009c60 tx=0x7f5828009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:29.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.490+0000 7f582fe00700 1 -- 172.21.15.97:0/2259742626 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5830096260 con 0x7f583810d5c0 2024-03-11T19:56:29.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.766+0000 7f583e200700 1 -- 172.21.15.97:0/2259742626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f583819fac0 con 0x7f583810d5c0 2024-03-11T19:56:29.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.772+0000 7f582fe00700 1 -- 172.21.15.97:0/2259742626 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260661 (secure 0 0 0) 0x7f5830099920 con 0x7f583810d5c0 2024-03-11T19:56:29.775 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:56:29.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.778+0000 7f583e200700 1 -- 172.21.15.97:0/2259742626 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f581c0a1b50 msgr2=0x7f581c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:29.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.778+0000 7f583e200700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f581c0a1b50 0x7f581c0a4010 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5828009c60 tx=0x7f5828009260 comp rx=0 tx=0).stop 2024-03-11T19:56:29.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.778+0000 7f583e200700 1 -- 172.21.15.97:0/2259742626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f583810d5c0 msgr2=0x7f583819e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:29.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.778+0000 7f583e200700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f583810d5c0 0x7f583819e590 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f583000ef30 tx=0x7f583000c3a0 comp rx=0 tx=0).stop 2024-03-11T19:56:29.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.778+0000 7f583e200700 1 -- 172.21.15.97:0/2259742626 shutdown_connections 2024-03-11T19:56:29.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.778+0000 7f583e200700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f581c0a1b50 0x7f581c0a4010 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:29.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.778+0000 7f583e200700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f58381084e0 0x7f58381a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:29.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.778+0000 7f583e200700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f583810d5c0 0x7f583819e590 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:29.781 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.778+0000 7f583e200700 1 --2- 172.21.15.97:0/2259742626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5838108e90 0x7f58381a4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:29.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.778+0000 7f583e200700 1 -- 172.21.15.97:0/2259742626 >> 172.21.15.97:0/2259742626 conn(0x7f5838075440 msgr2=0x7f5838075f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:29.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.778+0000 7f583e200700 1 -- 172.21.15.97:0/2259742626 shutdown_connections 2024-03-11T19:56:29.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:29.778+0000 7f583e200700 1 -- 172.21.15.97:0/2259742626 wait complete. 2024-03-11T19:56:30.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:29 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2259742626' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T19:56:30.240 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:29 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2259742626' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T19:56:30.286 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:29 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2259742626' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T19:56:30.622 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":3389151509},{"type":"v1","addr":"172.21.15.97:6801","nonce":3389151509}]},"active_addr":"172.21.15.97:6801/3389151509","active_change":"2024-03-11T19:47:44.356581+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.97:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":3879069067}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":206978801}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":3511489140}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":839620487}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":1350515825}]}]} 2024-03-11T19:56:30.659 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T19:56:30.659 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T19:56:30.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T19:56:31.110 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:31.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:30 smithi097 ceph-mon[31898]: pgmap v342: 129 pgs: 129 active+clean; 630 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 KiB/s wr, 20 op/s 2024-03-11T19:56:31.240 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:30 smithi114 ceph-mon[36376]: pgmap v342: 129 pgs: 129 active+clean; 630 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 KiB/s wr, 20 op/s 2024-03-11T19:56:31.286 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:30 smithi161 ceph-mon[36332]: pgmap v342: 129 pgs: 129 active+clean; 630 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 KiB/s wr, 20 op/s 2024-03-11T19:56:33.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.419+0000 7f6cbd600700 1 -- 172.21.15.97:0/1842973478 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb810d5c0 msgr2=0x7f6cb8115190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:33.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.419+0000 7f6cbd600700 1 --2- 172.21.15.97:0/1842973478 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb810d5c0 0x7f6cb8115190 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0009f90 tx=0x7f6cb000b330 comp rx=0 tx=0).stop 2024-03-11T19:56:33.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.419+0000 7f6cbd600700 1 -- 172.21.15.97:0/1842973478 shutdown_connections 2024-03-11T19:56:33.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.419+0000 7f6cbd600700 1 --2- 172.21.15.97:0/1842973478 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6cb81084e0 0x7f6cb81088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:33.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.419+0000 7f6cbd600700 1 --2- 172.21.15.97:0/1842973478 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb810d5c0 0x7f6cb8115190 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:33.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.419+0000 7f6cbd600700 1 --2- 172.21.15.97:0/1842973478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cb8108e90 0x7f6cb810cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:33.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.419+0000 7f6cbd600700 1 -- 172.21.15.97:0/1842973478 >> 172.21.15.97:0/1842973478 conn(0x7f6cb8075440 msgr2=0x7f6cb8077860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:33.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.419+0000 7f6cbd600700 1 -- 172.21.15.97:0/1842973478 shutdown_connections 2024-03-11T19:56:33.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.419+0000 7f6cbd600700 1 -- 172.21.15.97:0/1842973478 wait complete. 2024-03-11T19:56:33.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.421+0000 7f6cbd600700 1 Processor -- start 2024-03-11T19:56:33.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.421+0000 7f6cbd600700 1 -- start start 2024-03-11T19:56:33.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.421+0000 7f6cbd600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6cb81084e0 0x7f6cb81a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:33.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.421+0000 7f6cbd600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb8108e90 0x7f6cb81a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:33.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.421+0000 7f6cbd600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cb810d5c0 0x7f6cb819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:33.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.421+0000 7f6cbd600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb819ee90 con 0x7f6cb81084e0 2024-03-11T19:56:33.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.421+0000 7f6cbd600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb819f000 con 0x7f6cb810d5c0 2024-03-11T19:56:33.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.421+0000 7f6cbd600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb819f1c0 con 0x7f6cb8108e90 2024-03-11T19:56:33.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.422+0000 7f6cafe00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb8108e90 0x7f6cb81a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:33.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.422+0000 7f6cb7400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cb810d5c0 0x7f6cb819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:33.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.422+0000 7f6cafe00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb8108e90 0x7f6cb81a4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:45210/0 (socket says 172.21.15.97:45210) 2024-03-11T19:56:33.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.422+0000 7f6cafe00700 1 -- 172.21.15.97:0/182617210 learned_addr learned my addr 172.21.15.97:0/182617210 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:33.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.422+0000 7f6cb6a00700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6cb81084e0 0x7f6cb81a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:33.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.422+0000 7f6cb7400700 1 -- 172.21.15.97:0/182617210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb8108e90 msgr2=0x7f6cb81a4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:33.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.422+0000 7f6cb7400700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb8108e90 0x7f6cb81a4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:33.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.422+0000 7f6cb7400700 1 -- 172.21.15.97:0/182617210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6cb81084e0 msgr2=0x7f6cb81a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:33.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.422+0000 7f6cb7400700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6cb81084e0 0x7f6cb81a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:33.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.422+0000 7f6cb7400700 1 -- 172.21.15.97:0/182617210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb0009c40 con 0x7f6cb810d5c0 2024-03-11T19:56:33.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.422+0000 7f6cafe00700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb8108e90 0x7f6cb81a4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:33.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.422+0000 7f6cb7400700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cb810d5c0 0x7f6cb819e6e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0009f90 tx=0x7f6cb0009420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:33.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.423+0000 7f6caf400700 1 -- 172.21.15.97:0/182617210 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb000e040 con 0x7f6cb810d5c0 2024-03-11T19:56:33.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.423+0000 7f6caf400700 1 -- 172.21.15.97:0/182617210 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6cb001d470 con 0x7f6cb810d5c0 2024-03-11T19:56:33.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.423+0000 7f6caf400700 1 -- 172.21.15.97:0/182617210 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb00079e0 con 0x7f6cb810d5c0 2024-03-11T19:56:33.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.423+0000 7f6cbd600700 1 -- 172.21.15.97:0/182617210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb819f440 con 0x7f6cb810d5c0 2024-03-11T19:56:33.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.423+0000 7f6cbd600700 1 -- 172.21.15.97:0/182617210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb81ab540 con 0x7f6cb810d5c0 2024-03-11T19:56:33.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.424+0000 7f6cbd600700 1 -- 172.21.15.97:0/182617210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb8110830 con 0x7f6cb810d5c0 2024-03-11T19:56:33.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.426+0000 7f6caf400700 1 -- 172.21.15.97:0/182617210 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f6cb001d5e0 con 0x7f6cb810d5c0 2024-03-11T19:56:33.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.427+0000 7f6caf400700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6c9c0a1bd0 0x7f6c9c0a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:33.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.427+0000 7f6caf400700 1 -- 172.21.15.97:0/182617210 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f6cb00ceca0 con 0x7f6cb810d5c0 2024-03-11T19:56:33.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.427+0000 7f6cb6a00700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6c9c0a1bd0 0x7f6c9c0a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:33.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.428+0000 7f6cb6a00700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6c9c0a1bd0 0x7f6c9c0a4090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009c60 tx=0x7f6ca8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:33.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.429+0000 7f6caf400700 1 -- 172.21.15.97:0/182617210 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6cb0097b40 con 0x7f6cb810d5c0 2024-03-11T19:56:33.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.653+0000 7f6cbd600700 1 -- 172.21.15.97:0/182617210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cb81099c0 con 0x7f6cb810d5c0 2024-03-11T19:56:33.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.655+0000 7f6caf400700 1 -- 172.21.15.97:0/182617210 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24152 (secure 0 0 0) 0x7f6cb009b200 con 0x7f6cb810d5c0 2024-03-11T19:56:33.657 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:56:33.657 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":89,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:55:50.540865+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:56:33.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.658+0000 7f6cbd600700 1 -- 172.21.15.97:0/182617210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6c9c0a1bd0 msgr2=0x7f6c9c0a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:33.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.658+0000 7f6cbd600700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6c9c0a1bd0 0x7f6c9c0a4090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009c60 tx=0x7f6ca8009260 comp rx=0 tx=0).stop 2024-03-11T19:56:33.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.658+0000 7f6cbd600700 1 -- 172.21.15.97:0/182617210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cb810d5c0 msgr2=0x7f6cb819e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:33.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.658+0000 7f6cbd600700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cb810d5c0 0x7f6cb819e6e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0009f90 tx=0x7f6cb0009420 comp rx=0 tx=0).stop 2024-03-11T19:56:33.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.658+0000 7f6cbd600700 1 -- 172.21.15.97:0/182617210 shutdown_connections 2024-03-11T19:56:33.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.658+0000 7f6cbd600700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f6c9c0a1bd0 0x7f6c9c0a4090 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:33.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.658+0000 7f6cbd600700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6cb81084e0 0x7f6cb81a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:33.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.658+0000 7f6cbd600700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cb810d5c0 0x7f6cb819e6e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:33.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.658+0000 7f6cbd600700 1 --2- 172.21.15.97:0/182617210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6cb8108e90 0x7f6cb81a4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:33.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.658+0000 7f6cbd600700 1 -- 172.21.15.97:0/182617210 >> 172.21.15.97:0/182617210 conn(0x7f6cb8075440 msgr2=0x7f6cb8076000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:33.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.658+0000 7f6cbd600700 1 -- 172.21.15.97:0/182617210 shutdown_connections 2024-03-11T19:56:33.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:33.659+0000 7f6cbd600700 1 -- 172.21.15.97:0/182617210 wait complete. 2024-03-11T19:56:33.678 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:33 smithi097 ceph-mon[31898]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.0 KiB/s wr, 22 op/s 2024-03-11T19:56:33.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:33 smithi114 ceph-mon[36376]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.0 KiB/s wr, 22 op/s 2024-03-11T19:56:34.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:33 smithi161 ceph-mon[36332]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.0 KiB/s wr, 22 op/s 2024-03-11T19:56:34.461 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T19:56:34.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T19:56:34.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:34 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/182617210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:34.898 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:34.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:34 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/182617210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:35.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:34 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/182617210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:35.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:35 smithi097 ceph-mon[31898]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-03-11T19:56:35.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:35 smithi114 ceph-mon[36376]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-03-11T19:56:36.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:35 smithi161 ceph-mon[36332]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 2.7 KiB/s wr, 15 op/s 2024-03-11T19:56:37.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.142+0000 7f6018e00700 1 -- 172.21.15.97:0/1807107908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6014108e90 msgr2=0x7f601410cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:37.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.142+0000 7f6018e00700 1 --2- 172.21.15.97:0/1807107908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6014108e90 0x7f601410cee0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009990 tx=0x7f5ffc009ca0 comp rx=0 tx=0).stop 2024-03-11T19:56:37.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.142+0000 7f6018e00700 1 -- 172.21.15.97:0/1807107908 shutdown_connections 2024-03-11T19:56:37.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.142+0000 7f6018e00700 1 --2- 172.21.15.97:0/1807107908 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f60141084e0 0x7f60141088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:37.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.142+0000 7f6018e00700 1 --2- 172.21.15.97:0/1807107908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f601410d5c0 0x7f6014115190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:37.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.142+0000 7f6018e00700 1 --2- 172.21.15.97:0/1807107908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6014108e90 0x7f601410cee0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:37.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.142+0000 7f6018e00700 1 -- 172.21.15.97:0/1807107908 >> 172.21.15.97:0/1807107908 conn(0x7f6014075440 msgr2=0x7f6014077860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:37.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.143+0000 7f6018e00700 1 -- 172.21.15.97:0/1807107908 shutdown_connections 2024-03-11T19:56:37.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.143+0000 7f6018e00700 1 -- 172.21.15.97:0/1807107908 wait complete. 2024-03-11T19:56:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.144+0000 7f6018e00700 1 Processor -- start 2024-03-11T19:56:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.144+0000 7f6018e00700 1 -- start start 2024-03-11T19:56:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.144+0000 7f6018e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60141084e0 0x7f60141a4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.144+0000 7f6018e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6014108e90 0x7f60141a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.144+0000 7f6018e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601410d5c0 0x7f601419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.144+0000 7f6018e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601419ec90 con 0x7f6014108e90 2024-03-11T19:56:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.144+0000 7f6018e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601419edd0 con 0x7f601410d5c0 2024-03-11T19:56:37.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.144+0000 7f6018e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601419ef90 con 0x7f60141084e0 2024-03-11T19:56:37.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.145+0000 7f600be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6014108e90 0x7f60141a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:37.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.145+0000 7f6012800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601410d5c0 0x7f601419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:37.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.145+0000 7f600be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6014108e90 0x7f60141a4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53484/0 (socket says 172.21.15.97:53484) 2024-03-11T19:56:37.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.145+0000 7f600be00700 1 -- 172.21.15.97:0/3193432651 learned_addr learned my addr 172.21.15.97:0/3193432651 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:37.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.145+0000 7f6011e00700 1 --2- 172.21.15.97:0/3193432651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60141084e0 0x7f60141a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:37.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.145+0000 7f6012800700 1 -- 172.21.15.97:0/3193432651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60141084e0 msgr2=0x7f60141a4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:37.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.145+0000 7f6012800700 1 --2- 172.21.15.97:0/3193432651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60141084e0 0x7f60141a4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:37.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.145+0000 7f6012800700 1 -- 172.21.15.97:0/3193432651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6014108e90 msgr2=0x7f60141a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:37.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.145+0000 7f6012800700 1 --2- 172.21.15.97:0/3193432651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6014108e90 0x7f60141a4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:37.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.145+0000 7f6012800700 1 -- 172.21.15.97:0/3193432651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc0095f0 con 0x7f601410d5c0 2024-03-11T19:56:37.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.146+0000 7f6012800700 1 --2- 172.21.15.97:0/3193432651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601410d5c0 0x7f601419e630 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f600c00efc0 tx=0x7f600c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:37.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.146+0000 7f600aa00700 1 -- 172.21.15.97:0/3193432651 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600c018070 con 0x7f601410d5c0 2024-03-11T19:56:37.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.146+0000 7f600aa00700 1 -- 172.21.15.97:0/3193432651 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f600c00f040 con 0x7f601410d5c0 2024-03-11T19:56:37.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.146+0000 7f600aa00700 1 -- 172.21.15.97:0/3193432651 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600c00cd10 con 0x7f601410d5c0 2024-03-11T19:56:37.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.146+0000 7f6018e00700 1 -- 172.21.15.97:0/3193432651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601419f210 con 0x7f601410d5c0 2024-03-11T19:56:37.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.146+0000 7f6018e00700 1 -- 172.21.15.97:0/3193432651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60141ab4c0 con 0x7f601410d5c0 2024-03-11T19:56:37.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.147+0000 7f6018e00700 1 -- 172.21.15.97:0/3193432651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60141108e0 con 0x7f601410d5c0 2024-03-11T19:56:37.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.149+0000 7f600aa00700 1 -- 172.21.15.97:0/3193432651 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f600c004ad0 con 0x7f601410d5c0 2024-03-11T19:56:37.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.150+0000 7f600aa00700 1 --2- 172.21.15.97:0/3193432651 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5ff80a1b50 0x7f5ff80a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:37.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.151+0000 7f600aa00700 1 -- 172.21.15.97:0/3193432651 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f600c0cd460 con 0x7f601410d5c0 2024-03-11T19:56:37.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.151+0000 7f6011e00700 1 --2- 172.21.15.97:0/3193432651 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5ff80a1b50 0x7f5ff80a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:37.153 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.151+0000 7f6011e00700 1 --2- 172.21.15.97:0/3193432651 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5ff80a1b50 0x7f5ff80a4010 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6004009c60 tx=0x7f6004009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:37.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.153+0000 7f600aa00700 1 -- 172.21.15.97:0/3193432651 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f600c096300 con 0x7f601410d5c0 2024-03-11T19:56:37.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.373+0000 7f6018e00700 1 -- 172.21.15.97:0/3193432651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f601419fb10 con 0x7f601410d5c0 2024-03-11T19:56:37.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.375+0000 7f600aa00700 1 -- 172.21.15.97:0/3193432651 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24152 (secure 0 0 0) 0x7f600c04de70 con 0x7f601410d5c0 2024-03-11T19:56:37.378 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:56:37.378 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":89,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:55:50.540865+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:56:37.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.378+0000 7f6018e00700 1 -- 172.21.15.97:0/3193432651 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5ff80a1b50 msgr2=0x7f5ff80a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:37.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.378+0000 7f6018e00700 1 --2- 172.21.15.97:0/3193432651 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5ff80a1b50 0x7f5ff80a4010 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6004009c60 tx=0x7f6004009260 comp rx=0 tx=0).stop 2024-03-11T19:56:37.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.378+0000 7f6018e00700 1 -- 172.21.15.97:0/3193432651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601410d5c0 msgr2=0x7f601419e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:37.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.378+0000 7f6018e00700 1 --2- 172.21.15.97:0/3193432651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601410d5c0 0x7f601419e630 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f600c00efc0 tx=0x7f600c00c3a0 comp rx=0 tx=0).stop 2024-03-11T19:56:37.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.379+0000 7f6018e00700 1 -- 172.21.15.97:0/3193432651 shutdown_connections 2024-03-11T19:56:37.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.379+0000 7f6018e00700 1 --2- 172.21.15.97:0/3193432651 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5ff80a1b50 0x7f5ff80a4010 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:37.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.379+0000 7f6018e00700 1 --2- 172.21.15.97:0/3193432651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f601410d5c0 0x7f601419e630 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:37.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.379+0000 7f6018e00700 1 --2- 172.21.15.97:0/3193432651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60141084e0 0x7f60141a4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:37.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.379+0000 7f6018e00700 1 --2- 172.21.15.97:0/3193432651 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6014108e90 0x7f60141a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:37.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.379+0000 7f6018e00700 1 -- 172.21.15.97:0/3193432651 >> 172.21.15.97:0/3193432651 conn(0x7f6014075440 msgr2=0x7f60140777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:37.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.379+0000 7f6018e00700 1 -- 172.21.15.97:0/3193432651 shutdown_connections 2024-03-11T19:56:37.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:37.379+0000 7f6018e00700 1 -- 172.21.15.97:0/3193432651 wait complete. 2024-03-11T19:56:37.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:37 smithi097 ceph-mon[31898]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-03-11T19:56:37.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:37 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3193432651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:37.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:37 smithi114 ceph-mon[36376]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-03-11T19:56:37.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:37 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3193432651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:38.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:37 smithi161 ceph-mon[36332]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-03-11T19:56:38.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:37 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3193432651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:56:38.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T19:56:38.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T19:56:38.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T19:56:38.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T19:56:38.186 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T19:56:38.186 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T19:56:38.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T19:56:38.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T19:56:38.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-11T19:56:38.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-11T19:56:38.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-11T19:56:38.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-11T19:56:39.173 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:39.582 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:39.583 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:39.587 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:39.604 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:39.615 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:39.618 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:39.618 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:39.625 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:39.630 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:39.776 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:39 smithi097 ceph-mon[31898]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-03-11T19:56:39.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:39 smithi114 ceph-mon[36376]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-03-11T19:56:40.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:39 smithi161 ceph-mon[36332]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2024-03-11T19:56:40.156 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:40.159 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:56:41.868 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:41 smithi097 ceph-mon[31898]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-03-11T19:56:41.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:41 smithi114 ceph-mon[36376]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-03-11T19:56:42.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:41 smithi161 ceph-mon[36332]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-03-11T19:56:43.897 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:43 smithi097 ceph-mon[31898]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-11T19:56:43.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:43 smithi114 ceph-mon[36376]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-11T19:56:44.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:43 smithi161 ceph-mon[36332]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-11T19:56:44.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.337+0000 7f58a1c00700 1 -- 172.21.15.97:0/560047558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f589c101530 msgr2=0x7f589c10d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:44.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.337+0000 7f58a1c00700 1 --2- 172.21.15.97:0/560047558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f589c101530 0x7f589c10d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5894009f90 tx=0x7f589400b330 comp rx=0 tx=0).stop 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.337+0000 7f58a1c00700 1 -- 172.21.15.97:0/560047558 shutdown_connections 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.337+0000 7f58a1c00700 1 --2- 172.21.15.97:0/560047558 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f589c106b80 0x7f589c106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.337+0000 7f58a1c00700 1 --2- 172.21.15.97:0/560047558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f589c101530 0x7f589c10d9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.337+0000 7f58a1c00700 1 --2- 172.21.15.97:0/560047558 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f589c100b70 0x7f589c100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.337+0000 7f58a1c00700 1 -- 172.21.15.97:0/560047558 >> 172.21.15.97:0/560047558 conn(0x7f589c077aa0 msgr2=0x7f589c077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.337+0000 7f58a1c00700 1 -- 172.21.15.97:0/560047558 shutdown_connections 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.337+0000 7f58a1c00700 1 -- 172.21.15.97:0/560047558 wait complete. 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.338+0000 7f58a1c00700 1 Processor -- start 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.338+0000 7f58a1c00700 1 -- start start 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.339+0000 7f58a1c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f589c100b70 0x7f589c19a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.339+0000 7f58a1c00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f589c101530 0x7f589c19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.339+0000 7f58a1c00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f589c106b80 0x7f589c19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.339+0000 7f58a1c00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589c19f310 con 0x7f589c101530 2024-03-11T19:56:44.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.339+0000 7f58a1c00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589c19f480 con 0x7f589c100b70 2024-03-11T19:56:44.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.339+0000 7f58a1c00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589c19f640 con 0x7f589c106b80 2024-03-11T19:56:44.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.339+0000 7f589a000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f589c101530 0x7f589c19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:44.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.339+0000 7f589a000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f589c101530 0x7f589c19a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56340/0 (socket says 172.21.15.97:56340) 2024-03-11T19:56:44.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.339+0000 7f589a000700 1 -- 172.21.15.97:0/492970210 learned_addr learned my addr 172.21.15.97:0/492970210 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:44.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.339+0000 7f589aa00700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f589c100b70 0x7f589c19a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:44.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.339+0000 7f589b400700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f589c106b80 0x7f589c19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:44.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.340+0000 7f589b400700 1 -- 172.21.15.97:0/492970210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f589c100b70 msgr2=0x7f589c19a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:44.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.340+0000 7f589b400700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f589c100b70 0x7f589c19a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:44.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.340+0000 7f589b400700 1 -- 172.21.15.97:0/492970210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f589c101530 msgr2=0x7f589c19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:44.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.340+0000 7f589b400700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f589c101530 0x7f589c19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:44.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.340+0000 7f589b400700 1 -- 172.21.15.97:0/492970210 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c0095f0 con 0x7f589c106b80 2024-03-11T19:56:44.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.340+0000 7f589aa00700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f589c100b70 0x7f589c19a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:44.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.340+0000 7f589b400700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f589c106b80 0x7f589c19ed00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f589400b6c0 tx=0x7f5894009460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:44.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.340+0000 7f5893400700 1 -- 172.21.15.97:0/492970210 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589400e040 con 0x7f589c106b80 2024-03-11T19:56:44.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.341+0000 7f5893400700 1 -- 172.21.15.97:0/492970210 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f589401d470 con 0x7f589c106b80 2024-03-11T19:56:44.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.341+0000 7f58a1c00700 1 -- 172.21.15.97:0/492970210 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5894009c40 con 0x7f589c106b80 2024-03-11T19:56:44.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.341+0000 7f5893400700 1 -- 172.21.15.97:0/492970210 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58940079e0 con 0x7f589c106b80 2024-03-11T19:56:44.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.341+0000 7f58a1c00700 1 -- 172.21.15.97:0/492970210 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f589c1a6fb0 con 0x7f589c106b80 2024-03-11T19:56:44.389 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.342+0000 7f58a1c00700 1 -- 172.21.15.97:0/492970210 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f589c110ee0 con 0x7f589c106b80 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.344+0000 7f5893400700 1 -- 172.21.15.97:0/492970210 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f589401d5e0 con 0x7f589c106b80 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.345+0000 7f5893400700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f58800a1b50 0x7f58800a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.345+0000 7f5893400700 1 -- 172.21.15.97:0/492970210 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f58940cf240 con 0x7f589c106b80 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.345+0000 7f589aa00700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f58800a1b50 0x7f58800a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.345+0000 7f5893400700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703] conn(0x7f58800ac810 0x7f58800aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.345+0000 7f5893400700 1 -- 172.21.15.97:0/492970210 --> [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58800af320 con 0x7f58800ac810 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.345+0000 7f5893400700 1 -- 172.21.15.97:0/492970210 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58940cf5b0 con 0x7f589c106b80 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.346+0000 7f589a000700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703] conn(0x7f58800ac810 0x7f58800aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.346+0000 7f589aa00700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f58800a1b50 0x7f58800a4010 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f588c005fd0 tx=0x7f588c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.346+0000 7f589a000700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703] conn(0x7f58800ac810 0x7f58800aec50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.347+0000 7f5893400700 1 -- 172.21.15.97:0/492970210 <== osd.11 v2:172.21.15.161:6824/1523383703 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f58800af320 con 0x7f58800ac810 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.365+0000 7f58a1c00700 1 -- 172.21.15.97:0/492970210 --> [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f589c100660 con 0x7f58800ac810 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.366+0000 7f5893400700 1 -- 172.21.15.97:0/492970210 <== osd.11 v2:172.21.15.161:6824/1523383703 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f589c100660 con 0x7f58800ac810 2024-03-11T19:56:44.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.367+0000 7f58a1c00700 1 -- 172.21.15.97:0/492970210 >> [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703] conn(0x7f58800ac810 msgr2=0x7f58800aec50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:44.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.367+0000 7f58a1c00700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703] conn(0x7f58800ac810 0x7f58800aec50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:44.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.367+0000 7f58a1c00700 1 -- 172.21.15.97:0/492970210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f58800a1b50 msgr2=0x7f58800a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:44.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.367+0000 7f58a1c00700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f58800a1b50 0x7f58800a4010 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f588c005fd0 tx=0x7f588c009310 comp rx=0 tx=0).stop 2024-03-11T19:56:44.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.367+0000 7f58a1c00700 1 -- 172.21.15.97:0/492970210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f589c106b80 msgr2=0x7f589c19ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:44.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.367+0000 7f58a1c00700 1 --2- 172.21.15.97:0/492970210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f589c106b80 0x7f589c19ed00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f589400b6c0 tx=0x7f5894009460 comp rx=0 tx=0).stop 2024-03-11T19:56:44.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.367+0000 7f589b400700 1 -- 172.21.15.97:0/492970210 reap_dead start 2024-03-11T19:56:44.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.367+0000 7f58a1c00700 1 -- 172.21.15.97:0/492970210 shutdown_connections 2024-03-11T19:56:44.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.367+0000 7f58a1c00700 1 -- 172.21.15.97:0/492970210 >> 172.21.15.97:0/492970210 conn(0x7f589c077aa0 msgr2=0x7f589c0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:44.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.367+0000 7f58a1c00700 1 -- 172.21.15.97:0/492970210 shutdown_connections 2024-03-11T19:56:44.391 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:44.367+0000 7f58a1c00700 1 -- 172.21.15.97:0/492970210 wait complete. 2024-03-11T19:56:44.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-03-11T19:56:44.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-11T19:56:44.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-03-11T19:56:44.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-11T19:56:44.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-11T19:56:44.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-03-11T19:56:44.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-03-11T19:56:44.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-11T19:56:44.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:44 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-11T19:56:44.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:44 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-03-11T19:56:44.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:44 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-11T19:56:44.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:44 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-03-11T19:56:44.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:44 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-11T19:56:44.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:44 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-11T19:56:44.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:44 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-03-11T19:56:44.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:44 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-03-11T19:56:44.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:44 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-11T19:56:44.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:44 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-11T19:56:45.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:44 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-03-11T19:56:45.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:44 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-11T19:56:45.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:44 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-03-11T19:56:45.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:44 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-11T19:56:45.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:44 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-11T19:56:45.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:44 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-03-11T19:56:45.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:44 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-03-11T19:56:45.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:44 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-11T19:56:45.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:44 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-11T19:56:45.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.085+0000 7f038ec00700 1 -- 172.21.15.97:0/792569443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03881028a0 msgr2=0x7f03881068f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:45.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.085+0000 7f038ec00700 1 --2- 172.21.15.97:0/792569443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03881028a0 0x7f03881068f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0370009990 tx=0x7f0370009ca0 comp rx=0 tx=0).stop 2024-03-11T19:56:45.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.085+0000 7f038ec00700 1 -- 172.21.15.97:0/792569443 shutdown_connections 2024-03-11T19:56:45.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.085+0000 7f038ec00700 1 --2- 172.21.15.97:0/792569443 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0388106fd0 0x7f038810eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:45.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.085+0000 7f038ec00700 1 --2- 172.21.15.97:0/792569443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0388101ef0 0x7f03881022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:45.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.085+0000 7f038ec00700 1 --2- 172.21.15.97:0/792569443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f03881028a0 0x7f03881068f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:45.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.085+0000 7f038ec00700 1 -- 172.21.15.97:0/792569443 >> 172.21.15.97:0/792569443 conn(0x7f03880fd740 msgr2=0x7f03880ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:45.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.085+0000 7f038ec00700 1 -- 172.21.15.97:0/792569443 shutdown_connections 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.085+0000 7f038ec00700 1 -- 172.21.15.97:0/792569443 wait complete. 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.086+0000 7f038ec00700 1 Processor -- start 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.086+0000 7f038ec00700 1 -- start start 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f038ec00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0388101ef0 0x7f0388198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f038ec00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03881028a0 0x7f0388198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f038ec00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0388106fd0 0x7f038819cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f038ec00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038819d0d0 con 0x7f0388101ef0 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f038ec00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038819d240 con 0x7f03881028a0 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f038ec00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038819d400 con 0x7f0388106fd0 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f0387e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0388101ef0 0x7f0388198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f0387e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0388101ef0 0x7f0388198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56366/0 (socket says 172.21.15.97:56366) 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f0387e00700 1 -- 172.21.15.97:0/1683412012 learned_addr learned my addr 172.21.15.97:0/1683412012 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f037fe00700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03881028a0 0x7f0388198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:45.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f038cc00700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0388106fd0 0x7f038819cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f0387e00700 1 -- 172.21.15.97:0/1683412012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0388106fd0 msgr2=0x7f038819cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f0387e00700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0388106fd0 0x7f038819cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f0387e00700 1 -- 172.21.15.97:0/1683412012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03881028a0 msgr2=0x7f0388198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f0387e00700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03881028a0 0x7f0388198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f0387e00700 1 -- 172.21.15.97:0/1683412012 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0378009520 con 0x7f0388101ef0 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.087+0000 7f037fe00700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f03881028a0 0x7f0388198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.088+0000 7f0387e00700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0388101ef0 0x7f0388198240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f037800eb30 tx=0x7f037800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.088+0000 7f0385600700 1 -- 172.21.15.97:0/1683412012 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0378009880 con 0x7f0388101ef0 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.088+0000 7f0385600700 1 -- 172.21.15.97:0/1683412012 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0378004500 con 0x7f0388101ef0 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.088+0000 7f0385600700 1 -- 172.21.15.97:0/1683412012 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0378003e70 con 0x7f0388101ef0 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.088+0000 7f038ec00700 1 -- 172.21.15.97:0/1683412012 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03700095f0 con 0x7f0388101ef0 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.088+0000 7f038ec00700 1 -- 172.21.15.97:0/1683412012 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03881a4d70 con 0x7f0388101ef0 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.089+0000 7f038ec00700 1 -- 172.21.15.97:0/1683412012 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f03881033d0 con 0x7f0388101ef0 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.092+0000 7f0385600700 1 -- 172.21.15.97:0/1683412012 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f037800cbd0 con 0x7f0388101ef0 2024-03-11T19:56:45.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.094+0000 7f0385600700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f036c0a1bd0 0x7f036c0a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.094+0000 7f0385600700 1 -- 172.21.15.97:0/1683412012 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f0378014070 con 0x7f0388101ef0 2024-03-11T19:56:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.094+0000 7f037fe00700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f036c0a1bd0 0x7f036c0a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.094+0000 7f0385600700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993] conn(0x7f036c0accb0 0x7f036c0af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.094+0000 7f0385600700 1 -- 172.21.15.97:0/1683412012 --> [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f036c0af820 con 0x7f036c0accb0 2024-03-11T19:56:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.094+0000 7f0385600700 1 -- 172.21.15.97:0/1683412012 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03780ce310 con 0x7f0388101ef0 2024-03-11T19:56:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.095+0000 7f037fe00700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f036c0a1bd0 0x7f036c0a4090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f037000b2e0 tx=0x7f0370005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.095+0000 7f038cc00700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993] conn(0x7f036c0accb0 0x7f036c0af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.096+0000 7f038cc00700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993] conn(0x7f036c0accb0 0x7f036c0af150 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.096+0000 7f0385600700 1 -- 172.21.15.97:0/1683412012 <== osd.2 v2:172.21.15.161:6800/4070161993 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f036c0af820 con 0x7f036c0accb0 2024-03-11T19:56:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.114+0000 7f038ec00700 1 -- 172.21.15.97:0/1683412012 --> [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0388112020 con 0x7f036c0accb0 2024-03-11T19:56:45.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.115+0000 7f0385600700 1 -- 172.21.15.97:0/1683412012 <== osd.2 v2:172.21.15.161:6800/4070161993 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0388112020 con 0x7f036c0accb0 2024-03-11T19:56:45.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.115+0000 7f038ec00700 1 -- 172.21.15.97:0/1683412012 >> [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993] conn(0x7f036c0accb0 msgr2=0x7f036c0af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:45.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.115+0000 7f038ec00700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993] conn(0x7f036c0accb0 0x7f036c0af150 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:45.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.116+0000 7f038ec00700 1 -- 172.21.15.97:0/1683412012 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f036c0a1bd0 msgr2=0x7f036c0a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:45.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.116+0000 7f038ec00700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f036c0a1bd0 0x7f036c0a4090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f037000b2e0 tx=0x7f0370005960 comp rx=0 tx=0).stop 2024-03-11T19:56:45.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.116+0000 7f038ec00700 1 -- 172.21.15.97:0/1683412012 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0388101ef0 msgr2=0x7f0388198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:45.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.116+0000 7f038ec00700 1 --2- 172.21.15.97:0/1683412012 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0388101ef0 0x7f0388198240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f037800eb30 tx=0x7f037800ee40 comp rx=0 tx=0).stop 2024-03-11T19:56:45.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.116+0000 7f038cc00700 1 -- 172.21.15.97:0/1683412012 reap_dead start 2024-03-11T19:56:45.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.116+0000 7f038ec00700 1 -- 172.21.15.97:0/1683412012 shutdown_connections 2024-03-11T19:56:45.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.116+0000 7f038ec00700 1 -- 172.21.15.97:0/1683412012 >> 172.21.15.97:0/1683412012 conn(0x7f03880fd740 msgr2=0x7f03880fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:45.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.116+0000 7f038ec00700 1 -- 172.21.15.97:0/1683412012 shutdown_connections 2024-03-11T19:56:45.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:45.116+0000 7f038ec00700 1 -- 172.21.15.97:0/1683412012 wait complete. 2024-03-11T19:56:45.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:45 smithi097 ceph-mon[31898]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:56:45.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-03-11T19:56:45.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-11T19:56:45.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-03-11T19:56:45.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-11T19:56:45.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-11T19:56:45.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-03-11T19:56:45.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-03-11T19:56:45.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-11T19:56:45.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-11T19:56:45.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:45 smithi097 ceph-mon[31898]: osdmap e90: 12 total, 12 up, 12 in 2024-03-11T19:56:45.919 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:56:45.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:45 smithi114 ceph-mon[36376]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:56:45.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-03-11T19:56:45.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-11T19:56:45.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-03-11T19:56:45.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-11T19:56:45.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-11T19:56:45.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-03-11T19:56:45.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-03-11T19:56:45.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-11T19:56:45.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-11T19:56:45.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:45 smithi114 ceph-mon[36376]: osdmap e90: 12 total, 12 up, 12 in 2024-03-11T19:56:45.991 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:56:46.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:45 smithi161 ceph-mon[36332]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:56:46.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-03-11T19:56:46.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-11T19:56:46.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-03-11T19:56:46.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-11T19:56:46.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-11T19:56:46.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-03-11T19:56:46.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-03-11T19:56:46.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-11T19:56:46.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-11T19:56:46.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:45 smithi161 ceph-mon[36332]: osdmap e90: 12 total, 12 up, 12 in 2024-03-11T19:56:46.037 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:56:46.974 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:46 smithi097 ceph-mon[31898]: osdmap e91: 12 total, 12 up, 12 in 2024-03-11T19:56:46.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:46 smithi114 ceph-mon[36376]: osdmap e91: 12 total, 12 up, 12 in 2024-03-11T19:56:47.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:46 smithi161 ceph-mon[36332]: osdmap e91: 12 total, 12 up, 12 in 2024-03-11T19:56:47.892 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:47 smithi097 ceph-mon[31898]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T19:56:47.893 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:47 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:56:47.893 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:47 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:56:47.893 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:47 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:56:47.893 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:47 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:56:47.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:47 smithi114 ceph-mon[36376]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T19:56:47.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:47 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:56:47.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:47 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:56:47.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:47 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:56:47.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:47 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:56:48.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:47 smithi161 ceph-mon[36332]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T19:56:48.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:47 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:56:48.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:47 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:56:48.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:47 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:56:48.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:47 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:56:49.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.217+0000 7fe2df200700 1 -- 172.21.15.97:0/1397634761 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e01008a0 msgr2=0x7fe2e0100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:49.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.217+0000 7fe2df200700 1 --2- 172.21.15.97:0/1397634761 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e01008a0 0x7fe2e0100c80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009990 tx=0x7fe2d0009ca0 comp rx=0 tx=0).stop 2024-03-11T19:56:49.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.217+0000 7fe2df200700 1 -- 172.21.15.97:0/1397634761 shutdown_connections 2024-03-11T19:56:49.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.217+0000 7fe2df200700 1 --2- 172.21.15.97:0/1397634761 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e01008a0 0x7fe2e0100c80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:49.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.217+0000 7fe2df200700 1 --2- 172.21.15.97:0/1397634761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2e010e450 0x7fe2e0110840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:49.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.217+0000 7fe2df200700 1 --2- 172.21.15.97:0/1397634761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2e0101250 0x7fe2e010df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.217+0000 7fe2df200700 1 -- 172.21.15.97:0/1397634761 >> 172.21.15.97:0/1397634761 conn(0x7fe2e00fc540 msgr2=0x7fe2e00fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.217+0000 7fe2df200700 1 -- 172.21.15.97:0/1397634761 shutdown_connections 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.217+0000 7fe2df200700 1 -- 172.21.15.97:0/1397634761 wait complete. 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.218+0000 7fe2df200700 1 Processor -- start 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.219+0000 7fe2df200700 1 -- start start 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.219+0000 7fe2df200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2e01008a0 0x7fe2e019a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.219+0000 7fe2df200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2e0101250 0x7fe2e019a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.219+0000 7fe2df200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e010e450 0x7fe2e019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.219+0000 7fe2df200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e019f1c0 con 0x7fe2e010e450 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.219+0000 7fe2df200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e019f330 con 0x7fe2e0101250 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.219+0000 7fe2df200700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e019f4f0 con 0x7fe2e01008a0 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.219+0000 7fe2dd400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2e0101250 0x7fe2e019a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.220+0000 7fe2dd400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2e0101250 0x7fe2e019a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:39598/0 (socket says 172.21.15.97:39598) 2024-03-11T19:56:49.912 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.220+0000 7fe2dd400700 1 -- 172.21.15.97:0/2305353916 learned_addr learned my addr 172.21.15.97:0/2305353916 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.220+0000 7fe2dde00700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2e01008a0 0x7fe2e019a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.220+0000 7fe2de800700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e010e450 0x7fe2e019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.220+0000 7fe2dd400700 1 -- 172.21.15.97:0/2305353916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2e01008a0 msgr2=0x7fe2e019a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.220+0000 7fe2dd400700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe2e01008a0 0x7fe2e019a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.220+0000 7fe2dd400700 1 -- 172.21.15.97:0/2305353916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e010e450 msgr2=0x7fe2e019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.220+0000 7fe2dd400700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e010e450 0x7fe2e019ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.220+0000 7fe2dd400700 1 -- 172.21.15.97:0/2305353916 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d00095f0 con 0x7fe2e0101250 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.220+0000 7fe2de800700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe2e010e450 0x7fe2e019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.220+0000 7fe2dd400700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2e0101250 0x7fe2e019a870 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00e9a0 tx=0x7fe2cc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.221+0000 7fe2d6a00700 1 -- 172.21.15.97:0/2305353916 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc009790 con 0x7fe2e0101250 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.221+0000 7fe2df200700 1 -- 172.21.15.97:0/2305353916 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e019f7d0 con 0x7fe2e0101250 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.221+0000 7fe2d6a00700 1 -- 172.21.15.97:0/2305353916 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2cc00cbd0 con 0x7fe2e0101250 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.221+0000 7fe2d6a00700 1 -- 172.21.15.97:0/2305353916 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc018680 con 0x7fe2e0101250 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.221+0000 7fe2df200700 1 -- 172.21.15.97:0/2305353916 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e01a7030 con 0x7fe2e0101250 2024-03-11T19:56:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.222+0000 7fe2df200700 1 -- 172.21.15.97:0/2305353916 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe2e0114150 con 0x7fe2e0101250 2024-03-11T19:56:49.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.226+0000 7fe2d6a00700 1 -- 172.21.15.97:0/2305353916 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fe2cc00cd40 con 0x7fe2e0101250 2024-03-11T19:56:49.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.227+0000 7fe2d6a00700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe2c40a1bd0 0x7fe2c40a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:49.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.227+0000 7fe2d6a00700 1 -- 172.21.15.97:0/2305353916 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fe2cc014070 con 0x7fe2e0101250 2024-03-11T19:56:49.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.227+0000 7fe2dde00700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe2c40a1bd0 0x7fe2c40a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:49.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.227+0000 7fe2d6a00700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465] conn(0x7fe2c40accb0 0x7fe2c40af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:49.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.227+0000 7fe2d6a00700 1 -- 172.21.15.97:0/2305353916 --> [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2c40af820 con 0x7fe2c40accb0 2024-03-11T19:56:49.914 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.227+0000 7fe2d6a00700 1 -- 172.21.15.97:0/2305353916 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2cc0ce560 con 0x7fe2e0101250 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.228+0000 7fe2de800700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465] conn(0x7fe2c40accb0 0x7fe2c40af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.228+0000 7fe2dde00700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe2c40a1bd0 0x7fe2c40a4090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000b2e0 tx=0x7fe2d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.228+0000 7fe2de800700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465] conn(0x7fe2c40accb0 0x7fe2c40af150 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.229+0000 7fe2d6a00700 1 -- 172.21.15.97:0/2305353916 <== osd.5 v2:172.21.15.161:6808/2197634465 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe2c40af820 con 0x7fe2c40accb0 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.247+0000 7fe2df200700 1 -- 172.21.15.97:0/2305353916 --> [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe2e0062330 con 0x7fe2c40accb0 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.248+0000 7fe2d6a00700 1 -- 172.21.15.97:0/2305353916 <== osd.5 v2:172.21.15.161:6808/2197634465 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe2e0062330 con 0x7fe2c40accb0 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.249+0000 7fe2df200700 1 -- 172.21.15.97:0/2305353916 >> [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465] conn(0x7fe2c40accb0 msgr2=0x7fe2c40af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.249+0000 7fe2df200700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465] conn(0x7fe2c40accb0 0x7fe2c40af150 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.249+0000 7fe2df200700 1 -- 172.21.15.97:0/2305353916 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe2c40a1bd0 msgr2=0x7fe2c40a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.249+0000 7fe2df200700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe2c40a1bd0 0x7fe2c40a4090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000b2e0 tx=0x7fe2d0005fd0 comp rx=0 tx=0).stop 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.249+0000 7fe2df200700 1 -- 172.21.15.97:0/2305353916 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2e0101250 msgr2=0x7fe2e019a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.249+0000 7fe2df200700 1 --2- 172.21.15.97:0/2305353916 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe2e0101250 0x7fe2e019a870 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00e9a0 tx=0x7fe2cc00ecb0 comp rx=0 tx=0).stop 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.249+0000 7fe2de800700 1 -- 172.21.15.97:0/2305353916 reap_dead start 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.249+0000 7fe2df200700 1 -- 172.21.15.97:0/2305353916 shutdown_connections 2024-03-11T19:56:49.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.249+0000 7fe2df200700 1 -- 172.21.15.97:0/2305353916 >> 172.21.15.97:0/2305353916 conn(0x7fe2e00fc540 msgr2=0x7fe2e010f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:49.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.249+0000 7fe2df200700 1 -- 172.21.15.97:0/2305353916 shutdown_connections 2024-03-11T19:56:49.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:48.249+0000 7fe2df200700 1 -- 172.21.15.97:0/2305353916 wait complete. 2024-03-11T19:56:49.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:49 smithi114 ceph-mon[36376]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-03-11T19:56:50.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:49 smithi161 ceph-mon[36332]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-03-11T19:56:50.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:49 smithi097 ceph-mon[31898]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-03-11T19:56:51.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:51 smithi114 ceph-mon[36376]: pgmap v354: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 8 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-03-11T19:56:52.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:51 smithi161 ceph-mon[36332]: pgmap v354: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 8 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-03-11T19:56:52.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:51 smithi097 ceph-mon[31898]: pgmap v354: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 8 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-03-11T19:56:52.560 INFO:teuthology.orchestra.run.smithi097.stdout:77309411407 2024-03-11T19:56:52.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T19:56:53.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:53 smithi097 ceph-mon[31898]: pgmap v355: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-03-11T19:56:53.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:53 smithi114 ceph-mon[36376]: pgmap v355: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-03-11T19:56:54.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:53 smithi161 ceph-mon[36332]: pgmap v355: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-03-11T19:56:54.794 INFO:teuthology.orchestra.run.smithi097.stdout:163208757310 2024-03-11T19:56:54.795 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T19:56:55.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.546+0000 7f72a8000700 1 -- 172.21.15.97:0/765873223 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a0107cc0 msgr2=0x7f72a01080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:55.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.546+0000 7f72a8000700 1 --2- 172.21.15.97:0/765873223 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a0107cc0 0x7f72a01080a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f72940099c0 tx=0x7f7294009cd0 comp rx=0 tx=0).stop 2024-03-11T19:56:55.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.546+0000 7f72a8000700 1 -- 172.21.15.97:0/765873223 shutdown_connections 2024-03-11T19:56:55.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.546+0000 7f72a8000700 1 --2- 172.21.15.97:0/765873223 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72a006a2c0 0x7f72a010eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:55.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.546+0000 7f72a8000700 1 --2- 172.21.15.97:0/765873223 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a0107cc0 0x7f72a01080a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:55.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.546+0000 7f72a8000700 1 --2- 172.21.15.97:0/765873223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a0069920 0x7f72a0069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:55.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.546+0000 7f72a8000700 1 -- 172.21.15.97:0/765873223 >> 172.21.15.97:0/765873223 conn(0x7f72a0077aa0 msgr2=0x7f72a0077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:55.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.547+0000 7f72a8000700 1 -- 172.21.15.97:0/765873223 shutdown_connections 2024-03-11T19:56:55.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.547+0000 7f72a8000700 1 -- 172.21.15.97:0/765873223 wait complete. 2024-03-11T19:56:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.549+0000 7f72a8000700 1 Processor -- start 2024-03-11T19:56:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.549+0000 7f72a8000700 1 -- start start 2024-03-11T19:56:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.549+0000 7f72a8000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a0069920 0x7f72a019a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.549+0000 7f72a8000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a006a2c0 0x7f72a019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.549+0000 7f72a8000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72a0107cc0 0x7f72a019ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.549+0000 7f72a8000700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a019f290 con 0x7f72a0107cc0 2024-03-11T19:56:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.549+0000 7f72a8000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a019f400 con 0x7f72a006a2c0 2024-03-11T19:56:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.549+0000 7f72a8000700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a019f5c0 con 0x7f72a0069920 2024-03-11T19:56:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.550+0000 7f72a4c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a006a2c0 0x7f72a019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.550+0000 7f72a4c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a006a2c0 0x7f72a019a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:44042/0 (socket says 172.21.15.97:44042) 2024-03-11T19:56:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.550+0000 7f72a4c00700 1 -- 172.21.15.97:0/292543704 learned_addr learned my addr 172.21.15.97:0/292543704 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:56:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.550+0000 7f72a6000700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72a0107cc0 0x7f72a019ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:55.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.550+0000 7f72a5600700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a0069920 0x7f72a019a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.550+0000 7f72a4c00700 1 -- 172.21.15.97:0/292543704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a0069920 msgr2=0x7f72a019a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.550+0000 7f72a4c00700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a0069920 0x7f72a019a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.550+0000 7f72a4c00700 1 -- 172.21.15.97:0/292543704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72a0107cc0 msgr2=0x7f72a019ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.550+0000 7f72a4c00700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f72a0107cc0 0x7f72a019ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.550+0000 7f72a4c00700 1 -- 172.21.15.97:0/292543704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7298009c40 con 0x7f72a006a2c0 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.550+0000 7f72a5600700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72a0069920 0x7f72a019a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.550+0000 7f72a4c00700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a006a2c0 0x7f72a019a8f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f728c00e9a0 tx=0x7f728c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.551+0000 7f729e000700 1 -- 172.21.15.97:0/292543704 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728c00eee0 con 0x7f72a006a2c0 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.551+0000 7f729e000700 1 -- 172.21.15.97:0/292543704 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f728c00cbd0 con 0x7f72a006a2c0 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.551+0000 7f72a8000700 1 -- 172.21.15.97:0/292543704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72940095f0 con 0x7f72a006a2c0 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.551+0000 7f729e000700 1 -- 172.21.15.97:0/292543704 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728c009790 con 0x7f72a006a2c0 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.551+0000 7f72a8000700 1 -- 172.21.15.97:0/292543704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a01a6f30 con 0x7f72a006a2c0 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.552+0000 7f72a8000700 1 -- 172.21.15.97:0/292543704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f72a0103370 con 0x7f72a006a2c0 2024-03-11T19:56:55.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.554+0000 7f729e000700 1 -- 172.21.15.97:0/292543704 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f728c00cd40 con 0x7f72a006a2c0 2024-03-11T19:56:55.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.555+0000 7f729e000700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f72880a1bd0 0x7f72880a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:55.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.556+0000 7f729e000700 1 -- 172.21.15.97:0/292543704 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f728c014070 con 0x7f72a006a2c0 2024-03-11T19:56:55.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.556+0000 7f72a5600700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f72880a1bd0 0x7f72880a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:55.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.556+0000 7f729e000700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520] conn(0x7f72880accb0 0x7f72880af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:56:55.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.556+0000 7f729e000700 1 -- 172.21.15.97:0/292543704 --> [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72880af820 con 0x7f72880accb0 2024-03-11T19:56:55.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.556+0000 7f729e000700 1 -- 172.21.15.97:0/292543704 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f728c0d22a0 con 0x7f72a006a2c0 2024-03-11T19:56:55.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.556+0000 7f72a6000700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520] conn(0x7f72880accb0 0x7f72880af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:56:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.557+0000 7f72a5600700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f72880a1bd0 0x7f72880a4090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7294000c00 tx=0x7f7294005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.557+0000 7f72a6000700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520] conn(0x7f72880accb0 0x7f72880af150 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:56:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.558+0000 7f729e000700 1 -- 172.21.15.97:0/292543704 <== osd.0 v2:172.21.15.97:6802/2786748520 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f72880af820 con 0x7f72880accb0 2024-03-11T19:56:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.576+0000 7f72a8000700 1 -- 172.21.15.97:0/292543704 --> [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f72a0111ff0 con 0x7f72880accb0 2024-03-11T19:56:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.577+0000 7f729e000700 1 -- 172.21.15.97:0/292543704 <== osd.0 v2:172.21.15.97:6802/2786748520 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f72a0111ff0 con 0x7f72880accb0 2024-03-11T19:56:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.577+0000 7f72a8000700 1 -- 172.21.15.97:0/292543704 >> [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520] conn(0x7f72880accb0 msgr2=0x7f72880af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.577+0000 7f72a8000700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520] conn(0x7f72880accb0 0x7f72880af150 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:56:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.577+0000 7f72a8000700 1 -- 172.21.15.97:0/292543704 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f72880a1bd0 msgr2=0x7f72880a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.577+0000 7f72a8000700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f72880a1bd0 0x7f72880a4090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7294000c00 tx=0x7f7294005fb0 comp rx=0 tx=0).stop 2024-03-11T19:56:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.577+0000 7f72a8000700 1 -- 172.21.15.97:0/292543704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a006a2c0 msgr2=0x7f72a019a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:56:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.577+0000 7f72a8000700 1 --2- 172.21.15.97:0/292543704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a006a2c0 0x7f72a019a8f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f728c00e9a0 tx=0x7f728c00ed60 comp rx=0 tx=0).stop 2024-03-11T19:56:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.577+0000 7f72a6000700 1 -- 172.21.15.97:0/292543704 reap_dead start 2024-03-11T19:56:55.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.578+0000 7f72a8000700 1 -- 172.21.15.97:0/292543704 shutdown_connections 2024-03-11T19:56:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.578+0000 7f72a8000700 1 -- 172.21.15.97:0/292543704 >> 172.21.15.97:0/292543704 conn(0x7f72a0077aa0 msgr2=0x7f72a010ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:56:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.578+0000 7f72a8000700 1 -- 172.21.15.97:0/292543704 shutdown_connections 2024-03-11T19:56:55.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:56:55.578+0000 7f72a8000700 1 -- 172.21.15.97:0/292543704 wait complete. 2024-03-11T19:56:55.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:55 smithi097 ceph-mon[31898]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 59 B/s, 1 keys/s, 1 objects/s recovering 2024-03-11T19:56:55.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:55 smithi114 ceph-mon[36376]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 59 B/s, 1 keys/s, 1 objects/s recovering 2024-03-11T19:56:56.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:55 smithi161 ceph-mon[36332]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 59 B/s, 1 keys/s, 1 objects/s recovering 2024-03-11T19:56:57.086 INFO:teuthology.orchestra.run.smithi097.stdout:343597383707 2024-03-11T19:56:57.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-11T19:56:57.755 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:57 smithi097 ceph-mon[31898]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 53 B/s, 0 keys/s, 1 objects/s recovering 2024-03-11T19:56:57.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:57 smithi114 ceph-mon[36376]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 53 B/s, 0 keys/s, 1 objects/s recovering 2024-03-11T19:56:58.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:57 smithi161 ceph-mon[36332]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 53 B/s, 0 keys/s, 1 objects/s recovering 2024-03-11T19:56:59.618 INFO:teuthology.orchestra.run.smithi097.stdout:34359738459 2024-03-11T19:56:59.618 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T19:56:59.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:56:59 smithi097 ceph-mon[31898]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-03-11T19:56:59.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:56:59 smithi114 ceph-mon[36376]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-03-11T19:57:00.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:56:59 smithi161 ceph-mon[36332]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s; 48 B/s, 0 keys/s, 0 objects/s recovering 2024-03-11T19:57:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.183+0000 7f5627a00700 1 -- 172.21.15.97:0/181312152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5620106ff0 msgr2=0x7f562010ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.183+0000 7f5627a00700 1 --2- 172.21.15.97:0/181312152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5620106ff0 0x7f562010ebc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f561800b0a0 tx=0x7f561800b3b0 comp rx=0 tx=0).stop 2024-03-11T19:57:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.183+0000 7f5627a00700 1 -- 172.21.15.97:0/181312152 shutdown_connections 2024-03-11T19:57:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.183+0000 7f5627a00700 1 --2- 172.21.15.97:0/181312152 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5620101f20 0x7f5620102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.183+0000 7f5627a00700 1 --2- 172.21.15.97:0/181312152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5620106ff0 0x7f562010ebc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.183+0000 7f5627a00700 1 --2- 172.21.15.97:0/181312152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56201028d0 0x7f5620106910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.183+0000 7f5627a00700 1 -- 172.21.15.97:0/181312152 >> 172.21.15.97:0/181312152 conn(0x7f5620078dd0 msgr2=0x7f56200791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.183+0000 7f5627a00700 1 -- 172.21.15.97:0/181312152 shutdown_connections 2024-03-11T19:57:01.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.183+0000 7f5627a00700 1 -- 172.21.15.97:0/181312152 wait complete. 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.184+0000 7f5627a00700 1 Processor -- start 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.184+0000 7f5627a00700 1 -- start start 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.185+0000 7f5627a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5620101f20 0x7f56201983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.185+0000 7f5627a00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56201028d0 0x7f56201988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.185+0000 7f5627a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5620106ff0 0x7f562019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.185+0000 7f5627a00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562019d280 con 0x7f5620106ff0 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.185+0000 7f5627a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562019d3f0 con 0x7f5620101f20 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.185+0000 7f5627a00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562019d5b0 con 0x7f56201028d0 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.185+0000 7f561fe00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56201028d0 0x7f56201988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.185+0000 7f561fe00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56201028d0 0x7f56201988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:56546/0 (socket says 172.21.15.97:56546) 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.185+0000 7f561fe00700 1 -- 172.21.15.97:0/2688237703 learned_addr learned my addr 172.21.15.97:0/2688237703 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.185+0000 7f5625a00700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5620106ff0 0x7f562019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.185+0000 7f5625000700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5620101f20 0x7f56201983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.186+0000 7f561fe00700 1 -- 172.21.15.97:0/2688237703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5620101f20 msgr2=0x7f56201983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:01.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.186+0000 7f561fe00700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5620101f20 0x7f56201983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:01.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.186+0000 7f561fe00700 1 -- 172.21.15.97:0/2688237703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5620106ff0 msgr2=0x7f562019cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:01.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.186+0000 7f561fe00700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5620106ff0 0x7f562019cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:01.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.186+0000 7f561fe00700 1 -- 172.21.15.97:0/2688237703 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5618009d00 con 0x7f56201028d0 2024-03-11T19:57:01.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.186+0000 7f561fe00700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56201028d0 0x7f56201988e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f560c00e970 tx=0x7f560c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:01.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.186+0000 7f561d600700 1 -- 172.21.15.97:0/2688237703 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c009790 con 0x7f56201028d0 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.186+0000 7f561d600700 1 -- 172.21.15.97:0/2688237703 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f560c00c7e0 con 0x7f56201028d0 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.186+0000 7f561d600700 1 -- 172.21.15.97:0/2688237703 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c005580 con 0x7f56201028d0 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.186+0000 7f5627a00700 1 -- 172.21.15.97:0/2688237703 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562019d890 con 0x7f56201028d0 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.186+0000 7f5627a00700 1 -- 172.21.15.97:0/2688237703 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5620074070 con 0x7f56201028d0 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.187+0000 7f5627a00700 1 -- 172.21.15.97:0/2688237703 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5620112040 con 0x7f56201028d0 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.189+0000 7f561d600700 1 -- 172.21.15.97:0/2688237703 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f560c00cce0 con 0x7f56201028d0 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.190+0000 7f561d600700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f56080a5f60 0x7f56080a8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.191+0000 7f561d600700 1 -- 172.21.15.97:0/2688237703 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f560c014070 con 0x7f56201028d0 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.191+0000 7f5625000700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f56080a5f60 0x7f56080a8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.191+0000 7f561d600700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844] conn(0x7f56080b1040 0x7f56080b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.191+0000 7f561d600700 1 -- 172.21.15.97:0/2688237703 --> [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56080b3bb0 con 0x7f56080b1040 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.191+0000 7f561d600700 1 -- 172.21.15.97:0/2688237703 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f560c0ce050 con 0x7f56201028d0 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.191+0000 7f5625a00700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844] conn(0x7f56080b1040 0x7f56080b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.191+0000 7f5625000700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f56080a5f60 0x7f56080a8420 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5614009c60 tx=0x7f5614009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:01.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.192+0000 7f5625a00700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844] conn(0x7f56080b1040 0x7f56080b34e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.193+0000 7f561d600700 1 -- 172.21.15.97:0/2688237703 <== osd.1 v2:172.21.15.114:6800/2861651844 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f56080b3bb0 con 0x7f56080b1040 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.211+0000 7f5627a00700 1 -- 172.21.15.97:0/2688237703 --> [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5620062380 con 0x7f56080b1040 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.212+0000 7f561d600700 1 -- 172.21.15.97:0/2688237703 <== osd.1 v2:172.21.15.114:6800/2861651844 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5620062380 con 0x7f56080b1040 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.212+0000 7f5627a00700 1 -- 172.21.15.97:0/2688237703 >> [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844] conn(0x7f56080b1040 msgr2=0x7f56080b34e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.212+0000 7f5627a00700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844] conn(0x7f56080b1040 0x7f56080b34e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.212+0000 7f5627a00700 1 -- 172.21.15.97:0/2688237703 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f56080a5f60 msgr2=0x7f56080a8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.212+0000 7f5627a00700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f56080a5f60 0x7f56080a8420 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5614009c60 tx=0x7f5614009260 comp rx=0 tx=0).stop 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.212+0000 7f5627a00700 1 -- 172.21.15.97:0/2688237703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56201028d0 msgr2=0x7f56201988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.212+0000 7f5627a00700 1 --2- 172.21.15.97:0/2688237703 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56201028d0 0x7f56201988e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f560c00e970 tx=0x7f560c00ec80 comp rx=0 tx=0).stop 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.213+0000 7f5625a00700 1 -- 172.21.15.97:0/2688237703 reap_dead start 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.213+0000 7f5627a00700 1 -- 172.21.15.97:0/2688237703 shutdown_connections 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.213+0000 7f5627a00700 1 -- 172.21.15.97:0/2688237703 >> 172.21.15.97:0/2688237703 conn(0x7f5620078dd0 msgr2=0x7f56200ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.213+0000 7f5627a00700 1 -- 172.21.15.97:0/2688237703 shutdown_connections 2024-03-11T19:57:01.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.213+0000 7f5627a00700 1 -- 172.21.15.97:0/2688237703 wait complete. 2024-03-11T19:57:01.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:01 smithi097 ceph-mon[31898]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:57:02.035 INFO:teuthology.orchestra.run.smithi097.stdout:55834574934 2024-03-11T19:57:02.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T19:57:02.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:01 smithi161 ceph-mon[36332]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:57:02.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.317+0000 7f87f6400700 1 -- 172.21.15.97:0/158386977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87f0104090 msgr2=0x7f87f0104470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:02.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.317+0000 7f87f6400700 1 --2- 172.21.15.97:0/158386977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87f0104090 0x7f87f0104470 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f87e00098f0 tx=0x7f87e0009c00 comp rx=0 tx=0).stop 2024-03-11T19:57:02.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.317+0000 7f87f6400700 1 -- 172.21.15.97:0/158386977 shutdown_connections 2024-03-11T19:57:02.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.317+0000 7f87f6400700 1 --2- 172.21.15.97:0/158386977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f0109170 0x7f87f0110d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:02.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.317+0000 7f87f6400700 1 --2- 172.21.15.97:0/158386977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87f0104090 0x7f87f0104470 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.317+0000 7f87f6400700 1 --2- 172.21.15.97:0/158386977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87f0104a40 0x7f87f0108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.317+0000 7f87f6400700 1 -- 172.21.15.97:0/158386977 >> 172.21.15.97:0/158386977 conn(0x7f87f00ff940 msgr2=0x7f87f0101d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.317+0000 7f87f6400700 1 -- 172.21.15.97:0/158386977 shutdown_connections 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.317+0000 7f87f6400700 1 -- 172.21.15.97:0/158386977 wait complete. 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.318+0000 7f87f6400700 1 Processor -- start 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.318+0000 7f87f6400700 1 -- start start 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87f6400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f0104090 0x7f87f019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87f6400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87f0104a40 0x7f87f019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87f6400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87f0109170 0x7f87f019ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87f6400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f019f350 con 0x7f87f0109170 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87f6400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f019f4c0 con 0x7f87f0104090 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87f6400700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f019f680 con 0x7f87f0104a40 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87eea00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87f0104a40 0x7f87f019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87eea00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87f0104a40 0x7f87f019aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:56562/0 (socket says 172.21.15.97:56562) 2024-03-11T19:57:02.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87eea00700 1 -- 172.21.15.97:0/545354298 learned_addr learned my addr 172.21.15.97:0/545354298 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87ef400700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f0104090 0x7f87f019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87efe00700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87f0109170 0x7f87f019ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87eea00700 1 -- 172.21.15.97:0/545354298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f0104090 msgr2=0x7f87f019a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87eea00700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87f0104090 0x7f87f019a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87eea00700 1 -- 172.21.15.97:0/545354298 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87f0109170 msgr2=0x7f87f019ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87eea00700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87f0109170 0x7f87f019ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.319+0000 7f87eea00700 1 -- 172.21.15.97:0/545354298 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e0009520 con 0x7f87f0104a40 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.320+0000 7f87eea00700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87f0104a40 0x7f87f019aa00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00e9f0 tx=0x7f87dc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.320+0000 7f87e7e00700 1 -- 172.21.15.97:0/545354298 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc009690 con 0x7f87f0104a40 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.320+0000 7f87e7e00700 1 -- 172.21.15.97:0/545354298 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87dc010430 con 0x7f87f0104a40 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.320+0000 7f87e7e00700 1 -- 172.21.15.97:0/545354298 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc00cea0 con 0x7f87f0104a40 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.320+0000 7f87f6400700 1 -- 172.21.15.97:0/545354298 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f019f960 con 0x7f87f0104a40 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.320+0000 7f87f6400700 1 -- 172.21.15.97:0/545354298 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f01a71c0 con 0x7f87f0104a40 2024-03-11T19:57:02.096 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.321+0000 7f87f6400700 1 -- 172.21.15.97:0/545354298 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f87f0105570 con 0x7f87f0104a40 2024-03-11T19:57:02.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.323+0000 7f87e7e00700 1 -- 172.21.15.97:0/545354298 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f87dc0105a0 con 0x7f87f0104a40 2024-03-11T19:57:02.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.324+0000 7f87e7e00700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f87d40a1bd0 0x7f87d40a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:02.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.324+0000 7f87e7e00700 1 -- 172.21.15.97:0/545354298 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f87dc014070 con 0x7f87f0104a40 2024-03-11T19:57:02.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.324+0000 7f87ef400700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f87d40a1bd0 0x7f87d40a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:02.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.325+0000 7f87e7e00700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963] conn(0x7f87d40accb0 0x7f87d40af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:02.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.325+0000 7f87e7e00700 1 -- 172.21.15.97:0/545354298 --> [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87d40af820 con 0x7f87d40accb0 2024-03-11T19:57:02.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.325+0000 7f87e7e00700 1 -- 172.21.15.97:0/545354298 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87dc0ce340 con 0x7f87f0104a40 2024-03-11T19:57:02.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.325+0000 7f87efe00700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963] conn(0x7f87d40accb0 0x7f87d40af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:02.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.325+0000 7f87ef400700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f87d40a1bd0 0x7f87d40a4090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009810 tx=0x7f87e0019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:02.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.326+0000 7f87efe00700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963] conn(0x7f87d40accb0 0x7f87d40af150 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:02.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.327+0000 7f87e7e00700 1 -- 172.21.15.97:0/545354298 <== osd.7 v2:172.21.15.114:6816/3855344963 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f87d40af820 con 0x7f87d40accb0 2024-03-11T19:57:02.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.344+0000 7f87f6400700 1 -- 172.21.15.97:0/545354298 --> [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f87f0114220 con 0x7f87d40accb0 2024-03-11T19:57:02.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.345+0000 7f87e7e00700 1 -- 172.21.15.97:0/545354298 <== osd.7 v2:172.21.15.114:6816/3855344963 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f87f0114220 con 0x7f87d40accb0 2024-03-11T19:57:02.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.346+0000 7f87f6400700 1 -- 172.21.15.97:0/545354298 >> [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963] conn(0x7f87d40accb0 msgr2=0x7f87d40af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:02.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.346+0000 7f87f6400700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963] conn(0x7f87d40accb0 0x7f87d40af150 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:02.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.346+0000 7f87f6400700 1 -- 172.21.15.97:0/545354298 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f87d40a1bd0 msgr2=0x7f87d40a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:02.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.346+0000 7f87f6400700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f87d40a1bd0 0x7f87d40a4090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009810 tx=0x7f87e0019040 comp rx=0 tx=0).stop 2024-03-11T19:57:02.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.346+0000 7f87f6400700 1 -- 172.21.15.97:0/545354298 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87f0104a40 msgr2=0x7f87f019aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:02.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.346+0000 7f87f6400700 1 --2- 172.21.15.97:0/545354298 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f87f0104a40 0x7f87f019aa00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00e9f0 tx=0x7f87dc00ed00 comp rx=0 tx=0).stop 2024-03-11T19:57:02.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.346+0000 7f87efe00700 1 -- 172.21.15.97:0/545354298 reap_dead start 2024-03-11T19:57:02.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.346+0000 7f87f6400700 1 -- 172.21.15.97:0/545354298 shutdown_connections 2024-03-11T19:57:02.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.346+0000 7f87f6400700 1 -- 172.21.15.97:0/545354298 >> 172.21.15.97:0/545354298 conn(0x7f87f00ff940 msgr2=0x7f87f0108300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:02.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.346+0000 7f87f6400700 1 -- 172.21.15.97:0/545354298 shutdown_connections 2024-03-11T19:57:02.099 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:01.346+0000 7f87f6400700 1 -- 172.21.15.97:0/545354298 wait complete. 2024-03-11T19:57:02.240 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:01 smithi114 ceph-mon[36376]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:57:03.468 INFO:teuthology.orchestra.run.smithi097.stdout:219043332149 2024-03-11T19:57:03.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T19:57:03.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.720+0000 7f5652a00700 1 -- 172.21.15.97:0/2874850423 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f564c1091c0 msgr2=0x7f564c110d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:03.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.720+0000 7f5652a00700 1 --2- 172.21.15.97:0/2874850423 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f564c1091c0 0x7f564c110d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f564400b0a0 tx=0x7f564400b3b0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.720+0000 7f5652a00700 1 -- 172.21.15.97:0/2874850423 shutdown_connections 2024-03-11T19:57:03.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.720+0000 7f5652a00700 1 --2- 172.21.15.97:0/2874850423 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f564c1091c0 0x7f564c110d90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.720+0000 7f5652a00700 1 --2- 172.21.15.97:0/2874850423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f564c1040e0 0x7f564c1044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.720+0000 7f5652a00700 1 --2- 172.21.15.97:0/2874850423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f564c104a90 0x7f564c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.720+0000 7f5652a00700 1 -- 172.21.15.97:0/2874850423 >> 172.21.15.97:0/2874850423 conn(0x7f564c0ff970 msgr2=0x7f564c101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:03.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.721+0000 7f5652a00700 1 -- 172.21.15.97:0/2874850423 shutdown_connections 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.721+0000 7f5652a00700 1 -- 172.21.15.97:0/2874850423 wait complete. 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.722+0000 7f5652a00700 1 Processor -- start 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.722+0000 7f5652a00700 1 -- start start 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.723+0000 7f5652a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f564c1040e0 0x7f564c19a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.723+0000 7f5652a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f564c104a90 0x7f564c19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.723+0000 7f5652a00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f564c1091c0 0x7f564c19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.723+0000 7f5652a00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c19f400 con 0x7f564c1040e0 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.723+0000 7f5652a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c19f570 con 0x7f564c104a90 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.723+0000 7f5652a00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c19f730 con 0x7f564c1091c0 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.723+0000 7f564be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f564c1040e0 0x7f564c19a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.723+0000 7f5650a00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f564c1091c0 0x7f564c19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.723+0000 7f564be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f564c1040e0 0x7f564c19a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53788/0 (socket says 172.21.15.97:53788) 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.723+0000 7f564be00700 1 -- 172.21.15.97:0/1652234489 learned_addr learned my addr 172.21.15.97:0/1652234489 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.723+0000 7f564b400700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f564c104a90 0x7f564c19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:03.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.724+0000 7f564be00700 1 -- 172.21.15.97:0/1652234489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f564c1091c0 msgr2=0x7f564c19edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.724+0000 7f564be00700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f564c1091c0 0x7f564c19edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.724+0000 7f564be00700 1 -- 172.21.15.97:0/1652234489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f564c104a90 msgr2=0x7f564c19aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.724+0000 7f564be00700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f564c104a90 0x7f564c19aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.724+0000 7f564be00700 1 -- 172.21.15.97:0/1652234489 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c009520 con 0x7f564c1040e0 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.724+0000 7f5650a00700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f564c1091c0 0x7f564c19edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.724+0000 7f564be00700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f564c1040e0 0x7f564c19a520 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f563c00eb30 tx=0x7f563c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.724+0000 7f5648c00700 1 -- 172.21.15.97:0/1652234489 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c009880 con 0x7f564c1040e0 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.725+0000 7f5648c00700 1 -- 172.21.15.97:0/1652234489 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f563c004500 con 0x7f564c1040e0 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.725+0000 7f5648c00700 1 -- 172.21.15.97:0/1652234489 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c003e70 con 0x7f564c1040e0 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.725+0000 7f5652a00700 1 -- 172.21.15.97:0/1652234489 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5638009520 con 0x7f564c1040e0 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.725+0000 7f5652a00700 1 -- 172.21.15.97:0/1652234489 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5644009d00 con 0x7f564c1040e0 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.726+0000 7f5652a00700 1 -- 172.21.15.97:0/1652234489 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f564c114280 con 0x7f564c1040e0 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.728+0000 7f5648c00700 1 -- 172.21.15.97:0/1652234489 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f563c00cbd0 con 0x7f564c1040e0 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.729+0000 7f5648c00700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f56300a1b50 0x7f56300a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:03.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.730+0000 7f5648c00700 1 -- 172.21.15.97:0/1652234489 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f563c014070 con 0x7f564c1040e0 2024-03-11T19:57:03.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.730+0000 7f564b400700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f56300a1b50 0x7f56300a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:03.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.730+0000 7f5648c00700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537] conn(0x7f56300acc30 0x7f56300af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:03.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.730+0000 7f5648c00700 1 -- 172.21.15.97:0/1652234489 --> [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56300af7a0 con 0x7f56300acc30 2024-03-11T19:57:03.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.730+0000 7f5648c00700 1 -- 172.21.15.97:0/1652234489 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f563c0ce340 con 0x7f564c1040e0 2024-03-11T19:57:03.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.730+0000 7f5650a00700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537] conn(0x7f56300acc30 0x7f56300af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:03.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.731+0000 7f564b400700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f56300a1b50 0x7f56300a4010 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f564c10a6d0 tx=0x7f5638009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:03.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.731+0000 7f5650a00700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537] conn(0x7f56300acc30 0x7f56300af0d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:03.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.732+0000 7f5648c00700 1 -- 172.21.15.97:0/1652234489 <== osd.10 v2:172.21.15.114:6824/275586537 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f56300af7a0 con 0x7f56300acc30 2024-03-11T19:57:03.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.751+0000 7f5652a00700 1 -- 172.21.15.97:0/1652234489 --> [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f564c100da0 con 0x7f56300acc30 2024-03-11T19:57:03.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.751+0000 7f5648c00700 1 -- 172.21.15.97:0/1652234489 <== osd.10 v2:172.21.15.114:6824/275586537 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f564c100da0 con 0x7f56300acc30 2024-03-11T19:57:03.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.752+0000 7f5652a00700 1 -- 172.21.15.97:0/1652234489 >> [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537] conn(0x7f56300acc30 msgr2=0x7f56300af0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:03.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.752+0000 7f5652a00700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537] conn(0x7f56300acc30 0x7f56300af0d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.752+0000 7f5652a00700 1 -- 172.21.15.97:0/1652234489 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f56300a1b50 msgr2=0x7f56300a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:03.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.752+0000 7f5652a00700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f56300a1b50 0x7f56300a4010 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f564c10a6d0 tx=0x7f5638009260 comp rx=0 tx=0).stop 2024-03-11T19:57:03.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.752+0000 7f5652a00700 1 -- 172.21.15.97:0/1652234489 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f564c1040e0 msgr2=0x7f564c19a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:03.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.752+0000 7f5652a00700 1 --2- 172.21.15.97:0/1652234489 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f564c1040e0 0x7f564c19a520 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f563c00eb30 tx=0x7f563c00ee40 comp rx=0 tx=0).stop 2024-03-11T19:57:03.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.752+0000 7f5650a00700 1 -- 172.21.15.97:0/1652234489 reap_dead start 2024-03-11T19:57:03.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.752+0000 7f5652a00700 1 -- 172.21.15.97:0/1652234489 shutdown_connections 2024-03-11T19:57:03.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.752+0000 7f5652a00700 1 -- 172.21.15.97:0/1652234489 >> 172.21.15.97:0/1652234489 conn(0x7f564c0ff970 msgr2=0x7f564c1003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:03.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.752+0000 7f5652a00700 1 -- 172.21.15.97:0/1652234489 shutdown_connections 2024-03-11T19:57:03.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.752+0000 7f5652a00700 1 -- 172.21.15.97:0/1652234489 wait complete. 2024-03-11T19:57:03.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.831+0000 7f6304400700 1 -- 172.21.15.97:0/1912036359 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62fc1028d0 msgr2=0x7f62fc106910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:03.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.831+0000 7f6304400700 1 --2- 172.21.15.97:0/1912036359 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62fc1028d0 0x7f62fc106910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f62e8009990 tx=0x7f62e8009ca0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.831+0000 7f6304400700 1 -- 172.21.15.97:0/1912036359 shutdown_connections 2024-03-11T19:57:03.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.831+0000 7f6304400700 1 --2- 172.21.15.97:0/1912036359 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62fc101f20 0x7f62fc102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.950 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.831+0000 7f6304400700 1 --2- 172.21.15.97:0/1912036359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62fc106ff0 0x7f62fc10ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.831+0000 7f6304400700 1 --2- 172.21.15.97:0/1912036359 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62fc1028d0 0x7f62fc106910 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.831+0000 7f6304400700 1 -- 172.21.15.97:0/1912036359 >> 172.21.15.97:0/1912036359 conn(0x7f62fc078dd0 msgr2=0x7f62fc0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.831+0000 7f6304400700 1 -- 172.21.15.97:0/1912036359 shutdown_connections 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.831+0000 7f6304400700 1 -- 172.21.15.97:0/1912036359 wait complete. 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.832+0000 7f6304400700 1 Processor -- start 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.833+0000 7f6304400700 1 -- start start 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.833+0000 7f6304400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62fc101f20 0x7f62fc198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.833+0000 7f6304400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62fc1028d0 0x7f62fc198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.833+0000 7f6304400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62fc106ff0 0x7f62fc19cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.833+0000 7f6304400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc19d190 con 0x7f62fc106ff0 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.833+0000 7f6304400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc19d300 con 0x7f62fc1028d0 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.833+0000 7f6304400700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc19d4c0 con 0x7f62fc101f20 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.833+0000 7f6302400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62fc106ff0 0x7f62fc19cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:03.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.833+0000 7f6302400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62fc106ff0 0x7f62fc19cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:53814/0 (socket says 172.21.15.97:53814) 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.833+0000 7f6302400700 1 -- 172.21.15.97:0/1330257025 learned_addr learned my addr 172.21.15.97:0/1330257025 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.833+0000 7f6301a00700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62fc101f20 0x7f62fc198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.833+0000 7f6301000700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62fc1028d0 0x7f62fc198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.834+0000 7f6302400700 1 -- 172.21.15.97:0/1330257025 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62fc101f20 msgr2=0x7f62fc198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.834+0000 7f6302400700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62fc101f20 0x7f62fc198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.834+0000 7f6302400700 1 -- 172.21.15.97:0/1330257025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62fc1028d0 msgr2=0x7f62fc198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.834+0000 7f6302400700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62fc1028d0 0x7f62fc198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.834+0000 7f6302400700 1 -- 172.21.15.97:0/1330257025 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e80095f0 con 0x7f62fc106ff0 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.834+0000 7f6301a00700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62fc101f20 0x7f62fc198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.834+0000 7f6302400700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62fc106ff0 0x7f62fc19cb80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f62f400efc0 tx=0x7f62f400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.834+0000 7f62fa000700 1 -- 172.21.15.97:0/1330257025 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4018070 con 0x7f62fc106ff0 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.834+0000 7f6304400700 1 -- 172.21.15.97:0/1330257025 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62fc19d740 con 0x7f62fc106ff0 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.834+0000 7f62fa000700 1 -- 172.21.15.97:0/1330257025 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62f400f040 con 0x7f62fc106ff0 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.834+0000 7f62fa000700 1 -- 172.21.15.97:0/1330257025 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f400cc80 con 0x7f62fc106ff0 2024-03-11T19:57:03.952 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.835+0000 7f6304400700 1 -- 172.21.15.97:0/1330257025 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62fc1a5000 con 0x7f62fc106ff0 2024-03-11T19:57:03.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.835+0000 7f6304400700 1 -- 172.21.15.97:0/1330257025 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f62fc112000 con 0x7f62fc106ff0 2024-03-11T19:57:03.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.837+0000 7f62fa000700 1 -- 172.21.15.97:0/1330257025 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f62f4003a00 con 0x7f62fc106ff0 2024-03-11T19:57:03.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.839+0000 7f62fa000700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f62e40a1bd0 0x7f62e40a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:03.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.839+0000 7f62fa000700 1 -- 172.21.15.97:0/1330257025 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f62f40cdb40 con 0x7f62fc106ff0 2024-03-11T19:57:03.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.839+0000 7f6301a00700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f62e40a1bd0 0x7f62e40a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:03.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.839+0000 7f62fa000700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177] conn(0x7f62e40accb0 0x7f62e40af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:03.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.839+0000 7f62fa000700 1 -- 172.21.15.97:0/1330257025 --> [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62e40af820 con 0x7f62e40accb0 2024-03-11T19:57:03.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.839+0000 7f62fa000700 1 -- 172.21.15.97:0/1330257025 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62f40cdeb0 con 0x7f62fc106ff0 2024-03-11T19:57:03.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.840+0000 7f6301000700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177] conn(0x7f62e40accb0 0x7f62e40af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:03.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.840+0000 7f6301a00700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f62e40a1bd0 0x7f62e40a4090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f62f0011490 tx=0x7f62f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:03.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.840+0000 7f6301000700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177] conn(0x7f62e40accb0 0x7f62e40af150 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:03.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.841+0000 7f62fa000700 1 -- 172.21.15.97:0/1330257025 <== osd.4 v2:172.21.15.114:6808/1414652177 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f62e40af820 con 0x7f62e40accb0 2024-03-11T19:57:03.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.859+0000 7f6304400700 1 -- 172.21.15.97:0/1330257025 --> [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f62fc062330 con 0x7f62e40accb0 2024-03-11T19:57:03.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.860+0000 7f62fa000700 1 -- 172.21.15.97:0/1330257025 <== osd.4 v2:172.21.15.114:6808/1414652177 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f62fc062330 con 0x7f62e40accb0 2024-03-11T19:57:03.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.860+0000 7f6304400700 1 -- 172.21.15.97:0/1330257025 >> [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177] conn(0x7f62e40accb0 msgr2=0x7f62e40af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:03.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.860+0000 7f6304400700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177] conn(0x7f62e40accb0 0x7f62e40af150 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.860+0000 7f6304400700 1 -- 172.21.15.97:0/1330257025 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f62e40a1bd0 msgr2=0x7f62e40a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:03.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.860+0000 7f6304400700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f62e40a1bd0 0x7f62e40a4090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f62f0011490 tx=0x7f62f0009260 comp rx=0 tx=0).stop 2024-03-11T19:57:03.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.860+0000 7f6304400700 1 -- 172.21.15.97:0/1330257025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62fc106ff0 msgr2=0x7f62fc19cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:03.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.860+0000 7f6304400700 1 --2- 172.21.15.97:0/1330257025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62fc106ff0 0x7f62fc19cb80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f62f400efc0 tx=0x7f62f400c3a0 comp rx=0 tx=0).stop 2024-03-11T19:57:03.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.860+0000 7f6302400700 1 -- 172.21.15.97:0/1330257025 reap_dead start 2024-03-11T19:57:03.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.860+0000 7f6304400700 1 -- 172.21.15.97:0/1330257025 shutdown_connections 2024-03-11T19:57:03.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.860+0000 7f6304400700 1 -- 172.21.15.97:0/1330257025 >> 172.21.15.97:0/1330257025 conn(0x7f62fc078dd0 msgr2=0x7f62fc0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:03.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.861+0000 7f6304400700 1 -- 172.21.15.97:0/1330257025 shutdown_connections 2024-03-11T19:57:03.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:03.861+0000 7f6304400700 1 -- 172.21.15.97:0/1330257025 wait complete. 2024-03-11T19:57:04.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:03 smithi161 ceph-mon[36332]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:57:04.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:03 smithi097 ceph-mon[31898]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:57:04.239 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:03 smithi114 ceph-mon[36376]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:57:04.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.939+0000 7fc3c8c00700 1 -- 172.21.15.97:0/3915542189 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3c41040c0 msgr2=0x7fc3c41044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:04.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.939+0000 7fc3c8c00700 1 --2- 172.21.15.97:0/3915542189 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3c41040c0 0x7fc3c41044a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc3b40098f0 tx=0x7fc3b4009c00 comp rx=0 tx=0).stop 2024-03-11T19:57:04.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.939+0000 7fc3c8c00700 1 -- 172.21.15.97:0/3915542189 shutdown_connections 2024-03-11T19:57:04.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.939+0000 7fc3c8c00700 1 --2- 172.21.15.97:0/3915542189 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3c41091a0 0x7fc3c4110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:04.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.939+0000 7fc3c8c00700 1 --2- 172.21.15.97:0/3915542189 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3c41040c0 0x7fc3c41044a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:04.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.939+0000 7fc3c8c00700 1 --2- 172.21.15.97:0/3915542189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3c4104a70 0x7fc3c4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:04.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.939+0000 7fc3c8c00700 1 -- 172.21.15.97:0/3915542189 >> 172.21.15.97:0/3915542189 conn(0x7fc3c40ff950 msgr2=0x7fc3c4101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:04.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.939+0000 7fc3c8c00700 1 -- 172.21.15.97:0/3915542189 shutdown_connections 2024-03-11T19:57:04.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.939+0000 7fc3c8c00700 1 -- 172.21.15.97:0/3915542189 wait complete. 2024-03-11T19:57:04.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.940+0000 7fc3c8c00700 1 Processor -- start 2024-03-11T19:57:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.940+0000 7fc3c8c00700 1 -- start start 2024-03-11T19:57:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.941+0000 7fc3c8c00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3c41040c0 0x7fc3c419a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.941+0000 7fc3c8c00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3c4104a70 0x7fc3c419a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.941+0000 7fc3c8c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3c41091a0 0x7fc3c419ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.941+0000 7fc3c8c00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c419f280 con 0x7fc3c4104a70 2024-03-11T19:57:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.941+0000 7fc3c8c00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c419f3f0 con 0x7fc3c41091a0 2024-03-11T19:57:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.941+0000 7fc3c8c00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c419f5b0 con 0x7fc3c41040c0 2024-03-11T19:57:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.941+0000 7fc3c1e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3c41040c0 0x7fc3c419a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.941+0000 7fc3c1400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3c4104a70 0x7fc3c419a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.941+0000 7fc3c1e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3c41040c0 0x7fc3c419a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:48576/0 (socket says 172.21.15.97:48576) 2024-03-11T19:57:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.941+0000 7fc3c1e00700 1 -- 172.21.15.97:0/27529036 learned_addr learned my addr 172.21.15.97:0/27529036 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:04.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.941+0000 7fc3c2800700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3c41091a0 0x7fc3c419ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.942+0000 7fc3c1e00700 1 -- 172.21.15.97:0/27529036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3c41091a0 msgr2=0x7fc3c419ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.942+0000 7fc3c1e00700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3c41091a0 0x7fc3c419ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.942+0000 7fc3c1e00700 1 -- 172.21.15.97:0/27529036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3c4104a70 msgr2=0x7fc3c419a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.942+0000 7fc3c1e00700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3c4104a70 0x7fc3c419a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.942+0000 7fc3c1e00700 1 -- 172.21.15.97:0/27529036 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b4009520 con 0x7fc3c41040c0 2024-03-11T19:57:04.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.942+0000 7fc3c1400700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3c4104a70 0x7fc3c419a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.942+0000 7fc3c1e00700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3c41040c0 0x7fc3c419a3a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009230 tx=0x7fc3b400fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.942+0000 7fc3baa00700 1 -- 172.21.15.97:0/27529036 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b401c070 con 0x7fc3c41040c0 2024-03-11T19:57:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.942+0000 7fc3baa00700 1 -- 172.21.15.97:0/27529036 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3b400b560 con 0x7fc3c41040c0 2024-03-11T19:57:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.942+0000 7fc3baa00700 1 -- 172.21.15.97:0/27529036 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4004cc0 con 0x7fc3c41040c0 2024-03-11T19:57:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.942+0000 7fc3c8c00700 1 -- 172.21.15.97:0/27529036 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c419f830 con 0x7fc3c41040c0 2024-03-11T19:57:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.943+0000 7fc3c8c00700 1 -- 172.21.15.97:0/27529036 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c41a6f20 con 0x7fc3c41040c0 2024-03-11T19:57:04.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.943+0000 7fc3c8c00700 1 -- 172.21.15.97:0/27529036 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc3c41141c0 con 0x7fc3c41040c0 2024-03-11T19:57:04.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.945+0000 7fc3baa00700 1 -- 172.21.15.97:0/27529036 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fc3b400b6d0 con 0x7fc3c41040c0 2024-03-11T19:57:04.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.947+0000 7fc3baa00700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc3a80a1bd0 0x7fc3a80a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:04.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.947+0000 7fc3baa00700 1 -- 172.21.15.97:0/27529036 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fc3b40cf420 con 0x7fc3c41040c0 2024-03-11T19:57:04.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.947+0000 7fc3c1400700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc3a80a1bd0 0x7fc3a80a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:04.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.947+0000 7fc3baa00700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214] conn(0x7fc3a80accb0 0x7fc3a80af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:04.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.947+0000 7fc3baa00700 1 -- 172.21.15.97:0/27529036 --> [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3a80af820 con 0x7fc3a80accb0 2024-03-11T19:57:04.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.947+0000 7fc3baa00700 1 -- 172.21.15.97:0/27529036 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3b40cf830 con 0x7fc3c41040c0 2024-03-11T19:57:04.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.948+0000 7fc3c2800700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214] conn(0x7fc3a80accb0 0x7fc3a80af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:04.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.948+0000 7fc3c1400700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc3a80a1bd0 0x7fc3a80a4090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4100e50 tx=0x7fc3b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:04.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.948+0000 7fc3c2800700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214] conn(0x7fc3a80accb0 0x7fc3a80af150 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:04.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.949+0000 7fc3baa00700 1 -- 172.21.15.97:0/27529036 <== osd.3 v2:172.21.15.97:6810/2971789214 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc3a80af820 con 0x7fc3a80accb0 2024-03-11T19:57:04.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.967+0000 7fc3c8c00700 1 -- 172.21.15.97:0/27529036 --> [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc3c4062380 con 0x7fc3a80accb0 2024-03-11T19:57:04.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.967+0000 7fc3baa00700 1 -- 172.21.15.97:0/27529036 <== osd.3 v2:172.21.15.97:6810/2971789214 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc3c4062380 con 0x7fc3a80accb0 2024-03-11T19:57:04.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.968+0000 7fc3c8c00700 1 -- 172.21.15.97:0/27529036 >> [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214] conn(0x7fc3a80accb0 msgr2=0x7fc3a80af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:04.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.968+0000 7fc3c8c00700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214] conn(0x7fc3a80accb0 0x7fc3a80af150 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:04.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.968+0000 7fc3c8c00700 1 -- 172.21.15.97:0/27529036 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc3a80a1bd0 msgr2=0x7fc3a80a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:04.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.968+0000 7fc3c8c00700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc3a80a1bd0 0x7fc3a80a4090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4100e50 tx=0x7fc3b0009260 comp rx=0 tx=0).stop 2024-03-11T19:57:04.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.968+0000 7fc3c8c00700 1 -- 172.21.15.97:0/27529036 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3c41040c0 msgr2=0x7fc3c419a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:04.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.968+0000 7fc3c8c00700 1 --2- 172.21.15.97:0/27529036 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3c41040c0 0x7fc3c419a3a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009230 tx=0x7fc3b400fa70 comp rx=0 tx=0).stop 2024-03-11T19:57:04.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.968+0000 7fc3c2800700 1 -- 172.21.15.97:0/27529036 reap_dead start 2024-03-11T19:57:04.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.968+0000 7fc3c8c00700 1 -- 172.21.15.97:0/27529036 shutdown_connections 2024-03-11T19:57:04.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.968+0000 7fc3c8c00700 1 -- 172.21.15.97:0/27529036 >> 172.21.15.97:0/27529036 conn(0x7fc3c40ff950 msgr2=0x7fc3c4101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:04.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.968+0000 7fc3c8c00700 1 -- 172.21.15.97:0/27529036 shutdown_connections 2024-03-11T19:57:04.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:04.969+0000 7fc3c8c00700 1 -- 172.21.15.97:0/27529036 wait complete. 2024-03-11T19:57:06.004 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:05 smithi097 ceph-mon[31898]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-11T19:57:06.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:05 smithi161 ceph-mon[36332]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-11T19:57:06.041 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:05 smithi114 ceph-mon[36376]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-11T19:57:07.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.694+0000 7f3c98c00700 1 -- 172.21.15.97:0/3429840514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c941084e0 msgr2=0x7f3c941088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:07.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.694+0000 7f3c98c00700 1 --2- 172.21.15.97:0/3429840514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c941084e0 0x7f3c941088c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3c84009990 tx=0x7f3c84009ca0 comp rx=0 tx=0).stop 2024-03-11T19:57:07.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.694+0000 7f3c98c00700 1 -- 172.21.15.97:0/3429840514 shutdown_connections 2024-03-11T19:57:07.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.694+0000 7f3c98c00700 1 --2- 172.21.15.97:0/3429840514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c9410d5c0 0x7f3c94115190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:07.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.694+0000 7f3c98c00700 1 --2- 172.21.15.97:0/3429840514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c941084e0 0x7f3c941088c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:07.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.694+0000 7f3c98c00700 1 --2- 172.21.15.97:0/3429840514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c94108e90 0x7f3c9410cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:07.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.694+0000 7f3c98c00700 1 -- 172.21.15.97:0/3429840514 >> 172.21.15.97:0/3429840514 conn(0x7f3c94075440 msgr2=0x7f3c94077860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:07.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.694+0000 7f3c98c00700 1 -- 172.21.15.97:0/3429840514 shutdown_connections 2024-03-11T19:57:07.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.694+0000 7f3c98c00700 1 -- 172.21.15.97:0/3429840514 wait complete. 2024-03-11T19:57:07.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.696+0000 7f3c98c00700 1 Processor -- start 2024-03-11T19:57:07.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.696+0000 7f3c98c00700 1 -- start start 2024-03-11T19:57:07.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.696+0000 7f3c98c00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c941084e0 0x7f3c941a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:07.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.696+0000 7f3c98c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c94108e90 0x7f3c941a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:07.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.696+0000 7f3c98c00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c9410d5c0 0x7f3c9419e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:07.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.696+0000 7f3c98c00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9419ec20 con 0x7f3c9410d5c0 2024-03-11T19:57:07.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.696+0000 7f3c98c00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9419ed60 con 0x7f3c94108e90 2024-03-11T19:57:07.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.696+0000 7f3c98c00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9419ef20 con 0x7f3c941084e0 2024-03-11T19:57:07.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.697+0000 7f3c92800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c9410d5c0 0x7f3c9419e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:07.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.697+0000 7f3c8be00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c94108e90 0x7f3c941a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:07.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.697+0000 7f3c92800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c9410d5c0 0x7f3c9419e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43448/0 (socket says 172.21.15.97:43448) 2024-03-11T19:57:07.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.697+0000 7f3c92800700 1 -- 172.21.15.97:0/171957309 learned_addr learned my addr 172.21.15.97:0/171957309 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:07.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.697+0000 7f3c91e00700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c941084e0 0x7f3c941a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:07.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.697+0000 7f3c8be00700 1 -- 172.21.15.97:0/171957309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c941084e0 msgr2=0x7f3c941a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:07.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.697+0000 7f3c8be00700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c941084e0 0x7f3c941a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:07.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.697+0000 7f3c8be00700 1 -- 172.21.15.97:0/171957309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c9410d5c0 msgr2=0x7f3c9419e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:07.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.697+0000 7f3c8be00700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c9410d5c0 0x7f3c9419e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:07.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.697+0000 7f3c8be00700 1 -- 172.21.15.97:0/171957309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c840095f0 con 0x7f3c94108e90 2024-03-11T19:57:07.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.697+0000 7f3c92800700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3c9410d5c0 0x7f3c9419e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:07.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.698+0000 7f3c8be00700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c94108e90 0x7f3c941a4940 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c00e9c0 tx=0x7f3c7c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:07.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.698+0000 7f3c8aa00700 1 -- 172.21.15.97:0/171957309 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c00c9c0 con 0x7f3c94108e90 2024-03-11T19:57:07.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.698+0000 7f3c8aa00700 1 -- 172.21.15.97:0/171957309 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c7c010430 con 0x7f3c94108e90 2024-03-11T19:57:07.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.698+0000 7f3c8aa00700 1 -- 172.21.15.97:0/171957309 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c018640 con 0x7f3c94108e90 2024-03-11T19:57:07.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.698+0000 7f3c98c00700 1 -- 172.21.15.97:0/171957309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9419f200 con 0x7f3c94108e90 2024-03-11T19:57:07.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.698+0000 7f3c98c00700 1 -- 172.21.15.97:0/171957309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c941ab4a0 con 0x7f3c94108e90 2024-03-11T19:57:07.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.699+0000 7f3c98c00700 1 -- 172.21.15.97:0/171957309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3c941099c0 con 0x7f3c94108e90 2024-03-11T19:57:07.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.701+0000 7f3c8aa00700 1 -- 172.21.15.97:0/171957309 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f3c7c0105a0 con 0x7f3c94108e90 2024-03-11T19:57:07.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.702+0000 7f3c8aa00700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3c780a1bd0 0x7f3c780a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:07.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.702+0000 7f3c8aa00700 1 -- 172.21.15.97:0/171957309 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f3c7c014070 con 0x7f3c94108e90 2024-03-11T19:57:07.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.703+0000 7f3c91e00700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3c780a1bd0 0x7f3c780a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:07.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.703+0000 7f3c8aa00700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818] conn(0x7f3c780accb0 0x7f3c780af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:07.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.703+0000 7f3c8aa00700 1 -- 172.21.15.97:0/171957309 --> [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c780af820 con 0x7f3c780accb0 2024-03-11T19:57:07.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.703+0000 7f3c8aa00700 1 -- 172.21.15.97:0/171957309 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c7c0ce3a0 con 0x7f3c94108e90 2024-03-11T19:57:07.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.703+0000 7f3c92800700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818] conn(0x7f3c780accb0 0x7f3c780af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:07.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.703+0000 7f3c91e00700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3c780a1bd0 0x7f3c780a4090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400b2e0 tx=0x7f3c84005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:07.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.704+0000 7f3c92800700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818] conn(0x7f3c780accb0 0x7f3c780af150 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:07.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.705+0000 7f3c8aa00700 1 -- 172.21.15.97:0/171957309 <== osd.8 v2:172.21.15.161:6816/4061144818 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3c780af820 con 0x7f3c780accb0 2024-03-11T19:57:07.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.723+0000 7f3c98c00700 1 -- 172.21.15.97:0/171957309 --> [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3c94118670 con 0x7f3c780accb0 2024-03-11T19:57:07.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.723+0000 7f3c8aa00700 1 -- 172.21.15.97:0/171957309 <== osd.8 v2:172.21.15.161:6816/4061144818 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3c94118670 con 0x7f3c780accb0 2024-03-11T19:57:07.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.724+0000 7f3c98c00700 1 -- 172.21.15.97:0/171957309 >> [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818] conn(0x7f3c780accb0 msgr2=0x7f3c780af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:07.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.724+0000 7f3c98c00700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818] conn(0x7f3c780accb0 0x7f3c780af150 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:07.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.724+0000 7f3c98c00700 1 -- 172.21.15.97:0/171957309 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3c780a1bd0 msgr2=0x7f3c780a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:07.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.724+0000 7f3c98c00700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f3c780a1bd0 0x7f3c780a4090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400b2e0 tx=0x7f3c84005fd0 comp rx=0 tx=0).stop 2024-03-11T19:57:07.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.724+0000 7f3c98c00700 1 -- 172.21.15.97:0/171957309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c94108e90 msgr2=0x7f3c941a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:07.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.724+0000 7f3c98c00700 1 --2- 172.21.15.97:0/171957309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c94108e90 0x7f3c941a4940 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c00e9c0 tx=0x7f3c7c00ecd0 comp rx=0 tx=0).stop 2024-03-11T19:57:07.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.724+0000 7f3c92800700 1 -- 172.21.15.97:0/171957309 reap_dead start 2024-03-11T19:57:07.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.724+0000 7f3c98c00700 1 -- 172.21.15.97:0/171957309 shutdown_connections 2024-03-11T19:57:07.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.724+0000 7f3c98c00700 1 -- 172.21.15.97:0/171957309 >> 172.21.15.97:0/171957309 conn(0x7f3c94075440 msgr2=0x7f3c94077170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:07.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.725+0000 7f3c98c00700 1 -- 172.21.15.97:0/171957309 shutdown_connections 2024-03-11T19:57:07.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:06.725+0000 7f3c98c00700 1 -- 172.21.15.97:0/171957309 wait complete. 2024-03-11T19:57:07.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:07 smithi097 ceph-mon[31898]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-11T19:57:08.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:07 smithi161 ceph-mon[36332]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-11T19:57:08.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.142+0000 7f1239c00700 1 -- 172.21.15.97:0/639941706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1234108e90 msgr2=0x7f123410cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:08.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.142+0000 7f1239c00700 1 --2- 172.21.15.97:0/639941706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1234108e90 0x7f123410cee0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f121c009990 tx=0x7f121c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:57:08.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.142+0000 7f1239c00700 1 -- 172.21.15.97:0/639941706 shutdown_connections 2024-03-11T19:57:08.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.142+0000 7f1239c00700 1 --2- 172.21.15.97:0/639941706 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f123410d5c0 0x7f1234115190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:08.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.142+0000 7f1239c00700 1 --2- 172.21.15.97:0/639941706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12341084e0 0x7f12341088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:08.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.142+0000 7f1239c00700 1 --2- 172.21.15.97:0/639941706 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1234108e90 0x7f123410cee0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:08.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.142+0000 7f1239c00700 1 -- 172.21.15.97:0/639941706 >> 172.21.15.97:0/639941706 conn(0x7f1234075440 msgr2=0x7f1234077860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:08.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.143+0000 7f1239c00700 1 -- 172.21.15.97:0/639941706 shutdown_connections 2024-03-11T19:57:08.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.143+0000 7f1239c00700 1 -- 172.21.15.97:0/639941706 wait complete. 2024-03-11T19:57:08.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.144+0000 7f1239c00700 1 Processor -- start 2024-03-11T19:57:08.187 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.144+0000 7f1239c00700 1 -- start start 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.144+0000 7f1239c00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12341084e0 0x7f12341a4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.144+0000 7f1239c00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1234108e90 0x7f12341a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.144+0000 7f1239c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123410d5c0 0x7f123419e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.144+0000 7f1239c00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123419ec40 con 0x7f12341084e0 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.144+0000 7f1239c00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123419ed80 con 0x7f123410d5c0 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.144+0000 7f1239c00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123419ef40 con 0x7f1234108e90 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.145+0000 7f1232a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12341084e0 0x7f12341a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.145+0000 7f1232a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12341084e0 0x7f12341a4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43460/0 (socket says 172.21.15.97:43460) 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.145+0000 7f1232a00700 1 -- 172.21.15.97:0/793913365 learned_addr learned my addr 172.21.15.97:0/793913365 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.145+0000 7f122be00700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1234108e90 0x7f12341a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.145+0000 7f1233400700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123410d5c0 0x7f123419e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.145+0000 7f1232a00700 1 -- 172.21.15.97:0/793913365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1234108e90 msgr2=0x7f12341a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.145+0000 7f1232a00700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1234108e90 0x7f12341a4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:08.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.145+0000 7f1232a00700 1 -- 172.21.15.97:0/793913365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123410d5c0 msgr2=0x7f123419e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.145+0000 7f1232a00700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f123410d5c0 0x7f123419e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.145+0000 7f1232a00700 1 -- 172.21.15.97:0/793913365 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c0095f0 con 0x7f12341084e0 2024-03-11T19:57:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.145+0000 7f122be00700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1234108e90 0x7f12341a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.146+0000 7f1232a00700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12341084e0 0x7f12341a4420 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f122400d950 tx=0x7f122400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.146+0000 7f122b400700 1 -- 172.21.15.97:0/793913365 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122400f890 con 0x7f12341084e0 2024-03-11T19:57:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.146+0000 7f122b400700 1 -- 172.21.15.97:0/793913365 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1224010460 con 0x7f12341084e0 2024-03-11T19:57:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.146+0000 7f122b400700 1 -- 172.21.15.97:0/793913365 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122400e610 con 0x7f12341084e0 2024-03-11T19:57:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.146+0000 7f1239c00700 1 -- 172.21.15.97:0/793913365 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123419f220 con 0x7f12341084e0 2024-03-11T19:57:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.146+0000 7f1239c00700 1 -- 172.21.15.97:0/793913365 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12341ab4c0 con 0x7f12341084e0 2024-03-11T19:57:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.147+0000 7f1239c00700 1 -- 172.21.15.97:0/793913365 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f12341099c0 con 0x7f12341084e0 2024-03-11T19:57:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.150+0000 7f122b400700 1 -- 172.21.15.97:0/793913365 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1224010960 con 0x7f12341084e0 2024-03-11T19:57:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.151+0000 7f122b400700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12180a1bd0 0x7f12180a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:08.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.151+0000 7f122b400700 1 -- 172.21.15.97:0/793913365 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f12240cdb90 con 0x7f12341084e0 2024-03-11T19:57:08.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.151+0000 7f122be00700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12180a1bd0 0x7f12180a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:08.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.151+0000 7f122b400700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428] conn(0x7f12180accb0 0x7f12180af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:08.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.152+0000 7f122b400700 1 -- 172.21.15.97:0/793913365 --> [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f12180af820 con 0x7f12180accb0 2024-03-11T19:57:08.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.152+0000 7f122b400700 1 -- 172.21.15.97:0/793913365 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12240cdf00 con 0x7f12341084e0 2024-03-11T19:57:08.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.152+0000 7f1233400700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428] conn(0x7f12180accb0 0x7f12180af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:08.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.152+0000 7f122be00700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12180a1bd0 0x7f12180a4090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1234107510 tx=0x7f121c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:08.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.152+0000 7f1233400700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428] conn(0x7f12180accb0 0x7f12180af150 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:08.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.153+0000 7f122b400700 1 -- 172.21.15.97:0/793913365 <== osd.9 v2:172.21.15.97:6826/2375353428 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f12180af820 con 0x7f12180accb0 2024-03-11T19:57:08.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.171+0000 7f1239c00700 1 -- 172.21.15.97:0/793913365 --> [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1234118650 con 0x7f12180accb0 2024-03-11T19:57:08.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.172+0000 7f122b400700 1 -- 172.21.15.97:0/793913365 <== osd.9 v2:172.21.15.97:6826/2375353428 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1234118650 con 0x7f12180accb0 2024-03-11T19:57:08.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.172+0000 7f1239c00700 1 -- 172.21.15.97:0/793913365 >> [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428] conn(0x7f12180accb0 msgr2=0x7f12180af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:08.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.172+0000 7f1239c00700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428] conn(0x7f12180accb0 0x7f12180af150 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:08.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.173+0000 7f1239c00700 1 -- 172.21.15.97:0/793913365 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12180a1bd0 msgr2=0x7f12180a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:08.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.173+0000 7f1239c00700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f12180a1bd0 0x7f12180a4090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1234107510 tx=0x7f121c00b540 comp rx=0 tx=0).stop 2024-03-11T19:57:08.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.173+0000 7f1239c00700 1 -- 172.21.15.97:0/793913365 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12341084e0 msgr2=0x7f12341a4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:08.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.173+0000 7f1239c00700 1 --2- 172.21.15.97:0/793913365 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12341084e0 0x7f12341a4420 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f122400d950 tx=0x7f122400dc60 comp rx=0 tx=0).stop 2024-03-11T19:57:08.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.173+0000 7f1233400700 1 -- 172.21.15.97:0/793913365 reap_dead start 2024-03-11T19:57:08.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.173+0000 7f1239c00700 1 -- 172.21.15.97:0/793913365 shutdown_connections 2024-03-11T19:57:08.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.173+0000 7f1239c00700 1 -- 172.21.15.97:0/793913365 >> 172.21.15.97:0/793913365 conn(0x7f1234075440 msgr2=0x7f1234075f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:08.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.173+0000 7f1239c00700 1 -- 172.21.15.97:0/793913365 shutdown_connections 2024-03-11T19:57:08.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:08.173+0000 7f1239c00700 1 -- 172.21.15.97:0/793913365 wait complete. 2024-03-11T19:57:08.240 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:07 smithi114 ceph-mon[36376]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-11T19:57:09.267 INFO:teuthology.orchestra.run.smithi097.stdout:309237645347 2024-03-11T19:57:09.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-11T19:57:09.276 INFO:teuthology.orchestra.run.smithi097.stdout:141733920838 2024-03-11T19:57:09.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T19:57:09.795 INFO:teuthology.orchestra.run.smithi097.stdout:253403070513 2024-03-11T19:57:09.795 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-11T19:57:10.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:09 smithi161 ceph-mon[36332]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-03-11T19:57:10.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:09 smithi097 ceph-mon[31898]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-03-11T19:57:10.239 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:09 smithi114 ceph-mon[36376]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-03-11T19:57:10.273 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:10.295 INFO:teuthology.orchestra.run.smithi097.stdout:107374182476 2024-03-11T19:57:10.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T19:57:11.019 INFO:teuthology.orchestra.run.smithi097.stdout:287762808873 2024-03-11T19:57:11.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-11T19:57:11.026 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:11.367 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:11.372 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:11.376 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:11.854 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:11 smithi097 ceph-mon[31898]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-11T19:57:11.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.860+0000 7fa158400700 1 -- 172.21.15.97:0/3023579964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa150104a90 msgr2=0x7fa150108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:11.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.860+0000 7fa158400700 1 --2- 172.21.15.97:0/3023579964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa150104a90 0x7fa150108ae0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009990 tx=0x7fa13c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:57:11.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.860+0000 7fa158400700 1 -- 172.21.15.97:0/3023579964 shutdown_connections 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.860+0000 7fa158400700 1 --2- 172.21.15.97:0/3023579964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1501091c0 0x7fa150110d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.860+0000 7fa158400700 1 --2- 172.21.15.97:0/3023579964 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1501040e0 0x7fa1501044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.860+0000 7fa158400700 1 --2- 172.21.15.97:0/3023579964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa150104a90 0x7fa150108ae0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.860+0000 7fa158400700 1 -- 172.21.15.97:0/3023579964 >> 172.21.15.97:0/3023579964 conn(0x7fa1500ff970 msgr2=0x7fa150101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.860+0000 7fa158400700 1 -- 172.21.15.97:0/3023579964 shutdown_connections 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.860+0000 7fa158400700 1 -- 172.21.15.97:0/3023579964 wait complete. 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.861+0000 7fa158400700 1 Processor -- start 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.862+0000 7fa158400700 1 -- start start 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.862+0000 7fa158400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1501040e0 0x7fa15019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.862+0000 7fa158400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa150104a90 0x7fa15019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.862+0000 7fa158400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1501091c0 0x7fa15019ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.862+0000 7fa158400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15019f2c0 con 0x7fa1501091c0 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.862+0000 7fa158400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15019f430 con 0x7fa150104a90 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.862+0000 7fa158400700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15019f5f0 con 0x7fa1501040e0 2024-03-11T19:57:11.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.862+0000 7fa156400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1501091c0 0x7fa15019ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:11.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.862+0000 7fa156400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1501091c0 0x7fa15019ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:43474/0 (socket says 172.21.15.97:43474) 2024-03-11T19:57:11.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.863+0000 7fa156400700 1 -- 172.21.15.97:0/793401414 learned_addr learned my addr 172.21.15.97:0/793401414 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:11.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.863+0000 7fa155000700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa150104a90 0x7fa15019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:11.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.863+0000 7fa155a00700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1501040e0 0x7fa15019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:11.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.863+0000 7fa156400700 1 -- 172.21.15.97:0/793401414 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1501040e0 msgr2=0x7fa15019a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:11.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.863+0000 7fa156400700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1501040e0 0x7fa15019a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:11.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.863+0000 7fa156400700 1 -- 172.21.15.97:0/793401414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa150104a90 msgr2=0x7fa15019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:11.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.863+0000 7fa156400700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa150104a90 0x7fa15019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:11.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.863+0000 7fa156400700 1 -- 172.21.15.97:0/793401414 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c0095f0 con 0x7fa1501091c0 2024-03-11T19:57:11.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.863+0000 7fa156400700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1501091c0 0x7fa15019ecb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa148007ee0 tx=0x7fa14800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:11.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.863+0000 7fa14e000700 1 -- 172.21.15.97:0/793401414 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14800dd20 con 0x7fa1501091c0 2024-03-11T19:57:11.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.864+0000 7fa14e000700 1 -- 172.21.15.97:0/793401414 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa14800f040 con 0x7fa1501091c0 2024-03-11T19:57:11.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.864+0000 7fa14e000700 1 -- 172.21.15.97:0/793401414 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1480048e0 con 0x7fa1501091c0 2024-03-11T19:57:11.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.864+0000 7fa158400700 1 -- 172.21.15.97:0/793401414 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15019f8d0 con 0x7fa1501091c0 2024-03-11T19:57:11.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.864+0000 7fa158400700 1 -- 172.21.15.97:0/793401414 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1501a7130 con 0x7fa1501091c0 2024-03-11T19:57:11.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.865+0000 7fa158400700 1 -- 172.21.15.97:0/793401414 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa150114250 con 0x7fa1501091c0 2024-03-11T19:57:11.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.867+0000 7fa14e000700 1 -- 172.21.15.97:0/793401414 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fa148003a00 con 0x7fa1501091c0 2024-03-11T19:57:11.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.868+0000 7fa14e000700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa1380a1b50 0x7fa1380a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:11.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.868+0000 7fa14e000700 1 -- 172.21.15.97:0/793401414 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fa1480cd100 con 0x7fa1501091c0 2024-03-11T19:57:11.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.868+0000 7fa155a00700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa1380a1b50 0x7fa1380a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:11.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.868+0000 7fa14e000700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517] conn(0x7fa1380acac0 0x7fa1380aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.868+0000 7fa14e000700 1 -- 172.21.15.97:0/793401414 --> [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1380af630 con 0x7fa1380acac0 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.868+0000 7fa14e000700 1 -- 172.21.15.97:0/793401414 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1480cf6e0 con 0x7fa1501091c0 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.869+0000 7fa155000700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517] conn(0x7fa1380acac0 0x7fa1380aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.869+0000 7fa155a00700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa1380a1b50 0x7fa1380a4010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa144009c60 tx=0x7fa144009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.869+0000 7fa155000700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517] conn(0x7fa1380acac0 0x7fa1380aef60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.870+0000 7fa14e000700 1 -- 172.21.15.97:0/793401414 <== osd.6 v2:172.21.15.97:6818/2094652517 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa1380af630 con 0x7fa1380acac0 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.887+0000 7fa158400700 1 -- 172.21.15.97:0/793401414 --> [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa150062380 con 0x7fa1380acac0 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.888+0000 7fa14e000700 1 -- 172.21.15.97:0/793401414 <== osd.6 v2:172.21.15.97:6818/2094652517 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa150062380 con 0x7fa1380acac0 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.888+0000 7fa158400700 1 -- 172.21.15.97:0/793401414 >> [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517] conn(0x7fa1380acac0 msgr2=0x7fa1380aef60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.888+0000 7fa158400700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517] conn(0x7fa1380acac0 0x7fa1380aef60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.889+0000 7fa158400700 1 -- 172.21.15.97:0/793401414 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa1380a1b50 msgr2=0x7fa1380a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.889+0000 7fa158400700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa1380a1b50 0x7fa1380a4010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa144009c60 tx=0x7fa144009260 comp rx=0 tx=0).stop 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.889+0000 7fa158400700 1 -- 172.21.15.97:0/793401414 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1501091c0 msgr2=0x7fa15019ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.889+0000 7fa158400700 1 --2- 172.21.15.97:0/793401414 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1501091c0 0x7fa15019ecb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa148007ee0 tx=0x7fa14800d3b0 comp rx=0 tx=0).stop 2024-03-11T19:57:11.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.889+0000 7fa156400700 1 -- 172.21.15.97:0/793401414 reap_dead start 2024-03-11T19:57:11.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.889+0000 7fa158400700 1 -- 172.21.15.97:0/793401414 shutdown_connections 2024-03-11T19:57:11.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.889+0000 7fa158400700 1 -- 172.21.15.97:0/793401414 >> 172.21.15.97:0/793401414 conn(0x7fa1500ff970 msgr2=0x7fa150101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:11.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.889+0000 7fa158400700 1 -- 172.21.15.97:0/793401414 shutdown_connections 2024-03-11T19:57:11.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:11.889+0000 7fa158400700 1 -- 172.21.15.97:0/793401414 wait complete. 2024-03-11T19:57:12.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:11 smithi161 ceph-mon[36332]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-11T19:57:12.239 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:11 smithi114 ceph-mon[36376]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-11T19:57:12.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:12 smithi097 ceph-mon[31898]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 11 op/s 2024-03-11T19:57:13.239 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:12 smithi114 ceph-mon[36376]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 11 op/s 2024-03-11T19:57:13.285 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:12 smithi161 ceph-mon[36332]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 11 op/s 2024-03-11T19:57:14.141 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:15.647 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:15 smithi097 ceph-mon[31898]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:57:15.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:15 smithi114 ceph-mon[36376]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:57:16.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:15 smithi161 ceph-mon[36332]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:57:17.374 INFO:teuthology.orchestra.run.smithi097.stdout:197568495675 2024-03-11T19:57:17.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T19:57:17.897 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:17 smithi097 ceph-mon[31898]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-11T19:57:17.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:17 smithi114 ceph-mon[36376]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-11T19:57:18.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:17 smithi161 ceph-mon[36332]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-11T19:57:18.536 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:19.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:19 smithi097 ceph-mon[31898]: pgmap v368: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-11T19:57:19.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:19 smithi114 ceph-mon[36376]: pgmap v368: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-11T19:57:20.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:19 smithi161 ceph-mon[36332]: pgmap v368: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-11T19:57:21.238 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:21.241 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:21.363 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:21.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:20.997+0000 7f0677e00700 1 -- 172.21.15.97:0/3542759859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0678112b60 msgr2=0x7f0678114f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:21.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:20.997+0000 7f0677e00700 1 --2- 172.21.15.97:0/3542759859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0678112b60 0x7f0678114f50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f067000b0a0 tx=0x7f067000b3b0 comp rx=0 tx=0).stop 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:20.997+0000 7f0677e00700 1 -- 172.21.15.97:0/3542759859 shutdown_connections 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:20.997+0000 7f0677e00700 1 --2- 172.21.15.97:0/3542759859 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0678112b60 0x7f0678114f50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:20.997+0000 7f0677e00700 1 --2- 172.21.15.97:0/3542759859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0678074010 0x7f06780743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:20.997+0000 7f0677e00700 1 --2- 172.21.15.97:0/3542759859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0678074930 0x7f0678112620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:20.997+0000 7f0677e00700 1 -- 172.21.15.97:0/3542759859 >> 172.21.15.97:0/3542759859 conn(0x7f06780fd640 msgr2=0x7f06780ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:20.997+0000 7f0677e00700 1 -- 172.21.15.97:0/3542759859 shutdown_connections 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:20.999+0000 7f0677e00700 1 -- 172.21.15.97:0/3542759859 wait complete. 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.000+0000 7f0677e00700 1 Processor -- start 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.000+0000 7f0677e00700 1 -- start start 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.001+0000 7f0677e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0678074010 0x7f06781a4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.001+0000 7f0677e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0678074930 0x7f06781a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.001+0000 7f0677e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0678112b60 0x7f067819e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.001+0000 7f0677e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067819ec30 con 0x7f0678112b60 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.001+0000 7f0677e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067819ed70 con 0x7f0678074930 2024-03-11T19:57:21.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.001+0000 7f0677e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067819ef30 con 0x7f0678074010 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.001+0000 7f0676a00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0678074010 0x7f06781a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.001+0000 7f0677400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0678112b60 0x7f067819e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.001+0000 7f0676a00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0678074010 0x7f06781a4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:39676/0 (socket says 172.21.15.97:39676) 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.001+0000 7f0676a00700 1 -- 172.21.15.97:0/3595924680 learned_addr learned my addr 172.21.15.97:0/3595924680 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.001+0000 7f0676000700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0678074930 0x7f06781a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.002+0000 7f0676a00700 1 -- 172.21.15.97:0/3595924680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0678074930 msgr2=0x7f06781a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.002+0000 7f0676a00700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0678074930 0x7f06781a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.002+0000 7f0676a00700 1 -- 172.21.15.97:0/3595924680 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0678112b60 msgr2=0x7f067819e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.002+0000 7f0676a00700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0678112b60 0x7f067819e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.002+0000 7f0676a00700 1 -- 172.21.15.97:0/3595924680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0670009d00 con 0x7f0678074010 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.002+0000 7f0676000700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0678074930 0x7f06781a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.002+0000 7f0676a00700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0678074010 0x7f06781a4410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f066800e9a0 tx=0x7f066800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.002+0000 7f067ca00700 1 -- 172.21.15.97:0/3595924680 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0668009790 con 0x7f0678074010 2024-03-11T19:57:21.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.002+0000 7f0677e00700 1 -- 172.21.15.97:0/3595924680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067819f210 con 0x7f0678074010 2024-03-11T19:57:21.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.002+0000 7f067ca00700 1 -- 172.21.15.97:0/3595924680 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0668010430 con 0x7f0678074010 2024-03-11T19:57:21.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.002+0000 7f067ca00700 1 -- 172.21.15.97:0/3595924680 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0668018680 con 0x7f0678074010 2024-03-11T19:57:21.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.003+0000 7f0677e00700 1 -- 172.21.15.97:0/3595924680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06781ab4b0 con 0x7f0678074010 2024-03-11T19:57:21.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.004+0000 7f0677e00700 1 -- 172.21.15.97:0/3595924680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0678105040 con 0x7f0678074010 2024-03-11T19:57:21.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.006+0000 7f067ca00700 1 -- 172.21.15.97:0/3595924680 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f0668010910 con 0x7f0678074010 2024-03-11T19:57:21.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.007+0000 7f067ca00700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f065c0a1b50 0x7f065c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:21.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.007+0000 7f067ca00700 1 -- 172.21.15.97:0/3595924680 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f0668014070 con 0x7f0678074010 2024-03-11T19:57:21.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.007+0000 7f0676000700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f065c0a1b50 0x7f065c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:21.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.008+0000 7f0676000700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f065c0a1b50 0x7f065c0a4010 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0678105910 tx=0x7f0664009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:21.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.009+0000 7f067ca00700 1 -- 172.21.15.97:0/3595924680 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0668096d40 con 0x7f0678074010 2024-03-11T19:57:21.611 INFO:teuthology.orchestra.run.smithi097.stdout:163208757316 2024-03-11T19:57:21.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.254+0000 7f0677e00700 1 -- 172.21.15.97:0/3595924680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f067819fb00 con 0x7f0678074010 2024-03-11T19:57:21.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.254+0000 7f067ca00700 1 -- 172.21.15.97:0/3595924680 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f06680105a0 con 0x7f0678074010 2024-03-11T19:57:21.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.259+0000 7f0677e00700 1 -- 172.21.15.97:0/3595924680 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f065c0a1b50 msgr2=0x7f065c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:21.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.259+0000 7f0677e00700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f065c0a1b50 0x7f065c0a4010 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0678105910 tx=0x7f0664009260 comp rx=0 tx=0).stop 2024-03-11T19:57:21.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.259+0000 7f0677e00700 1 -- 172.21.15.97:0/3595924680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0678074010 msgr2=0x7f06781a4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:21.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.259+0000 7f0677e00700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0678074010 0x7f06781a4410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f066800e9a0 tx=0x7f066800ecb0 comp rx=0 tx=0).stop 2024-03-11T19:57:21.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.259+0000 7f0677e00700 1 -- 172.21.15.97:0/3595924680 shutdown_connections 2024-03-11T19:57:21.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.259+0000 7f0677e00700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f065c0a1b50 0x7f065c0a4010 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:21.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.259+0000 7f0677e00700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0678112b60 0x7f067819e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:21.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.259+0000 7f0677e00700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0678074010 0x7f06781a4410 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:21.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.259+0000 7f0677e00700 1 --2- 172.21.15.97:0/3595924680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0678074930 0x7f06781a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:21.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.259+0000 7f0677e00700 1 -- 172.21.15.97:0/3595924680 >> 172.21.15.97:0/3595924680 conn(0x7f06780fd640 msgr2=0x7f06780fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:21.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.260+0000 7f0677e00700 1 -- 172.21.15.97:0/3595924680 shutdown_connections 2024-03-11T19:57:21.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.260+0000 7f0677e00700 1 -- 172.21.15.97:0/3595924680 wait complete. 2024-03-11T19:57:21.667 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:21 smithi097 ceph-mon[31898]: pgmap v369: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-03-11T19:57:21.667 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:21 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3595924680' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T19:57:21.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:21 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2523079279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T19:57:21.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:21 smithi114 ceph-mon[36376]: pgmap v369: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-03-11T19:57:21.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:21 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3595924680' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T19:57:21.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:21 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2523079279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T19:57:22.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:21 smithi161 ceph-mon[36332]: pgmap v369: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-03-11T19:57:22.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:21 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3595924680' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T19:57:22.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:21 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2523079279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T19:57:22.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.121+0000 7f66aa000700 1 -- 172.21.15.97:0/866544301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4100b70 msgr2=0x7f66a4100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:22.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.121+0000 7f66aa000700 1 --2- 172.21.15.97:0/866544301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4100b70 0x7f66a4100ff0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6690009990 tx=0x7f6690009ca0 comp rx=0 tx=0).stop 2024-03-11T19:57:22.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.121+0000 7f66aa000700 1 -- 172.21.15.97:0/866544301 shutdown_connections 2024-03-11T19:57:22.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.121+0000 7f66aa000700 1 --2- 172.21.15.97:0/866544301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a4101530 0x7f66a410d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.121+0000 7f66aa000700 1 --2- 172.21.15.97:0/866544301 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66a4106b80 0x7f66a4106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.121+0000 7f66aa000700 1 --2- 172.21.15.97:0/866544301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4100b70 0x7f66a4100ff0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.121+0000 7f66aa000700 1 -- 172.21.15.97:0/866544301 >> 172.21.15.97:0/866544301 conn(0x7f66a4077aa0 msgr2=0x7f66a4077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.121+0000 7f66aa000700 1 -- 172.21.15.97:0/866544301 shutdown_connections 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.121+0000 7f66aa000700 1 -- 172.21.15.97:0/866544301 wait complete. 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.123+0000 7f66aa000700 1 Processor -- start 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.123+0000 7f66aa000700 1 -- start start 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.123+0000 7f66aa000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4100b70 0x7f66a419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.123+0000 7f66aa000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a4101530 0x7f66a419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.123+0000 7f66aa000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66a4106b80 0x7f66a419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.123+0000 7f66aa000700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a419f1d0 con 0x7f66a4100b70 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.123+0000 7f66aa000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a419f340 con 0x7f66a4101530 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.123+0000 7f66aa000700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a419f500 con 0x7f66a4106b80 2024-03-11T19:57:22.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.124+0000 7f66a2a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a4101530 0x7f66a419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.124+0000 7f66a3400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4100b70 0x7f66a419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.124+0000 7f66a2a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a4101530 0x7f66a419a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:48140/0 (socket says 172.21.15.97:48140) 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.124+0000 7f66a2a00700 1 -- 172.21.15.97:0/2523079279 learned_addr learned my addr 172.21.15.97:0/2523079279 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.124+0000 7f66a3e00700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66a4106b80 0x7f66a419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.124+0000 7f66a2a00700 1 -- 172.21.15.97:0/2523079279 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66a4106b80 msgr2=0x7f66a419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.124+0000 7f66a2a00700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66a4106b80 0x7f66a419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.124+0000 7f66a2a00700 1 -- 172.21.15.97:0/2523079279 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4100b70 msgr2=0x7f66a419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.124+0000 7f66a2a00700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4100b70 0x7f66a419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.124+0000 7f66a2a00700 1 -- 172.21.15.97:0/2523079279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66900095f0 con 0x7f66a4101530 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.124+0000 7f66a3e00700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66a4106b80 0x7f66a419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.125+0000 7f66a2a00700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a4101530 0x7f66a419a830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6690000c00 tx=0x7f6690005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.125+0000 7f669be00700 1 -- 172.21.15.97:0/2523079279 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669001d070 con 0x7f66a4101530 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.125+0000 7f669be00700 1 -- 172.21.15.97:0/2523079279 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f669000b610 con 0x7f66a4101530 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.125+0000 7f66aa000700 1 -- 172.21.15.97:0/2523079279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a419f780 con 0x7f66a4101530 2024-03-11T19:57:22.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.125+0000 7f669be00700 1 -- 172.21.15.97:0/2523079279 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66900173f0 con 0x7f66a4101530 2024-03-11T19:57:22.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.125+0000 7f66aa000700 1 -- 172.21.15.97:0/2523079279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a41a7040 con 0x7f66a4101530 2024-03-11T19:57:22.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.126+0000 7f66aa000700 1 -- 172.21.15.97:0/2523079279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a41021f0 con 0x7f66a4101530 2024-03-11T19:57:22.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.130+0000 7f669be00700 1 -- 172.21.15.97:0/2523079279 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f6690022670 con 0x7f66a4101530 2024-03-11T19:57:22.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.132+0000 7f669be00700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66880a1bd0 0x7f66880a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:22.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.132+0000 7f669be00700 1 -- 172.21.15.97:0/2523079279 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f66900cf250 con 0x7f66a4101530 2024-03-11T19:57:22.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.132+0000 7f66a3400700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66880a1bd0 0x7f66880a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:22.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.132+0000 7f669be00700 1 -- 172.21.15.97:0/2523079279 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6690097ff0 con 0x7f66a4101530 2024-03-11T19:57:22.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.133+0000 7f66a3400700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66880a1bd0 0x7f66880a4090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6694009c60 tx=0x7f6694009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:22.039 INFO:teuthology.orchestra.run.smithi097.stdout:77309411414 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.416+0000 7f66aa000700 1 -- 172.21.15.97:0/2523079279 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f66a4068930 con 0x7f66a4101530 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.416+0000 7f669be00700 1 -- 172.21.15.97:0/2523079279 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f669009b6b0 con 0x7f66a4101530 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.419+0000 7f66aa000700 1 -- 172.21.15.97:0/2523079279 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66880a1bd0 msgr2=0x7f66880a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.419+0000 7f66aa000700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66880a1bd0 0x7f66880a4090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6694009c60 tx=0x7f6694009260 comp rx=0 tx=0).stop 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.419+0000 7f66aa000700 1 -- 172.21.15.97:0/2523079279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a4101530 msgr2=0x7f66a419a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.419+0000 7f66aa000700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a4101530 0x7f66a419a830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6690000c00 tx=0x7f6690005740 comp rx=0 tx=0).stop 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.419+0000 7f66aa000700 1 -- 172.21.15.97:0/2523079279 shutdown_connections 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.419+0000 7f66aa000700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f66880a1bd0 0x7f66880a4090 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.419+0000 7f66aa000700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f66a4100b70 0x7f66a419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.419+0000 7f66aa000700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66a4106b80 0x7f66a419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.419+0000 7f66aa000700 1 --2- 172.21.15.97:0/2523079279 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66a4101530 0x7f66a419a830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.419+0000 7f66aa000700 1 -- 172.21.15.97:0/2523079279 >> 172.21.15.97:0/2523079279 conn(0x7f66a4077aa0 msgr2=0x7f66a4069920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.419+0000 7f66aa000700 1 -- 172.21.15.97:0/2523079279 shutdown_connections 2024-03-11T19:57:22.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:21.419+0000 7f66aa000700 1 -- 172.21.15.97:0/2523079279 wait complete. 2024-03-11T19:57:23.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.050+0000 7f990e600700 1 -- 172.21.15.97:0/1771067551 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9908101f20 msgr2=0x7f9908102300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:23.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.050+0000 7f990e600700 1 --2- 172.21.15.97:0/1771067551 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9908101f20 0x7f9908102300 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f98f8009990 tx=0x7f98f8009ca0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.051+0000 7f990e600700 1 -- 172.21.15.97:0/1771067551 shutdown_connections 2024-03-11T19:57:23.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.051+0000 7f990e600700 1 --2- 172.21.15.97:0/1771067551 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9908101f20 0x7f9908102300 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.051+0000 7f990e600700 1 --2- 172.21.15.97:0/1771067551 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9908106ff0 0x7f990810ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.051 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.051+0000 7f990e600700 1 --2- 172.21.15.97:0/1771067551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99081028d0 0x7f9908106910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.051+0000 7f990e600700 1 -- 172.21.15.97:0/1771067551 >> 172.21.15.97:0/1771067551 conn(0x7f9908078dd0 msgr2=0x7f99080791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:23.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.051+0000 7f990e600700 1 -- 172.21.15.97:0/1771067551 shutdown_connections 2024-03-11T19:57:23.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.051+0000 7f990e600700 1 -- 172.21.15.97:0/1771067551 wait complete. 2024-03-11T19:57:23.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.052+0000 7f990e600700 1 Processor -- start 2024-03-11T19:57:23.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.052+0000 7f990e600700 1 -- start start 2024-03-11T19:57:23.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.052+0000 7f990e600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9908101f20 0x7f990819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:23.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.052+0000 7f990e600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99081028d0 0x7f990819a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:23.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.052+0000 7f990e600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9908106ff0 0x7f990819ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:23.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.052+0000 7f990e600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990819f330 con 0x7f9908106ff0 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.052+0000 7f990e600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990819f4a0 con 0x7f9908101f20 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.053+0000 7f990e600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990819f660 con 0x7f99081028d0 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.053+0000 7f9907e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9908106ff0 0x7f990819ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.053+0000 7f9907e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9908106ff0 0x7f990819ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56294/0 (socket says 172.21.15.97:56294) 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.053+0000 7f9907400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9908101f20 0x7f990819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.053+0000 7f9907e00700 1 -- 172.21.15.97:0/458028605 learned_addr learned my addr 172.21.15.97:0/458028605 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.053+0000 7f9906a00700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99081028d0 0x7f990819a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.053+0000 7f9907e00700 1 -- 172.21.15.97:0/458028605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99081028d0 msgr2=0x7f990819a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.053+0000 7f9907e00700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99081028d0 0x7f990819a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.053+0000 7f9907e00700 1 -- 172.21.15.97:0/458028605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9908101f20 msgr2=0x7f990819a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.053+0000 7f9907e00700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9908101f20 0x7f990819a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.053+0000 7f9907e00700 1 -- 172.21.15.97:0/458028605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9900009c40 con 0x7f9908106ff0 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.053+0000 7f9907400700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9908101f20 0x7f990819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.054+0000 7f9907e00700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9908106ff0 0x7f990819ed20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f990000c0f0 tx=0x7f990000c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.054+0000 7f98ffe00700 1 -- 172.21.15.97:0/458028605 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990000cab0 con 0x7f9908106ff0 2024-03-11T19:57:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.054+0000 7f98ffe00700 1 -- 172.21.15.97:0/458028605 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f990000e030 con 0x7f9908106ff0 2024-03-11T19:57:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.054+0000 7f990e600700 1 -- 172.21.15.97:0/458028605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f80095f0 con 0x7f9908106ff0 2024-03-11T19:57:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.054+0000 7f98ffe00700 1 -- 172.21.15.97:0/458028605 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99000047a0 con 0x7f9908106ff0 2024-03-11T19:57:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.054+0000 7f990e600700 1 -- 172.21.15.97:0/458028605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99081a6fd0 con 0x7f9908106ff0 2024-03-11T19:57:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.055+0000 7f990e600700 1 -- 172.21.15.97:0/458028605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9908068930 con 0x7f9908106ff0 2024-03-11T19:57:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.058+0000 7f98ffe00700 1 -- 172.21.15.97:0/458028605 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f9900003a00 con 0x7f9908106ff0 2024-03-11T19:57:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.059+0000 7f98ffe00700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f98ec0a1b50 0x7f98ec0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.059+0000 7f98ffe00700 1 -- 172.21.15.97:0/458028605 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f99000cdd90 con 0x7f9908106ff0 2024-03-11T19:57:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.059+0000 7f9907400700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f98ec0a1b50 0x7f98ec0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.060+0000 7f9907400700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f98ec0a1b50 0x7f98ec0a4010 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f98f8006010 tx=0x7f98f800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:23.062 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.062+0000 7f98ffe00700 1 -- 172.21.15.97:0/458028605 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9900096b30 con 0x7f9908106ff0 2024-03-11T19:57:23.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.286+0000 7f990e600700 1 -- 172.21.15.97:0/458028605 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f990819b7b0 con 0x7f9908106ff0 2024-03-11T19:57:23.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.287+0000 7f98ffe00700 1 -- 172.21.15.97:0/458028605 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f990009a1f0 con 0x7f9908106ff0 2024-03-11T19:57:23.287 INFO:teuthology.orchestra.run.smithi097.stdout:34359738464 2024-03-11T19:57:23.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7f990e600700 1 -- 172.21.15.97:0/458028605 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f98ec0a1b50 msgr2=0x7f98ec0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:23.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7f990e600700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f98ec0a1b50 0x7f98ec0a4010 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f98f8006010 tx=0x7f98f800b260 comp rx=0 tx=0).stop 2024-03-11T19:57:23.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7f990e600700 1 -- 172.21.15.97:0/458028605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9908106ff0 msgr2=0x7f990819ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:23.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7f990e600700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9908106ff0 0x7f990819ed20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f990000c0f0 tx=0x7f990000c400 comp rx=0 tx=0).stop 2024-03-11T19:57:23.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7f990e600700 1 -- 172.21.15.97:0/458028605 shutdown_connections 2024-03-11T19:57:23.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7f990e600700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f98ec0a1b50 0x7f98ec0a4010 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7f990e600700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9908106ff0 0x7f990819ed20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7f990e600700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9908101f20 0x7f990819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7f990e600700 1 --2- 172.21.15.97:0/458028605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f99081028d0 0x7f990819a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7f990e600700 1 -- 172.21.15.97:0/458028605 >> 172.21.15.97:0/458028605 conn(0x7f9908078dd0 msgr2=0x7f99080fffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:23.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.291+0000 7f990e600700 1 -- 172.21.15.97:0/458028605 shutdown_connections 2024-03-11T19:57:23.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.291+0000 7f990e600700 1 -- 172.21.15.97:0/458028605 wait complete. 2024-03-11T19:57:23.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.286+0000 7fb2e8e00700 1 -- 172.21.15.97:0/420915372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e4112e40 msgr2=0x7fb2e4115230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:23.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.286+0000 7fb2e8e00700 1 --2- 172.21.15.97:0/420915372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e4112e40 0x7fb2e4115230 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc00b0a0 tx=0x7fb2dc00b3b0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.286+0000 7fb2e8e00700 1 -- 172.21.15.97:0/420915372 shutdown_connections 2024-03-11T19:57:23.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.286+0000 7fb2e8e00700 1 --2- 172.21.15.97:0/420915372 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e4112e40 0x7fb2e4115230 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.286+0000 7fb2e8e00700 1 --2- 172.21.15.97:0/420915372 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e4073ea0 0x7fb2e4074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.286+0000 7fb2e8e00700 1 --2- 172.21.15.97:0/420915372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e40747c0 0x7fb2e4112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.286+0000 7fb2e8e00700 1 -- 172.21.15.97:0/420915372 >> 172.21.15.97:0/420915372 conn(0x7fb2e4078dd0 msgr2=0x7fb2e40791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.287+0000 7fb2e8e00700 1 -- 172.21.15.97:0/420915372 shutdown_connections 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.287+0000 7fb2e8e00700 1 -- 172.21.15.97:0/420915372 wait complete. 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.288+0000 7fb2e8e00700 1 Processor -- start 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.288+0000 7fb2e8e00700 1 -- start start 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.289+0000 7fb2e8e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e4073ea0 0x7fb2e41a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.289+0000 7fb2e8e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e40747c0 0x7fb2e41a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.289+0000 7fb2e8e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e4112e40 0x7fb2e419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.289+0000 7fb2e8e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e419ee90 con 0x7fb2e4073ea0 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.289+0000 7fb2e8e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e419f000 con 0x7fb2e40747c0 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.289+0000 7fb2e8e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e419f1c0 con 0x7fb2e4112e40 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.289+0000 7fb2e1e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e4073ea0 0x7fb2e41a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.289+0000 7fb2e1e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e4073ea0 0x7fb2e41a45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:56312/0 (socket says 172.21.15.97:56312) 2024-03-11T19:57:23.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.289+0000 7fb2e1e00700 1 -- 172.21.15.97:0/1596160890 learned_addr learned my addr 172.21.15.97:0/1596160890 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.289+0000 7fb2e1400700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e40747c0 0x7fb2e41a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7fb2e2800700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e4112e40 0x7fb2e419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7fb2e1400700 1 -- 172.21.15.97:0/1596160890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e4112e40 msgr2=0x7fb2e419e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7fb2e1400700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e4112e40 0x7fb2e419e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7fb2e1400700 1 -- 172.21.15.97:0/1596160890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e4073ea0 msgr2=0x7fb2e41a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7fb2e1400700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e4073ea0 0x7fb2e41a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7fb2e1400700 1 -- 172.21.15.97:0/1596160890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0009520 con 0x7fb2e40747c0 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.290+0000 7fb2e1e00700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e4073ea0 0x7fb2e41a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.291+0000 7fb2e1400700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e40747c0 0x7fb2e41a4b00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000e9c0 tx=0x7fb2d000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.291+0000 7fb2daa00700 1 -- 172.21.15.97:0/1596160890 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d000cb70 con 0x7fb2e40747c0 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.292+0000 7fb2daa00700 1 -- 172.21.15.97:0/1596160890 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2d000ee60 con 0x7fb2e40747c0 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.292+0000 7fb2e8e00700 1 -- 172.21.15.97:0/1596160890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc009d00 con 0x7fb2e40747c0 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.292+0000 7fb2e8e00700 1 -- 172.21.15.97:0/1596160890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e41ab540 con 0x7fb2e40747c0 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.292+0000 7fb2daa00700 1 -- 172.21.15.97:0/1596160890 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0018680 con 0x7fb2e40747c0 2024-03-11T19:57:23.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.293+0000 7fb2e8e00700 1 -- 172.21.15.97:0/1596160890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c4005330 con 0x7fb2e40747c0 2024-03-11T19:57:23.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.295+0000 7fb2daa00700 1 -- 172.21.15.97:0/1596160890 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fb2d000ccd0 con 0x7fb2e40747c0 2024-03-11T19:57:23.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.296+0000 7fb2daa00700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb2c80a1bd0 0x7fb2c80a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:23.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.297+0000 7fb2daa00700 1 -- 172.21.15.97:0/1596160890 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fb2d0014070 con 0x7fb2e40747c0 2024-03-11T19:57:23.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.297+0000 7fb2e1e00700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb2c80a1bd0 0x7fb2c80a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:23.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.297+0000 7fb2e1e00700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb2c80a1bd0 0x7fb2c80a4090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009ea0 tx=0x7fb2d4009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:23.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.298+0000 7fb2daa00700 1 -- 172.21.15.97:0/1596160890 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb2d0096ac0 con 0x7fb2e40747c0 2024-03-11T19:57:23.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.525+0000 7fb2e8e00700 1 -- 172.21.15.97:0/1596160890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fb2c40058b0 con 0x7fb2e40747c0 2024-03-11T19:57:23.693 INFO:teuthology.orchestra.run.smithi097.stdout:343597383714 2024-03-11T19:57:23.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.526+0000 7fb2daa00700 1 -- 172.21.15.97:0/1596160890 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fb2d009a180 con 0x7fb2e40747c0 2024-03-11T19:57:23.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.529+0000 7fb2e8e00700 1 -- 172.21.15.97:0/1596160890 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb2c80a1bd0 msgr2=0x7fb2c80a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:23.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.529+0000 7fb2e8e00700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb2c80a1bd0 0x7fb2c80a4090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009ea0 tx=0x7fb2d4009210 comp rx=0 tx=0).stop 2024-03-11T19:57:23.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.529+0000 7fb2e8e00700 1 -- 172.21.15.97:0/1596160890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e40747c0 msgr2=0x7fb2e41a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:23.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.529+0000 7fb2e8e00700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e40747c0 0x7fb2e41a4b00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000e9c0 tx=0x7fb2d000ecd0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.529+0000 7fb2e8e00700 1 -- 172.21.15.97:0/1596160890 shutdown_connections 2024-03-11T19:57:23.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.529+0000 7fb2e8e00700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb2c80a1bd0 0x7fb2c80a4090 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.529+0000 7fb2e8e00700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb2e4073ea0 0x7fb2e41a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.529+0000 7fb2e8e00700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb2e4112e40 0x7fb2e419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.529+0000 7fb2e8e00700 1 --2- 172.21.15.97:0/1596160890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e40747c0 0x7fb2e41a4b00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:23.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.529+0000 7fb2e8e00700 1 -- 172.21.15.97:0/1596160890 >> 172.21.15.97:0/1596160890 conn(0x7fb2e4078dd0 msgr2=0x7fb2e4111b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:23.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.529+0000 7fb2e8e00700 1 -- 172.21.15.97:0/1596160890 shutdown_connections 2024-03-11T19:57:23.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:23.529+0000 7fb2e8e00700 1 -- 172.21.15.97:0/1596160890 wait complete. 2024-03-11T19:57:23.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:23 smithi097 ceph-mon[31898]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-03-11T19:57:23.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:23 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/458028605' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T19:57:23.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:23 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1596160890' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-11T19:57:23.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:23 smithi114 ceph-mon[36376]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-03-11T19:57:23.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:23 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/458028605' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T19:57:23.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:23 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1596160890' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-11T19:57:24.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:23 smithi161 ceph-mon[36332]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-03-11T19:57:24.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:23 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/458028605' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T19:57:24.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:23 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1596160890' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-11T19:57:25.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:25 smithi097 ceph-mon[31898]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-03-11T19:57:25.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:25 smithi114 ceph-mon[36376]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-03-11T19:57:26.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:25 smithi161 ceph-mon[36332]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-03-11T19:57:26.306 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757310 got 163208757316 for osd.5 2024-03-11T19:57:26.306 DEBUG:teuthology.parallel:result is None 2024-03-11T19:57:27.667 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:27 smithi097 ceph-mon[31898]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:57:27.670 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738459 got 34359738464 for osd.0 2024-03-11T19:57:27.670 DEBUG:teuthology.parallel:result is None 2024-03-11T19:57:27.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.917+0000 7ffb31e00700 1 -- 172.21.15.97:0/2199208837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffb2c074930 msgr2=0x7ffb2c112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:27.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.917+0000 7ffb31e00700 1 --2- 172.21.15.97:0/2199208837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffb2c074930 0x7ffb2c112620 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ffb18009990 tx=0x7ffb18009ca0 comp rx=0 tx=0).stop 2024-03-11T19:57:27.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.917+0000 7ffb31e00700 1 -- 172.21.15.97:0/2199208837 shutdown_connections 2024-03-11T19:57:27.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.917+0000 7ffb31e00700 1 --2- 172.21.15.97:0/2199208837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb2c112b60 0x7ffb2c114f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:27.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.917+0000 7ffb31e00700 1 --2- 172.21.15.97:0/2199208837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb2c074010 0x7ffb2c0743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:27.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.917+0000 7ffb31e00700 1 --2- 172.21.15.97:0/2199208837 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffb2c074930 0x7ffb2c112620 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:27.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.917+0000 7ffb31e00700 1 -- 172.21.15.97:0/2199208837 >> 172.21.15.97:0/2199208837 conn(0x7ffb2c0fd640 msgr2=0x7ffb2c0ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:27.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.917+0000 7ffb31e00700 1 -- 172.21.15.97:0/2199208837 shutdown_connections 2024-03-11T19:57:27.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.917+0000 7ffb31e00700 1 -- 172.21.15.97:0/2199208837 wait complete. 2024-03-11T19:57:27.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.918+0000 7ffb31e00700 1 Processor -- start 2024-03-11T19:57:27.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.918+0000 7ffb31e00700 1 -- start start 2024-03-11T19:57:27.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.919+0000 7ffb31e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb2c074010 0x7ffb2c19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:27.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.919+0000 7ffb31e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb2c074930 0x7ffb2c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:27.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.919+0000 7ffb31e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffb2c112b60 0x7ffb2c1a3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:27.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.919+0000 7ffb31e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c1a3610 con 0x7ffb2c112b60 2024-03-11T19:57:27.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.919+0000 7ffb31e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c1a3780 con 0x7ffb2c074930 2024-03-11T19:57:27.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.919+0000 7ffb31e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c1a3940 con 0x7ffb2c074010 2024-03-11T19:57:27.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.919+0000 7ffb31400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffb2c112b60 0x7ffb2c1a3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:27.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.919+0000 7ffb31400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffb2c112b60 0x7ffb2c1a3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44082/0 (socket says 172.21.15.97:44082) 2024-03-11T19:57:27.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.919+0000 7ffb31400700 1 -- 172.21.15.97:0/2110155768 learned_addr learned my addr 172.21.15.97:0/2110155768 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:27.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.919+0000 7ffb30a00700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb2c074010 0x7ffb2c19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:27.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.919+0000 7ffb2be00700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb2c074930 0x7ffb2c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:27.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.920+0000 7ffb31400700 1 -- 172.21.15.97:0/2110155768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb2c074010 msgr2=0x7ffb2c19e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:27.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.920+0000 7ffb31400700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb2c074010 0x7ffb2c19e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:27.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.920+0000 7ffb31400700 1 -- 172.21.15.97:0/2110155768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb2c074930 msgr2=0x7ffb2c19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:27.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.920+0000 7ffb31400700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb2c074930 0x7ffb2c19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:27.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.920+0000 7ffb31400700 1 -- 172.21.15.97:0/2110155768 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb180095f0 con 0x7ffb2c112b60 2024-03-11T19:57:27.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.920+0000 7ffb2be00700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb2c074930 0x7ffb2c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:57:27.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.920+0000 7ffb31400700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffb2c112b60 0x7ffb2c1a3000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffb2400ef90 tx=0x7ffb2400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:27.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.920+0000 7ffb29600700 1 -- 172.21.15.97:0/2110155768 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb24018070 con 0x7ffb2c112b60 2024-03-11T19:57:27.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.920+0000 7ffb31e00700 1 -- 172.21.15.97:0/2110155768 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb2c1a3bc0 con 0x7ffb2c112b60 2024-03-11T19:57:27.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.920+0000 7ffb29600700 1 -- 172.21.15.97:0/2110155768 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb2400f040 con 0x7ffb2c112b60 2024-03-11T19:57:27.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.920+0000 7ffb31e00700 1 -- 172.21.15.97:0/2110155768 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb2c1ab480 con 0x7ffb2c112b60 2024-03-11T19:57:27.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.921+0000 7ffb29600700 1 -- 172.21.15.97:0/2110155768 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb24004020 con 0x7ffb2c112b60 2024-03-11T19:57:27.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.922+0000 7ffb31e00700 1 -- 172.21.15.97:0/2110155768 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0c005330 con 0x7ffb2c112b60 2024-03-11T19:57:27.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.924+0000 7ffb29600700 1 -- 172.21.15.97:0/2110155768 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7ffb24004ad0 con 0x7ffb2c112b60 2024-03-11T19:57:27.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.926+0000 7ffb29600700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ffb140a1bd0 0x7ffb140a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:27.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.926+0000 7ffb29600700 1 -- 172.21.15.97:0/2110155768 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7ffb240cdd40 con 0x7ffb2c112b60 2024-03-11T19:57:27.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.926+0000 7ffb30a00700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ffb140a1bd0 0x7ffb140a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:27.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.927+0000 7ffb30a00700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ffb140a1bd0 0x7ffb140a4090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffb20009c60 tx=0x7ffb20009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:27.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.932+0000 7ffb29600700 1 -- 172.21.15.97:0/2110155768 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffb24096ae0 con 0x7ffb2c112b60 2024-03-11T19:57:27.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.785+0000 7f0d52400700 1 -- 172.21.15.97:0/3914991204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d4c074930 msgr2=0x7f0d4c112720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:27.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.785+0000 7f0d52400700 1 --2- 172.21.15.97:0/3914991204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d4c074930 0x7f0d4c112720 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0d380098f0 tx=0x7f0d38009c00 comp rx=0 tx=0).stop 2024-03-11T19:57:27.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.785+0000 7f0d52400700 1 -- 172.21.15.97:0/3914991204 shutdown_connections 2024-03-11T19:57:27.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.785+0000 7f0d52400700 1 --2- 172.21.15.97:0/3914991204 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d4c112c60 0x7f0d4c115050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:27.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.785+0000 7f0d52400700 1 --2- 172.21.15.97:0/3914991204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d4c074010 0x7f0d4c0743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:27.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.785+0000 7f0d52400700 1 --2- 172.21.15.97:0/3914991204 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d4c074930 0x7f0d4c112720 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:27.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.785+0000 7f0d52400700 1 -- 172.21.15.97:0/3914991204 >> 172.21.15.97:0/3914991204 conn(0x7f0d4c0fd720 msgr2=0x7f0d4c0ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:27.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.785+0000 7f0d52400700 1 -- 172.21.15.97:0/3914991204 shutdown_connections 2024-03-11T19:57:27.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.785+0000 7f0d52400700 1 -- 172.21.15.97:0/3914991204 wait complete. 2024-03-11T19:57:27.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.786+0000 7f0d52400700 1 Processor -- start 2024-03-11T19:57:27.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.787+0000 7f0d52400700 1 -- start start 2024-03-11T19:57:27.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.787+0000 7f0d52400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d4c074010 0x7f0d4c1a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:27.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.787+0000 7f0d52400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d4c074930 0x7f0d4c1a4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:27.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.787+0000 7f0d52400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d4c112c60 0x7f0d4c19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:27.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.787+0000 7f0d52400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c19ee30 con 0x7f0d4c112c60 2024-03-11T19:57:27.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.787+0000 7f0d52400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c19efa0 con 0x7f0d4c074930 2024-03-11T19:57:27.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.787+0000 7f0d52400700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c19f160 con 0x7f0d4c074010 2024-03-11T19:57:27.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.787+0000 7f0d4be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d4c112c60 0x7f0d4c19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.787+0000 7f0d4be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d4c112c60 0x7f0d4c19e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44058/0 (socket says 172.21.15.97:44058) 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.787+0000 7f0d4be00700 1 -- 172.21.15.97:0/3865263992 learned_addr learned my addr 172.21.15.97:0/3865263992 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.788+0000 7f0d4b400700 1 --2- 172.21.15.97:0/3865263992 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d4c074010 0x7f0d4c1a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.788+0000 7f0d4be00700 1 -- 172.21.15.97:0/3865263992 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d4c074010 msgr2=0x7f0d4c1a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.788+0000 7f0d4be00700 1 --2- 172.21.15.97:0/3865263992 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d4c074010 0x7f0d4c1a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.788+0000 7f0d4be00700 1 -- 172.21.15.97:0/3865263992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d4c074930 msgr2=0x7f0d4c1a4af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.788+0000 7f0d4be00700 1 --2- 172.21.15.97:0/3865263992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d4c074930 0x7f0d4c1a4af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.788+0000 7f0d4be00700 1 -- 172.21.15.97:0/3865263992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c0095f0 con 0x7f0d4c112c60 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.788+0000 7f0d4be00700 1 --2- 172.21.15.97:0/3865263992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d4c112c60 0x7f0d4c19e680 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400ef90 tx=0x7f0d4400c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.788+0000 7f0d43e00700 1 -- 172.21.15.97:0/3865263992 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44018070 con 0x7f0d4c112c60 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.789+0000 7f0d43e00700 1 -- 172.21.15.97:0/3865263992 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d4400f040 con 0x7f0d4c112c60 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.789+0000 7f0d52400700 1 -- 172.21.15.97:0/3865263992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d38009520 con 0x7f0d4c112c60 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.789+0000 7f0d43e00700 1 -- 172.21.15.97:0/3865263992 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44004020 con 0x7f0d4c112c60 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.789+0000 7f0d52400700 1 -- 172.21.15.97:0/3865263992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4c19f530 con 0x7f0d4c112c60 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.790+0000 7f0d52400700 1 -- 172.21.15.97:0/3865263992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d4c105140 con 0x7f0d4c112c60 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.792+0000 7f0d43e00700 1 -- 172.21.15.97:0/3865263992 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f0d44004ad0 con 0x7f0d4c112c60 2024-03-11T19:57:27.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.793+0000 7f0d43e00700 1 --2- 172.21.15.97:0/3865263992 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0d300a1b50 0x7f0d300a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:27.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.794+0000 7f0d43e00700 1 -- 172.21.15.97:0/3865263992 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f0d440cd5b0 con 0x7f0d4c112c60 2024-03-11T19:57:27.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.794+0000 7f0d4b400700 1 --2- 172.21.15.97:0/3865263992 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0d300a1b50 0x7f0d300a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:27.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.795+0000 7f0d4b400700 1 --2- 172.21.15.97:0/3865263992 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0d300a1b50 0x7f0d300a4010 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c005fd0 tx=0x7f0d3c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:27.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:27.803+0000 7f0d43e00700 1 -- 172.21.15.97:0/3865263992 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0d44096350 con 0x7f0d4c112c60 2024-03-11T19:57:27.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:27 smithi114 ceph-mon[36376]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:57:28.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:27 smithi161 ceph-mon[36332]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-11T19:57:28.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.061+0000 7f0d52400700 1 -- 172.21.15.97:0/3865263992 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f0d4c19fc10 con 0x7f0d4c112c60 2024-03-11T19:57:28.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.062+0000 7f0d43e00700 1 -- 172.21.15.97:0/3865263992 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f0d44099a10 con 0x7f0d4c112c60 2024-03-11T19:57:28.061 INFO:teuthology.orchestra.run.smithi097.stdout:309237645351 2024-03-11T19:57:28.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.065+0000 7f0d52400700 1 -- 172.21.15.97:0/3865263992 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0d300a1b50 msgr2=0x7f0d300a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:28.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.065+0000 7f0d52400700 1 --2- 172.21.15.97:0/3865263992 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0d300a1b50 0x7f0d300a4010 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c005fd0 tx=0x7f0d3c009310 comp rx=0 tx=0).stop 2024-03-11T19:57:28.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.065+0000 7f0d52400700 1 -- 172.21.15.97:0/3865263992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d4c112c60 msgr2=0x7f0d4c19e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:28.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.065+0000 7f0d52400700 1 --2- 172.21.15.97:0/3865263992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d4c112c60 0x7f0d4c19e680 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400ef90 tx=0x7f0d4400c450 comp rx=0 tx=0).stop 2024-03-11T19:57:28.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.065+0000 7f0d52400700 1 -- 172.21.15.97:0/3865263992 shutdown_connections 2024-03-11T19:57:28.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.065+0000 7f0d52400700 1 --2- 172.21.15.97:0/3865263992 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0d300a1b50 0x7f0d300a4010 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:28.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.065+0000 7f0d52400700 1 --2- 172.21.15.97:0/3865263992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0d4c112c60 0x7f0d4c19e680 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:28.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.065+0000 7f0d52400700 1 --2- 172.21.15.97:0/3865263992 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d4c074010 0x7f0d4c1a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:28.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.065+0000 7f0d52400700 1 --2- 172.21.15.97:0/3865263992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d4c074930 0x7f0d4c1a4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:28.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.065+0000 7f0d52400700 1 -- 172.21.15.97:0/3865263992 >> 172.21.15.97:0/3865263992 conn(0x7f0d4c0fd720 msgr2=0x7f0d4c0fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:28.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.065+0000 7f0d52400700 1 -- 172.21.15.97:0/3865263992 shutdown_connections 2024-03-11T19:57:28.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.065+0000 7f0d52400700 1 -- 172.21.15.97:0/3865263992 wait complete. 2024-03-11T19:57:28.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.175+0000 7ffb31e00700 1 -- 172.21.15.97:0/2110155768 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ffb0c0058b0 con 0x7ffb2c112b60 2024-03-11T19:57:28.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.176+0000 7ffb29600700 1 -- 172.21.15.97:0/2110155768 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffb24004de0 con 0x7ffb2c112b60 2024-03-11T19:57:28.176 INFO:teuthology.orchestra.run.smithi097.stdout:55834574939 2024-03-11T19:57:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.179+0000 7ffb31e00700 1 -- 172.21.15.97:0/2110155768 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ffb140a1bd0 msgr2=0x7ffb140a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.179+0000 7ffb31e00700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ffb140a1bd0 0x7ffb140a4090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ffb20009c60 tx=0x7ffb20009260 comp rx=0 tx=0).stop 2024-03-11T19:57:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.179+0000 7ffb31e00700 1 -- 172.21.15.97:0/2110155768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffb2c112b60 msgr2=0x7ffb2c1a3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.179+0000 7ffb31e00700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffb2c112b60 0x7ffb2c1a3000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffb2400ef90 tx=0x7ffb2400c3a0 comp rx=0 tx=0).stop 2024-03-11T19:57:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.179+0000 7ffb31e00700 1 -- 172.21.15.97:0/2110155768 shutdown_connections 2024-03-11T19:57:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.179+0000 7ffb31e00700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ffb140a1bd0 0x7ffb140a4090 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.179+0000 7ffb31e00700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffb2c112b60 0x7ffb2c1a3000 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.179+0000 7ffb31e00700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb2c074010 0x7ffb2c19e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.179+0000 7ffb31e00700 1 --2- 172.21.15.97:0/2110155768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffb2c074930 0x7ffb2c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:28.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.179+0000 7ffb31e00700 1 -- 172.21.15.97:0/2110155768 >> 172.21.15.97:0/2110155768 conn(0x7ffb2c0fd640 msgr2=0x7ffb2c0fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.179+0000 7ffb31e00700 1 -- 172.21.15.97:0/2110155768 shutdown_connections 2024-03-11T19:57:28.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:28.179+0000 7ffb31e00700 1 -- 172.21.15.97:0/2110155768 wait complete. 2024-03-11T19:57:28.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:28 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3865263992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-11T19:57:28.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:28 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2110155768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T19:57:28.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:28 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3865263992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-11T19:57:28.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:28 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2110155768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T19:57:29.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:28 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3865263992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-11T19:57:29.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:28 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2110155768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T19:57:29.685 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645347 got 309237645351 for osd.10 2024-03-11T19:57:29.685 DEBUG:teuthology.parallel:result is None 2024-03-11T19:57:29.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:29 smithi097 ceph-mon[31898]: pgmap v373: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:57:29.936 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:29 smithi161 ceph-mon[36332]: pgmap v373: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:57:29.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:29 smithi114 ceph-mon[36376]: pgmap v373: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:57:30.455 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383707 got 343597383714 for osd.11 2024-03-11T19:57:30.455 DEBUG:teuthology.parallel:result is None 2024-03-11T19:57:31.667 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:31 smithi097 ceph-mon[31898]: pgmap v374: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:57:31.831 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574934 got 55834574939 for osd.1 2024-03-11T19:57:31.831 DEBUG:teuthology.parallel:result is None 2024-03-11T19:57:31.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:31 smithi114 ceph-mon[36376]: pgmap v374: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:57:32.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:31 smithi161 ceph-mon[36332]: pgmap v374: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:57:32.935 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411407 got 77309411414 for osd.2 2024-03-11T19:57:32.935 DEBUG:teuthology.parallel:result is None 2024-03-11T19:57:33.854 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:33 smithi097 ceph-mon[31898]: pgmap v375: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:57:33.854 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:33 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/40874926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-11T19:57:33.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:33 smithi114 ceph-mon[36376]: pgmap v375: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:57:33.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:33 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/40874926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-11T19:57:34.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:33 smithi161 ceph-mon[36332]: pgmap v375: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:57:34.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:33 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/40874926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-11T19:57:34.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.803+0000 7fb4f5600700 1 -- 172.21.15.97:0/1758582725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4f0069920 msgr2=0x7fb4f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:34.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.803+0000 7fb4f5600700 1 --2- 172.21.15.97:0/1758582725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4f0069920 0x7fb4f0069d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc0098f0 tx=0x7fb4dc009c00 comp rx=0 tx=0).stop 2024-03-11T19:57:34.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.803+0000 7fb4f5600700 1 -- 172.21.15.97:0/1758582725 shutdown_connections 2024-03-11T19:57:34.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.803+0000 7fb4f5600700 1 --2- 172.21.15.97:0/1758582725 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4f006a2c0 0x7fb4f010eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:34.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.803+0000 7fb4f5600700 1 --2- 172.21.15.97:0/1758582725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4f0107cc0 0x7fb4f01080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:34.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.803+0000 7fb4f5600700 1 --2- 172.21.15.97:0/1758582725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4f0069920 0x7fb4f0069d80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:34.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.803+0000 7fb4f5600700 1 -- 172.21.15.97:0/1758582725 >> 172.21.15.97:0/1758582725 conn(0x7fb4f0077aa0 msgr2=0x7fb4f0077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:34.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.803+0000 7fb4f5600700 1 -- 172.21.15.97:0/1758582725 shutdown_connections 2024-03-11T19:57:34.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.803+0000 7fb4f5600700 1 -- 172.21.15.97:0/1758582725 wait complete. 2024-03-11T19:57:34.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.805+0000 7fb4f5600700 1 Processor -- start 2024-03-11T19:57:34.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.805+0000 7fb4f5600700 1 -- start start 2024-03-11T19:57:34.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.805+0000 7fb4f5600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4f0069920 0x7fb4f0100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:34.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.805+0000 7fb4f5600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4f006a2c0 0x7fb4f0101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:34.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.805+0000 7fb4f5600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4f0107cc0 0x7fb4f01016c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:34.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.805+0000 7fb4f5600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f01a6630 con 0x7fb4f006a2c0 2024-03-11T19:57:34.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.805+0000 7fb4f5600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f01a67a0 con 0x7fb4f0069920 2024-03-11T19:57:34.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.805+0000 7fb4f5600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f01a6960 con 0x7fb4f0107cc0 2024-03-11T19:57:34.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.805+0000 7fb4ee000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4f006a2c0 0x7fb4f0101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:34.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.806+0000 7fb4eea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4f0069920 0x7fb4f0100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:34.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.806+0000 7fb4eea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4f0069920 0x7fb4f0100c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:52540/0 (socket says 172.21.15.97:52540) 2024-03-11T19:57:34.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.806+0000 7fb4eea00700 1 -- 172.21.15.97:0/40874926 learned_addr learned my addr 172.21.15.97:0/40874926 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:34.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.806+0000 7fb4ef400700 1 --2- 172.21.15.97:0/40874926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4f0107cc0 0x7fb4f01016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:34.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.806+0000 7fb4eea00700 1 -- 172.21.15.97:0/40874926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4f0107cc0 msgr2=0x7fb4f01016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:34.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.806+0000 7fb4eea00700 1 --2- 172.21.15.97:0/40874926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4f0107cc0 0x7fb4f01016c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:34.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.806+0000 7fb4eea00700 1 -- 172.21.15.97:0/40874926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4f006a2c0 msgr2=0x7fb4f0101180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:34.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.806+0000 7fb4eea00700 1 --2- 172.21.15.97:0/40874926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4f006a2c0 0x7fb4f0101180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.806+0000 7fb4eea00700 1 -- 172.21.15.97:0/40874926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc009520 con 0x7fb4f0069920 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.806+0000 7fb4eea00700 1 --2- 172.21.15.97:0/40874926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4f0069920 0x7fb4f0100c40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb4e000e9a0 tx=0x7fb4e000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.807+0000 7fb4e7400700 1 -- 172.21.15.97:0/40874926 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0009790 con 0x7fb4f0069920 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.807+0000 7fb4e7400700 1 -- 172.21.15.97:0/40874926 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4e0010430 con 0x7fb4f0069920 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.807+0000 7fb4e7400700 1 -- 172.21.15.97:0/40874926 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0018680 con 0x7fb4f0069920 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.807+0000 7fb4f5600700 1 -- 172.21.15.97:0/40874926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f01a6c40 con 0x7fb4f0069920 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.807+0000 7fb4f5600700 1 -- 172.21.15.97:0/40874926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f01a7110 con 0x7fb4f0069920 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.808+0000 7fb4f5600700 1 -- 172.21.15.97:0/40874926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f010a1e0 con 0x7fb4f0069920 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.810+0000 7fb4e7400700 1 -- 172.21.15.97:0/40874926 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fb4e0010910 con 0x7fb4f0069920 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.811+0000 7fb4e7400700 1 --2- 172.21.15.97:0/40874926 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb4d40a1b50 0x7fb4d40a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.811+0000 7fb4e7400700 1 -- 172.21.15.97:0/40874926 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fb4e0014070 con 0x7fb4f0069920 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.811+0000 7fb4ee000700 1 --2- 172.21.15.97:0/40874926 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb4d40a1b50 0x7fb4d40a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.812+0000 7fb4ee000700 1 --2- 172.21.15.97:0/40874926 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb4d40a1b50 0x7fb4d40a4010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009810 tx=0x7fb4dc019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:32.813+0000 7fb4e7400700 1 -- 172.21.15.97:0/40874926 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb4e00967b0 con 0x7fb4f0069920 2024-03-11T19:57:34.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.042+0000 7fb4f5600700 1 -- 172.21.15.97:0/40874926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fb4f0111d00 con 0x7fb4f0069920 2024-03-11T19:57:34.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.043+0000 7fb4e7400700 1 -- 172.21.15.97:0/40874926 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb4e00105a0 con 0x7fb4f0069920 2024-03-11T19:57:34.296 INFO:teuthology.orchestra.run.smithi097.stdout:287762808878 2024-03-11T19:57:34.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.046+0000 7fb4f5600700 1 -- 172.21.15.97:0/40874926 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb4d40a1b50 msgr2=0x7fb4d40a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:34.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.046+0000 7fb4f5600700 1 --2- 172.21.15.97:0/40874926 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb4d40a1b50 0x7fb4d40a4010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009810 tx=0x7fb4dc019040 comp rx=0 tx=0).stop 2024-03-11T19:57:34.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.046+0000 7fb4f5600700 1 -- 172.21.15.97:0/40874926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4f0069920 msgr2=0x7fb4f0100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:34.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.046+0000 7fb4f5600700 1 --2- 172.21.15.97:0/40874926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4f0069920 0x7fb4f0100c40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb4e000e9a0 tx=0x7fb4e000ecb0 comp rx=0 tx=0).stop 2024-03-11T19:57:34.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.046+0000 7fb4f5600700 1 -- 172.21.15.97:0/40874926 shutdown_connections 2024-03-11T19:57:34.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.046+0000 7fb4f5600700 1 --2- 172.21.15.97:0/40874926 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb4d40a1b50 0x7fb4d40a4010 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:34.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.046+0000 7fb4f5600700 1 --2- 172.21.15.97:0/40874926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb4f0107cc0 0x7fb4f01016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:34.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.046+0000 7fb4f5600700 1 --2- 172.21.15.97:0/40874926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4f0069920 0x7fb4f0100c40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:34.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.046+0000 7fb4f5600700 1 --2- 172.21.15.97:0/40874926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb4f006a2c0 0x7fb4f0101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:34.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.046+0000 7fb4f5600700 1 -- 172.21.15.97:0/40874926 >> 172.21.15.97:0/40874926 conn(0x7fb4f0077aa0 msgr2=0x7fb4f00ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:34.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.046+0000 7fb4f5600700 1 -- 172.21.15.97:0/40874926 shutdown_connections 2024-03-11T19:57:34.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:33.046+0000 7fb4f5600700 1 -- 172.21.15.97:0/40874926 wait complete. 2024-03-11T19:57:35.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.951+0000 7f0ded600700 1 -- 172.21.15.97:0/4029135192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0de810a510 msgr2=0x7f0de8110ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.951+0000 7f0ded600700 1 --2- 172.21.15.97:0/4029135192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0de810a510 0x7f0de8110ee0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0de000b0a0 tx=0x7f0de000b3b0 comp rx=0 tx=0).stop 2024-03-11T19:57:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.952+0000 7f0ded600700 1 -- 172.21.15.97:0/4029135192 shutdown_connections 2024-03-11T19:57:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.952+0000 7f0ded600700 1 --2- 172.21.15.97:0/4029135192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0de810a510 0x7f0de8110ee0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.952+0000 7f0ded600700 1 --2- 172.21.15.97:0/4029135192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0de81008d0 0x7f0de8100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.952+0000 7f0ded600700 1 --2- 172.21.15.97:0/4029135192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0de8101280 0x7f0de8109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.952+0000 7f0ded600700 1 -- 172.21.15.97:0/4029135192 >> 172.21.15.97:0/4029135192 conn(0x7f0de8075ec0 msgr2=0x7f0de80762d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.952+0000 7f0ded600700 1 -- 172.21.15.97:0/4029135192 shutdown_connections 2024-03-11T19:57:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.952+0000 7f0ded600700 1 -- 172.21.15.97:0/4029135192 wait complete. 2024-03-11T19:57:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.953+0000 7f0ded600700 1 Processor -- start 2024-03-11T19:57:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.953+0000 7f0ded600700 1 -- start start 2024-03-11T19:57:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.954+0000 7f0ded600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0de81008d0 0x7f0de819a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:35.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.954+0000 7f0ded600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0de8101280 0x7f0de819aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.954+0000 7f0ded600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0de810a510 0x7f0de819ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.954+0000 7f0ded600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de819f440 con 0x7f0de81008d0 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.954+0000 7f0ded600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de819f5b0 con 0x7f0de8101280 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.954+0000 7f0ded600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de819f770 con 0x7f0de810a510 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.954+0000 7f0de6a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0de81008d0 0x7f0de819a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.954+0000 7f0de6a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0de81008d0 0x7f0de819a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54720/0 (socket says 172.21.15.97:54720) 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.954+0000 7f0de6a00700 1 -- 172.21.15.97:0/3546979635 learned_addr learned my addr 172.21.15.97:0/3546979635 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.954+0000 7f0de7400700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0de810a510 0x7f0de819ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.954+0000 7f0de6000700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0de8101280 0x7f0de819aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.955+0000 7f0de6a00700 1 -- 172.21.15.97:0/3546979635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0de810a510 msgr2=0x7f0de819ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.955+0000 7f0de6a00700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0de810a510 0x7f0de819ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.955+0000 7f0de6a00700 1 -- 172.21.15.97:0/3546979635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0de8101280 msgr2=0x7f0de819aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.955+0000 7f0de6a00700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0de8101280 0x7f0de819aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:35.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.955+0000 7f0de6a00700 1 -- 172.21.15.97:0/3546979635 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd8009520 con 0x7f0de81008d0 2024-03-11T19:57:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.955+0000 7f0de6a00700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0de81008d0 0x7f0de819a560 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800eb30 tx=0x7f0dd800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.955+0000 7f0ddf400700 1 -- 172.21.15.97:0/3546979635 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8009880 con 0x7f0de81008d0 2024-03-11T19:57:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.955+0000 7f0ddf400700 1 -- 172.21.15.97:0/3546979635 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0dd8004500 con 0x7f0de81008d0 2024-03-11T19:57:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.955+0000 7f0ded600700 1 -- 172.21.15.97:0/3546979635 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de0009d00 con 0x7f0de81008d0 2024-03-11T19:57:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.955+0000 7f0ddf400700 1 -- 172.21.15.97:0/3546979635 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8003e10 con 0x7f0de81008d0 2024-03-11T19:57:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.955+0000 7f0ded600700 1 -- 172.21.15.97:0/3546979635 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de81a70e0 con 0x7f0de81008d0 2024-03-11T19:57:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.956+0000 7f0ded600700 1 -- 172.21.15.97:0/3546979635 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de8068930 con 0x7f0de81008d0 2024-03-11T19:57:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.958+0000 7f0ddf400700 1 -- 172.21.15.97:0/3546979635 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f0dd800cbd0 con 0x7f0de81008d0 2024-03-11T19:57:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.960+0000 7f0ddf400700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0dcc0a1b50 0x7f0dcc0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.960+0000 7f0ddf400700 1 -- 172.21.15.97:0/3546979635 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f0dd8014070 con 0x7f0de81008d0 2024-03-11T19:57:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.960+0000 7f0de6000700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0dcc0a1b50 0x7f0dcc0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.961+0000 7f0de6000700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0dcc0a1b50 0x7f0dcc0a4010 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0de8102db0 tx=0x7f0dd4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:35.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:34.964+0000 7f0ddf400700 1 -- 172.21.15.97:0/3546979635 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0dd8096a30 con 0x7f0de81008d0 2024-03-11T19:57:35.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.185+0000 7f0ded600700 1 -- 172.21.15.97:0/3546979635 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f0de819b8c0 con 0x7f0de81008d0 2024-03-11T19:57:35.186 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.186+0000 7f0ddf400700 1 -- 172.21.15.97:0/3546979635 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0dd809a0f0 con 0x7f0de81008d0 2024-03-11T19:57:35.186 INFO:teuthology.orchestra.run.smithi097.stdout:253403070519 2024-03-11T19:57:35.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.189+0000 7f0ded600700 1 -- 172.21.15.97:0/3546979635 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0dcc0a1b50 msgr2=0x7f0dcc0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:35.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.189+0000 7f0ded600700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0dcc0a1b50 0x7f0dcc0a4010 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0de8102db0 tx=0x7f0dd4009260 comp rx=0 tx=0).stop 2024-03-11T19:57:35.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.189+0000 7f0ded600700 1 -- 172.21.15.97:0/3546979635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0de81008d0 msgr2=0x7f0de819a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:35.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.189+0000 7f0ded600700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0de81008d0 0x7f0de819a560 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800eb30 tx=0x7f0dd800ee40 comp rx=0 tx=0).stop 2024-03-11T19:57:35.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.189+0000 7f0ded600700 1 -- 172.21.15.97:0/3546979635 shutdown_connections 2024-03-11T19:57:35.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.189+0000 7f0ded600700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0dcc0a1b50 0x7f0dcc0a4010 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:35.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.189+0000 7f0ded600700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0de81008d0 0x7f0de819a560 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:35.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.189+0000 7f0ded600700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0de810a510 0x7f0de819ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:35.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.189+0000 7f0ded600700 1 --2- 172.21.15.97:0/3546979635 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0de8101280 0x7f0de819aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:35.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.189+0000 7f0ded600700 1 -- 172.21.15.97:0/3546979635 >> 172.21.15.97:0/3546979635 conn(0x7f0de8075ec0 msgr2=0x7f0de8107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:35.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.189+0000 7f0ded600700 1 -- 172.21.15.97:0/3546979635 shutdown_connections 2024-03-11T19:57:35.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:35.189+0000 7f0ded600700 1 -- 172.21.15.97:0/3546979635 wait complete. 2024-03-11T19:57:35.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:35 smithi097 ceph-mon[31898]: pgmap v376: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:57:35.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:35 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3546979635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-11T19:57:35.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:35 smithi114 ceph-mon[36376]: pgmap v376: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:57:35.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:35 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3546979635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-11T19:57:36.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:35 smithi161 ceph-mon[36332]: pgmap v376: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:57:36.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:35 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3546979635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-11T19:57:36.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.104+0000 7f1f17600700 1 -- 172.21.15.97:0/2984386101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f101040c0 msgr2=0x7f1f101044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:36.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.104+0000 7f1f17600700 1 --2- 172.21.15.97:0/2984386101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f101040c0 0x7f1f101044a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8009990 tx=0x7f1ef8009ca0 comp rx=0 tx=0).stop 2024-03-11T19:57:36.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.104+0000 7f1f17600700 1 -- 172.21.15.97:0/2984386101 shutdown_connections 2024-03-11T19:57:36.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.104+0000 7f1f17600700 1 --2- 172.21.15.97:0/2984386101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f101091a0 0x7f1f10110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:36.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.104+0000 7f1f17600700 1 --2- 172.21.15.97:0/2984386101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f101040c0 0x7f1f101044a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:36.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.104+0000 7f1f17600700 1 --2- 172.21.15.97:0/2984386101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f10104a70 0x7f1f10108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:36.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.104+0000 7f1f17600700 1 -- 172.21.15.97:0/2984386101 >> 172.21.15.97:0/2984386101 conn(0x7f1f100ff950 msgr2=0x7f1f10101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:36.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.104+0000 7f1f17600700 1 -- 172.21.15.97:0/2984386101 shutdown_connections 2024-03-11T19:57:36.105 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.104+0000 7f1f17600700 1 -- 172.21.15.97:0/2984386101 wait complete. 2024-03-11T19:57:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.105+0000 7f1f17600700 1 Processor -- start 2024-03-11T19:57:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.105+0000 7f1f17600700 1 -- start start 2024-03-11T19:57:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.106+0000 7f1f17600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f101040c0 0x7f1f1019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.106+0000 7f1f17600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f10104a70 0x7f1f1019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.106+0000 7f1f17600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f101091a0 0x7f1f1019ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.106+0000 7f1f17600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1019f2a0 con 0x7f1f101040c0 2024-03-11T19:57:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.106+0000 7f1f17600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1019f410 con 0x7f1f10104a70 2024-03-11T19:57:36.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.106+0000 7f1f17600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1019f5d0 con 0x7f1f101091a0 2024-03-11T19:57:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.106+0000 7f1f14c00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f101040c0 0x7f1f1019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.106+0000 7f1f14c00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f101040c0 0x7f1f1019a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54740/0 (socket says 172.21.15.97:54740) 2024-03-11T19:57:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.106+0000 7f1f14c00700 1 -- 172.21.15.97:0/2959367092 learned_addr learned my addr 172.21.15.97:0/2959367092 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.106+0000 7f1f15600700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f101091a0 0x7f1f1019ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.106+0000 7f1f0fe00700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f10104a70 0x7f1f1019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.106+0000 7f1f15600700 1 -- 172.21.15.97:0/2959367092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f10104a70 msgr2=0x7f1f1019a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.107+0000 7f1f15600700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f10104a70 0x7f1f1019a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.107+0000 7f1f15600700 1 -- 172.21.15.97:0/2959367092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f101040c0 msgr2=0x7f1f1019a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.107+0000 7f1f15600700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f101040c0 0x7f1f1019a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.107+0000 7f1f15600700 1 -- 172.21.15.97:0/2959367092 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f08009c40 con 0x7f1f101091a0 2024-03-11T19:57:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.107+0000 7f1f14c00700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f101040c0 0x7f1f1019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.107+0000 7f1f0fe00700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f10104a70 0x7f1f1019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:36.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.107+0000 7f1f15600700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f101091a0 0x7f1f1019ec90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1f0800efb0 tx=0x7f1f0800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:36.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.107+0000 7f1f0d600700 1 -- 172.21.15.97:0/2959367092 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f08003a00 con 0x7f1f101091a0 2024-03-11T19:57:36.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.107+0000 7f1f17600700 1 -- 172.21.15.97:0/2959367092 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef80095f0 con 0x7f1f101091a0 2024-03-11T19:57:36.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.107+0000 7f1f0d600700 1 -- 172.21.15.97:0/2959367092 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f0800f040 con 0x7f1f101091a0 2024-03-11T19:57:36.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.107+0000 7f1f0d600700 1 -- 172.21.15.97:0/2959367092 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f080046a0 con 0x7f1f101091a0 2024-03-11T19:57:36.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.107+0000 7f1f17600700 1 -- 172.21.15.97:0/2959367092 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f101a6f40 con 0x7f1f101091a0 2024-03-11T19:57:36.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.108+0000 7f1f17600700 1 -- 172.21.15.97:0/2959367092 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1010c460 con 0x7f1f101091a0 2024-03-11T19:57:36.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.111+0000 7f1f0d600700 1 -- 172.21.15.97:0/2959367092 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1f08003cf0 con 0x7f1f101091a0 2024-03-11T19:57:36.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.112+0000 7f1f0d600700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1ef40a1b50 0x7f1ef40a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:36.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.112+0000 7f1f0d600700 1 -- 172.21.15.97:0/2959367092 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f1f080cd3c0 con 0x7f1f101091a0 2024-03-11T19:57:36.112 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.112+0000 7f1f14c00700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1ef40a1b50 0x7f1ef40a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:36.113 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.113+0000 7f1f14c00700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1ef40a1b50 0x7f1ef40a4010 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8006010 tx=0x7f1ef800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:36.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.114+0000 7f1f0d600700 1 -- 172.21.15.97:0/2959367092 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1f080961c0 con 0x7f1f101091a0 2024-03-11T19:57:36.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.336+0000 7f1f17600700 1 -- 172.21.15.97:0/2959367092 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f1f10113f40 con 0x7f1f101091a0 2024-03-11T19:57:36.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.337+0000 7f1f0d600700 1 -- 172.21.15.97:0/2959367092 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1f08099880 con 0x7f1f101091a0 2024-03-11T19:57:36.338 INFO:teuthology.orchestra.run.smithi097.stdout:141733920844 2024-03-11T19:57:36.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.340+0000 7f1f17600700 1 -- 172.21.15.97:0/2959367092 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1ef40a1b50 msgr2=0x7f1ef40a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:36.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.340+0000 7f1f17600700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1ef40a1b50 0x7f1ef40a4010 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8006010 tx=0x7f1ef800b260 comp rx=0 tx=0).stop 2024-03-11T19:57:36.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.340+0000 7f1f17600700 1 -- 172.21.15.97:0/2959367092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f101091a0 msgr2=0x7f1f1019ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:36.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.340+0000 7f1f17600700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f101091a0 0x7f1f1019ec90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1f0800efb0 tx=0x7f1f0800c3a0 comp rx=0 tx=0).stop 2024-03-11T19:57:36.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.340+0000 7f1f17600700 1 -- 172.21.15.97:0/2959367092 shutdown_connections 2024-03-11T19:57:36.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.340+0000 7f1f17600700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f1ef40a1b50 0x7f1ef40a4010 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:36.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.340+0000 7f1f17600700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1f101040c0 0x7f1f1019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:36.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.340+0000 7f1f17600700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f101091a0 0x7f1f1019ec90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:36.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.341+0000 7f1f17600700 1 --2- 172.21.15.97:0/2959367092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f10104a70 0x7f1f1019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:36.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.341+0000 7f1f17600700 1 -- 172.21.15.97:0/2959367092 >> 172.21.15.97:0/2959367092 conn(0x7f1f100ff950 msgr2=0x7f1f10100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:36.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.341+0000 7f1f17600700 1 -- 172.21.15.97:0/2959367092 shutdown_connections 2024-03-11T19:57:36.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:36.341+0000 7f1f17600700 1 -- 172.21.15.97:0/2959367092 wait complete. 2024-03-11T19:57:36.497 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808873 got 287762808878 for osd.9 2024-03-11T19:57:36.498 DEBUG:teuthology.parallel:result is None 2024-03-11T19:57:36.831 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:36 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2959367092' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T19:57:36.884 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:36.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:36 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2959367092' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T19:57:37.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:36 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2959367092' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T19:57:37.312 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920838 got 141733920844 for osd.4 2024-03-11T19:57:37.312 DEBUG:teuthology.parallel:result is None 2024-03-11T19:57:37.704 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070513 got 253403070519 for osd.8 2024-03-11T19:57:37.704 DEBUG:teuthology.parallel:result is None 2024-03-11T19:57:37.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:37 smithi097 ceph-mon[31898]: pgmap v377: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:57:37.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:37 smithi114 ceph-mon[36376]: pgmap v377: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:57:38.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:37 smithi161 ceph-mon[36332]: pgmap v377: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:57:39.286 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:39.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.560+0000 7f1623800700 1 -- 172.21.15.97:0/2833176465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f161c112e40 msgr2=0x7f161c115230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:39.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.560+0000 7f1623800700 1 --2- 172.21.15.97:0/2833176465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f161c112e40 0x7f161c115230 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f161400b0a0 tx=0x7f161400b3b0 comp rx=0 tx=0).stop 2024-03-11T19:57:39.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.560+0000 7f1623800700 1 -- 172.21.15.97:0/2833176465 shutdown_connections 2024-03-11T19:57:39.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.560+0000 7f1623800700 1 --2- 172.21.15.97:0/2833176465 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f161c112e40 0x7f161c115230 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:39.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.560+0000 7f1623800700 1 --2- 172.21.15.97:0/2833176465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f161c073ea0 0x7f161c074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:39.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.560+0000 7f1623800700 1 --2- 172.21.15.97:0/2833176465 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f161c0747c0 0x7f161c112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:39.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.560+0000 7f1623800700 1 -- 172.21.15.97:0/2833176465 >> 172.21.15.97:0/2833176465 conn(0x7f161c078dd0 msgr2=0x7f161c0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:39.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.560+0000 7f1623800700 1 -- 172.21.15.97:0/2833176465 shutdown_connections 2024-03-11T19:57:39.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.560+0000 7f1623800700 1 -- 172.21.15.97:0/2833176465 wait complete. 2024-03-11T19:57:39.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.562+0000 7f1623800700 1 Processor -- start 2024-03-11T19:57:39.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.562+0000 7f1623800700 1 -- start start 2024-03-11T19:57:39.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.563+0000 7f1623800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f161c073ea0 0x7f161c1a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:39.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.563+0000 7f1623800700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f161c0747c0 0x7f161c1a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:39.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.563+0000 7f1623800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f161c112e40 0x7f161c19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:39.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.563+0000 7f1623800700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c19eea0 con 0x7f161c073ea0 2024-03-11T19:57:39.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.563+0000 7f1623800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c19f010 con 0x7f161c112e40 2024-03-11T19:57:39.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.563+0000 7f1623800700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c19f1d0 con 0x7f161c0747c0 2024-03-11T19:57:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.563+0000 7f1620e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f161c073ea0 0x7f161c1a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.563+0000 7f161be00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f161c0747c0 0x7f161c1a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.563+0000 7f1620e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f161c073ea0 0x7f161c1a4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54768/0 (socket says 172.21.15.97:54768) 2024-03-11T19:57:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.563+0000 7f1620e00700 1 -- 172.21.15.97:0/3781554710 learned_addr learned my addr 172.21.15.97:0/3781554710 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.563+0000 7f1621800700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f161c112e40 0x7f161c19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.564+0000 7f1620e00700 1 -- 172.21.15.97:0/3781554710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f161c0747c0 msgr2=0x7f161c1a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.564+0000 7f1620e00700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f161c0747c0 0x7f161c1a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.564+0000 7f1620e00700 1 -- 172.21.15.97:0/3781554710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f161c112e40 msgr2=0x7f161c19e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.564+0000 7f1620e00700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f161c112e40 0x7f161c19e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.564+0000 7f1620e00700 1 -- 172.21.15.97:0/3781554710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1614009d00 con 0x7f161c073ea0 2024-03-11T19:57:39.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.564+0000 7f1621800700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f161c112e40 0x7f161c19e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:39.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.564+0000 7f1620e00700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f161c073ea0 0x7f161c1a4620 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f161000e9a0 tx=0x7f161000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:39.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.565+0000 7f1619600700 1 -- 172.21.15.97:0/3781554710 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1610009790 con 0x7f161c073ea0 2024-03-11T19:57:39.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.565+0000 7f1619600700 1 -- 172.21.15.97:0/3781554710 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1610010430 con 0x7f161c073ea0 2024-03-11T19:57:39.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.565+0000 7f1623800700 1 -- 172.21.15.97:0/3781554710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161c19f4b0 con 0x7f161c073ea0 2024-03-11T19:57:39.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.565+0000 7f1619600700 1 -- 172.21.15.97:0/3781554710 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1610018680 con 0x7f161c073ea0 2024-03-11T19:57:39.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.565+0000 7f1623800700 1 -- 172.21.15.97:0/3781554710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f161c1ab5f0 con 0x7f161c073ea0 2024-03-11T19:57:39.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.566+0000 7f1623800700 1 -- 172.21.15.97:0/3781554710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f161c0755d0 con 0x7f161c073ea0 2024-03-11T19:57:39.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.568+0000 7f1619600700 1 -- 172.21.15.97:0/3781554710 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1610010910 con 0x7f161c073ea0 2024-03-11T19:57:39.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.570+0000 7f1619600700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f16040a1bd0 0x7f16040a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:39.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.571+0000 7f1619600700 1 -- 172.21.15.97:0/3781554710 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f1610014070 con 0x7f161c073ea0 2024-03-11T19:57:39.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.571+0000 7f161be00700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f16040a1bd0 0x7f16040a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:39.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.571+0000 7f161be00700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f16040a1bd0 0x7f16040a4090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f161c19feb0 tx=0x7f1608009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:39.574 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.574+0000 7f1619600700 1 -- 172.21.15.97:0/3781554710 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1610052d20 con 0x7f161c073ea0 2024-03-11T19:57:39.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:39 smithi097 ceph-mon[31898]: pgmap v378: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:57:39.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:57:39.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:39 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:57:39.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.800+0000 7f1623800700 1 -- 172.21.15.97:0/3781554710 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f161c118390 con 0x7f161c073ea0 2024-03-11T19:57:39.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.801+0000 7f1619600700 1 -- 172.21.15.97:0/3781554710 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f16100967d0 con 0x7f161c073ea0 2024-03-11T19:57:39.801 INFO:teuthology.orchestra.run.smithi097.stdout:219043332157 2024-03-11T19:57:39.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.803+0000 7f1623800700 1 -- 172.21.15.97:0/3781554710 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f16040a1bd0 msgr2=0x7f16040a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:39.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.803+0000 7f1623800700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f16040a1bd0 0x7f16040a4090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f161c19feb0 tx=0x7f1608009260 comp rx=0 tx=0).stop 2024-03-11T19:57:39.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.803+0000 7f1623800700 1 -- 172.21.15.97:0/3781554710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f161c073ea0 msgr2=0x7f161c1a4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:39.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.803+0000 7f1623800700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f161c073ea0 0x7f161c1a4620 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f161000e9a0 tx=0x7f161000ecb0 comp rx=0 tx=0).stop 2024-03-11T19:57:39.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.804+0000 7f1623800700 1 -- 172.21.15.97:0/3781554710 shutdown_connections 2024-03-11T19:57:39.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.804+0000 7f1623800700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f16040a1bd0 0x7f16040a4090 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:39.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.804+0000 7f1623800700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f161c073ea0 0x7f161c1a4620 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:39.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.804+0000 7f1623800700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f161c112e40 0x7f161c19e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:39.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.804+0000 7f1623800700 1 --2- 172.21.15.97:0/3781554710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f161c0747c0 0x7f161c1a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:39.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.804+0000 7f1623800700 1 -- 172.21.15.97:0/3781554710 >> 172.21.15.97:0/3781554710 conn(0x7f161c078dd0 msgr2=0x7f161c111b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:39.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.804+0000 7f1623800700 1 -- 172.21.15.97:0/3781554710 shutdown_connections 2024-03-11T19:57:39.805 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:39.804+0000 7f1623800700 1 -- 172.21.15.97:0/3781554710 wait complete. 2024-03-11T19:57:39.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:39 smithi114 ceph-mon[36376]: pgmap v378: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:57:39.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:57:39.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:39 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:57:40.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:39 smithi161 ceph-mon[36332]: pgmap v378: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:57:40.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:57:40.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:39 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:57:40.668 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332149 got 219043332157 for osd.7 2024-03-11T19:57:40.668 DEBUG:teuthology.parallel:result is None 2024-03-11T19:57:40.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:40 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3781554710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T19:57:40.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:40 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:57:40.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:40 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:57:40.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:40 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:57:40.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:40 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3781554710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T19:57:40.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:40 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:57:40.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:40 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:57:40.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:40 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:57:41.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:40 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3781554710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T19:57:41.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:40 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:57:41.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:40 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:57:41.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:40 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:57:41.667 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:41 smithi097 ceph-mon[31898]: pgmap v379: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:57:41.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:41 smithi114 ceph-mon[36376]: pgmap v379: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:57:42.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:41 smithi161 ceph-mon[36332]: pgmap v379: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:57:42.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.878+0000 7f2efd200700 1 -- 172.21.15.97:0/1731853884 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ef8101f20 msgr2=0x7f2ef8102300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:42.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.878+0000 7f2efd200700 1 --2- 172.21.15.97:0/1731853884 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ef8101f20 0x7f2ef8102300 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009990 tx=0x7f2ee8009ca0 comp rx=0 tx=0).stop 2024-03-11T19:57:42.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.878+0000 7f2efd200700 1 -- 172.21.15.97:0/1731853884 shutdown_connections 2024-03-11T19:57:42.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.878+0000 7f2efd200700 1 --2- 172.21.15.97:0/1731853884 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ef8101f20 0x7f2ef8102300 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:42.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.878+0000 7f2efd200700 1 --2- 172.21.15.97:0/1731853884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ef8106ff0 0x7f2ef810ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:42.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.878+0000 7f2efd200700 1 --2- 172.21.15.97:0/1731853884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef81028d0 0x7f2ef8106910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:42.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.878+0000 7f2efd200700 1 -- 172.21.15.97:0/1731853884 >> 172.21.15.97:0/1731853884 conn(0x7f2ef8078dd0 msgr2=0x7f2ef80791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:42.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.878+0000 7f2efd200700 1 -- 172.21.15.97:0/1731853884 shutdown_connections 2024-03-11T19:57:42.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.878+0000 7f2efd200700 1 -- 172.21.15.97:0/1731853884 wait complete. 2024-03-11T19:57:42.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.879+0000 7f2efd200700 1 Processor -- start 2024-03-11T19:57:42.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.880+0000 7f2efd200700 1 -- start start 2024-03-11T19:57:42.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.880+0000 7f2efd200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ef8101f20 0x7f2ef8198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:42.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.880+0000 7f2efd200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef81028d0 0x7f2ef8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:42.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.880+0000 7f2efd200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ef8106ff0 0x7f2ef819cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:42.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.880+0000 7f2efd200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef819d0e0 con 0x7f2ef8101f20 2024-03-11T19:57:42.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.880+0000 7f2efd200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef819d250 con 0x7f2ef8106ff0 2024-03-11T19:57:42.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.880+0000 7f2efd200700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef819d410 con 0x7f2ef81028d0 2024-03-11T19:57:42.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.880+0000 7f2ef5e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ef8101f20 0x7f2ef8198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:42.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.880+0000 7f2ef5e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ef8101f20 0x7f2ef8198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54778/0 (socket says 172.21.15.97:54778) 2024-03-11T19:57:42.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.880+0000 7f2ef5e00700 1 -- 172.21.15.97:0/1509425799 learned_addr learned my addr 172.21.15.97:0/1509425799 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:42.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.880+0000 7f2ef6800700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ef8106ff0 0x7f2ef819cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:42.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.880+0000 7f2ef5400700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef81028d0 0x7f2ef8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:42.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.881+0000 7f2ef5e00700 1 -- 172.21.15.97:0/1509425799 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef81028d0 msgr2=0x7f2ef8198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:42.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.881+0000 7f2ef5e00700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef81028d0 0x7f2ef8198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:42.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.881+0000 7f2ef5e00700 1 -- 172.21.15.97:0/1509425799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ef8106ff0 msgr2=0x7f2ef819cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:42.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.881+0000 7f2ef5e00700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ef8106ff0 0x7f2ef819cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:42.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.881+0000 7f2ef5e00700 1 -- 172.21.15.97:0/1509425799 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee80095f0 con 0x7f2ef8101f20 2024-03-11T19:57:42.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.881+0000 7f2ef5400700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef81028d0 0x7f2ef8198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:42.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.881+0000 7f2ef6800700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ef8106ff0 0x7f2ef819cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:42.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.881+0000 7f2ef5e00700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ef8101f20 0x7f2ef8198200 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8005770 tx=0x7f2ee80057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:42.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.881+0000 7f2eeea00700 1 -- 172.21.15.97:0/1509425799 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee801d070 con 0x7f2ef8101f20 2024-03-11T19:57:42.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.882+0000 7f2efd200700 1 -- 172.21.15.97:0/1509425799 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef819d690 con 0x7f2ef8101f20 2024-03-11T19:57:42.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.882+0000 7f2eeea00700 1 -- 172.21.15.97:0/1509425799 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ee800b610 con 0x7f2ef8101f20 2024-03-11T19:57:42.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.882+0000 7f2efd200700 1 -- 172.21.15.97:0/1509425799 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef81a4f50 con 0x7f2ef8101f20 2024-03-11T19:57:42.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.882+0000 7f2eeea00700 1 -- 172.21.15.97:0/1509425799 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8021dd0 con 0x7f2ef8101f20 2024-03-11T19:57:42.884 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.883+0000 7f2efd200700 1 -- 172.21.15.97:0/1509425799 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef810a220 con 0x7f2ef8101f20 2024-03-11T19:57:42.886 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.886+0000 7f2eeea00700 1 -- 172.21.15.97:0/1509425799 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f2ee8021510 con 0x7f2ef8101f20 2024-03-11T19:57:42.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.887+0000 7f2eeea00700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2edc0a1b50 0x7f2edc0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:42.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.888+0000 7f2eeea00700 1 -- 172.21.15.97:0/1509425799 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f2ee80cea20 con 0x7f2ef8101f20 2024-03-11T19:57:42.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.888+0000 7f2ef5400700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2edc0a1b50 0x7f2edc0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:42.888 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.888+0000 7f2ef5400700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2edc0a1b50 0x7f2edc0a4010 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2ef81018e0 tx=0x7f2ee4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:42.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:42.890+0000 7f2eeea00700 1 -- 172.21.15.97:0/1509425799 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2ee8097870 con 0x7f2ef8101f20 2024-03-11T19:57:43.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.118+0000 7f2efd200700 1 -- 172.21.15.97:0/1509425799 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2ef8199560 con 0x7f2ef8101f20 2024-03-11T19:57:43.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.118+0000 7f2eeea00700 1 -- 172.21.15.97:0/1509425799 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2ee809af30 con 0x7f2ef8101f20 2024-03-11T19:57:43.119 INFO:teuthology.orchestra.run.smithi097.stdout:107374182483 2024-03-11T19:57:43.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.121+0000 7f2efd200700 1 -- 172.21.15.97:0/1509425799 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2edc0a1b50 msgr2=0x7f2edc0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:43.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.121+0000 7f2efd200700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2edc0a1b50 0x7f2edc0a4010 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2ef81018e0 tx=0x7f2ee4009260 comp rx=0 tx=0).stop 2024-03-11T19:57:43.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.121+0000 7f2efd200700 1 -- 172.21.15.97:0/1509425799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ef8101f20 msgr2=0x7f2ef8198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:43.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.121+0000 7f2efd200700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ef8101f20 0x7f2ef8198200 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8005770 tx=0x7f2ee80057a0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.122+0000 7f2efd200700 1 -- 172.21.15.97:0/1509425799 shutdown_connections 2024-03-11T19:57:43.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.122+0000 7f2efd200700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2edc0a1b50 0x7f2edc0a4010 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.122+0000 7f2efd200700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ef8101f20 0x7f2ef8198200 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8005770 tx=0x7f2ee80057a0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.122+0000 7f2efd200700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ef8106ff0 0x7f2ef819cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.122+0000 7f2efd200700 1 --2- 172.21.15.97:0/1509425799 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ef81028d0 0x7f2ef8198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.122+0000 7f2efd200700 1 -- 172.21.15.97:0/1509425799 >> 172.21.15.97:0/1509425799 conn(0x7f2ef8078dd0 msgr2=0x7f2ef8106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:43.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.122+0000 7f2efd200700 1 -- 172.21.15.97:0/1509425799 shutdown_connections 2024-03-11T19:57:43.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.123+0000 7f2efd200700 1 -- 172.21.15.97:0/1509425799 wait complete. 2024-03-11T19:57:43.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.699+0000 7f2baee00700 1 -- 172.21.15.97:0/1420971780 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ba8075ca0 msgr2=0x7f2ba8073ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:43.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.699+0000 7f2baee00700 1 --2- 172.21.15.97:0/1420971780 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ba8075ca0 0x7f2ba8073ea0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009990 tx=0x7f2b98009ca0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.699+0000 7f2baee00700 1 -- 172.21.15.97:0/1420971780 shutdown_connections 2024-03-11T19:57:43.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.699+0000 7f2baee00700 1 --2- 172.21.15.97:0/1420971780 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ba8075ca0 0x7f2ba8073ea0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.699+0000 7f2baee00700 1 --2- 172.21.15.97:0/1420971780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ba8112900 0x7f2ba8114cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.699+0000 7f2baee00700 1 --2- 172.21.15.97:0/1420971780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ba80743e0 0x7f2ba8074860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.699+0000 7f2baee00700 1 -- 172.21.15.97:0/1420971780 >> 172.21.15.97:0/1420971780 conn(0x7f2ba8078da0 msgr2=0x7f2ba80791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:43.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.699+0000 7f2baee00700 1 -- 172.21.15.97:0/1420971780 shutdown_connections 2024-03-11T19:57:43.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.699+0000 7f2baee00700 1 -- 172.21.15.97:0/1420971780 wait complete. 2024-03-11T19:57:43.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.701+0000 7f2baee00700 1 Processor -- start 2024-03-11T19:57:43.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.701+0000 7f2baee00700 1 -- start start 2024-03-11T19:57:43.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.701+0000 7f2baee00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ba80743e0 0x7f2ba81a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:43.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.701+0000 7f2baee00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ba8075ca0 0x7f2ba81a4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:43.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.701+0000 7f2baee00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ba8112900 0x7f2ba819e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:43.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.701+0000 7f2baee00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba819ec50 con 0x7f2ba8075ca0 2024-03-11T19:57:43.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.701+0000 7f2baee00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba819ed90 con 0x7f2ba80743e0 2024-03-11T19:57:43.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.701+0000 7f2baee00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba819ef50 con 0x7f2ba8112900 2024-03-11T19:57:43.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.702+0000 7f2ba7e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ba80743e0 0x7f2ba81a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:43.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.702+0000 7f2ba7e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ba80743e0 0x7f2ba81a4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:47440/0 (socket says 172.21.15.97:47440) 2024-03-11T19:57:43.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.702+0000 7f2ba7e00700 1 -- 172.21.15.97:0/2937140485 learned_addr learned my addr 172.21.15.97:0/2937140485 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:43.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.702+0000 7f2b9fe00700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ba8075ca0 0x7f2ba81a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:43.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.702+0000 7f2bace00700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ba8112900 0x7f2ba819e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:43.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.702+0000 7f2ba7e00700 1 -- 172.21.15.97:0/2937140485 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ba8112900 msgr2=0x7f2ba819e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:43.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.702+0000 7f2ba7e00700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ba8112900 0x7f2ba819e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.702+0000 7f2ba7e00700 1 -- 172.21.15.97:0/2937140485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ba8075ca0 msgr2=0x7f2ba81a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:43.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.702+0000 7f2ba7e00700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ba8075ca0 0x7f2ba81a4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.702+0000 7f2ba7e00700 1 -- 172.21.15.97:0/2937140485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b980095f0 con 0x7f2ba80743e0 2024-03-11T19:57:43.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.702+0000 7f2ba7e00700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ba80743e0 0x7f2ba81a4430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2b98005910 tx=0x7f2b98004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:43.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.703+0000 7f2ba5600700 1 -- 172.21.15.97:0/2937140485 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9801d070 con 0x7f2ba80743e0 2024-03-11T19:57:43.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.703+0000 7f2ba5600700 1 -- 172.21.15.97:0/2937140485 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b9800b610 con 0x7f2ba80743e0 2024-03-11T19:57:43.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.703+0000 7f2baee00700 1 -- 172.21.15.97:0/2937140485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba819f1d0 con 0x7f2ba80743e0 2024-03-11T19:57:43.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.703+0000 7f2ba5600700 1 -- 172.21.15.97:0/2937140485 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b980173f0 con 0x7f2ba80743e0 2024-03-11T19:57:43.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.703+0000 7f2baee00700 1 -- 172.21.15.97:0/2937140485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba81ab4d0 con 0x7f2ba80743e0 2024-03-11T19:57:43.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.704+0000 7f2baee00700 1 -- 172.21.15.97:0/2937140485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba8068930 con 0x7f2ba80743e0 2024-03-11T19:57:43.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.706+0000 7f2ba5600700 1 -- 172.21.15.97:0/2937140485 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f2b98022470 con 0x7f2ba80743e0 2024-03-11T19:57:43.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.707+0000 7f2ba5600700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2b8c0a1bd0 0x7f2b8c0a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:43.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.707+0000 7f2ba5600700 1 -- 172.21.15.97:0/2937140485 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f2b980ceac0 con 0x7f2ba80743e0 2024-03-11T19:57:43.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.707+0000 7f2b9fe00700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2b8c0a1bd0 0x7f2b8c0a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:43.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.708+0000 7f2b9fe00700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2b8c0a1bd0 0x7f2b8c0a4090 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8079630 tx=0x7f2b90009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:43.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.710+0000 7f2ba5600700 1 -- 172.21.15.97:0/2937140485 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2b98097860 con 0x7f2ba80743e0 2024-03-11T19:57:43.894 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:43 smithi097 ceph-mon[31898]: pgmap v380: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:57:43.894 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:43 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1509425799' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T19:57:43.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.928+0000 7f2baee00700 1 -- 172.21.15.97:0/2937140485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2ba819fb20 con 0x7f2ba80743e0 2024-03-11T19:57:43.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.929+0000 7f2ba5600700 1 -- 172.21.15.97:0/2937140485 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2b9809af20 con 0x7f2ba80743e0 2024-03-11T19:57:43.930 INFO:teuthology.orchestra.run.smithi097.stdout:197568495681 2024-03-11T19:57:43.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.932+0000 7f2baee00700 1 -- 172.21.15.97:0/2937140485 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2b8c0a1bd0 msgr2=0x7f2b8c0a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:43.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.932+0000 7f2baee00700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2b8c0a1bd0 0x7f2b8c0a4090 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8079630 tx=0x7f2b90009260 comp rx=0 tx=0).stop 2024-03-11T19:57:43.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.932+0000 7f2baee00700 1 -- 172.21.15.97:0/2937140485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ba80743e0 msgr2=0x7f2ba81a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:43.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.932+0000 7f2baee00700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ba80743e0 0x7f2ba81a4430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2b98005910 tx=0x7f2b98004730 comp rx=0 tx=0).stop 2024-03-11T19:57:43.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.932+0000 7f2baee00700 1 -- 172.21.15.97:0/2937140485 shutdown_connections 2024-03-11T19:57:43.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.932+0000 7f2baee00700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2b8c0a1bd0 0x7f2b8c0a4090 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.932+0000 7f2baee00700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ba8112900 0x7f2ba819e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.932+0000 7f2baee00700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ba80743e0 0x7f2ba81a4430 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.932+0000 7f2baee00700 1 --2- 172.21.15.97:0/2937140485 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ba8075ca0 0x7f2ba81a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:43.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.932+0000 7f2baee00700 1 -- 172.21.15.97:0/2937140485 >> 172.21.15.97:0/2937140485 conn(0x7f2ba8078da0 msgr2=0x7f2ba8111660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:43.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.932+0000 7f2baee00700 1 -- 172.21.15.97:0/2937140485 shutdown_connections 2024-03-11T19:57:43.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:43.932+0000 7f2baee00700 1 -- 172.21.15.97:0/2937140485 wait complete. 2024-03-11T19:57:43.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:43 smithi114 ceph-mon[36376]: pgmap v380: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:57:43.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:43 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1509425799' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T19:57:44.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:43 smithi161 ceph-mon[36332]: pgmap v380: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:57:44.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:43 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1509425799' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T19:57:44.318 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182476 got 107374182483 for osd.3 2024-03-11T19:57:44.318 DEBUG:teuthology.parallel:result is None 2024-03-11T19:57:44.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:44 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2937140485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T19:57:44.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:44 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2937140485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T19:57:45.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:44 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2937140485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T19:57:45.125 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495675 got 197568495681 for osd.6 2024-03-11T19:57:45.126 DEBUG:teuthology.parallel:result is None 2024-03-11T19:57:45.126 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T19:57:45.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T19:57:45.578 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:45.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:45 smithi097 ceph-mon[31898]: pgmap v381: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:57:45.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-03-11T19:57:45.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-11T19:57:45.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-11T19:57:45.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:45 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-11T19:57:45.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:45 smithi114 ceph-mon[36376]: pgmap v381: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:57:45.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-03-11T19:57:45.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-11T19:57:45.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-11T19:57:45.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:45 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-11T19:57:46.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:45 smithi161 ceph-mon[36332]: pgmap v381: 129 pgs: 129 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:57:46.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-03-11T19:57:46.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-11T19:57:46.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-11T19:57:46.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:45 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-11T19:57:46.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:46 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-03-11T19:57:46.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:46 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-11T19:57:46.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:46 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-11T19:57:46.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:46 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-11T19:57:46.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:46 smithi114 ceph-mon[36376]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-11T19:57:46.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:46 smithi114 ceph-mon[36376]: osdmap e92: 12 total, 12 up, 12 in 2024-03-11T19:57:47.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:46 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-03-11T19:57:47.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:46 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-11T19:57:47.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:46 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-11T19:57:47.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:46 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-11T19:57:47.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:46 smithi161 ceph-mon[36332]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-11T19:57:47.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:46 smithi161 ceph-mon[36332]: osdmap e92: 12 total, 12 up, 12 in 2024-03-11T19:57:47.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:46 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-03-11T19:57:47.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:46 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-11T19:57:47.089 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:46 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-11T19:57:47.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:46 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-11T19:57:47.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:46 smithi097 ceph-mon[31898]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-11T19:57:47.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:46 smithi097 ceph-mon[31898]: osdmap e92: 12 total, 12 up, 12 in 2024-03-11T19:57:47.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.979+0000 7fcbadc00700 1 -- 172.21.15.97:0/690577775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8069920 msgr2=0x7fcba8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:47.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.979+0000 7fcbadc00700 1 --2- 172.21.15.97:0/690577775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8069920 0x7fcba8069d80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009990 tx=0x7fcb90009ca0 comp rx=0 tx=0).stop 2024-03-11T19:57:47.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.980+0000 7fcbadc00700 1 -- 172.21.15.97:0/690577775 shutdown_connections 2024-03-11T19:57:47.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.980+0000 7fcbadc00700 1 --2- 172.21.15.97:0/690577775 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba8107cc0 0x7fcba81080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:47.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.980+0000 7fcbadc00700 1 --2- 172.21.15.97:0/690577775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba806a2c0 0x7fcba810eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:47.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.980+0000 7fcbadc00700 1 --2- 172.21.15.97:0/690577775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8069920 0x7fcba8069d80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:47.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.980+0000 7fcbadc00700 1 -- 172.21.15.97:0/690577775 >> 172.21.15.97:0/690577775 conn(0x7fcba8077aa0 msgr2=0x7fcba8077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:47.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.980+0000 7fcbadc00700 1 -- 172.21.15.97:0/690577775 shutdown_connections 2024-03-11T19:57:47.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.980+0000 7fcbadc00700 1 -- 172.21.15.97:0/690577775 wait complete. 2024-03-11T19:57:47.981 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:47 smithi097 ceph-mon[31898]: pgmap v383: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-11T19:57:47.981 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:47 smithi097 ceph-mon[31898]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T19:57:47.981 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:47 smithi097 ceph-mon[31898]: osdmap e93: 12 total, 12 up, 12 in 2024-03-11T19:57:47.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.981+0000 7fcbadc00700 1 Processor -- start 2024-03-11T19:57:47.982 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.981+0000 7fcbadc00700 1 -- start start 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.981+0000 7fcbadc00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba8069920 0x7fcba819a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.981+0000 7fcbadc00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba806a2c0 0x7fcba819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcbadc00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8107cc0 0x7fcba819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcbadc00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba819f2a0 con 0x7fcba8069920 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcbadc00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba819f410 con 0x7fcba8107cc0 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcbadc00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba819f5d0 con 0x7fcba806a2c0 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcba6a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba8069920 0x7fcba819a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcba6a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba8069920 0x7fcba819a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58934/0 (socket says 172.21.15.97:58934) 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcba6a00700 1 -- 172.21.15.97:0/2961715535 learned_addr learned my addr 172.21.15.97:0/2961715535 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcb9fe00700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba806a2c0 0x7fcba819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcba7400700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8107cc0 0x7fcba819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcba6a00700 1 -- 172.21.15.97:0/2961715535 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba806a2c0 msgr2=0x7fcba819a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcba6a00700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba806a2c0 0x7fcba819a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcba6a00700 1 -- 172.21.15.97:0/2961715535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8107cc0 msgr2=0x7fcba819ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:47.983 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcba6a00700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8107cc0 0x7fcba819ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:47.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.982+0000 7fcba6a00700 1 -- 172.21.15.97:0/2961715535 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb900095f0 con 0x7fcba8069920 2024-03-11T19:57:47.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.983+0000 7fcba6a00700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba8069920 0x7fcba819a410 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800e9a0 tx=0x7fcb9800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:47.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.983+0000 7fcb9f400700 1 -- 172.21.15.97:0/2961715535 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98009790 con 0x7fcba8069920 2024-03-11T19:57:47.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.983+0000 7fcb9f400700 1 -- 172.21.15.97:0/2961715535 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb98010430 con 0x7fcba8069920 2024-03-11T19:57:47.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.983+0000 7fcbadc00700 1 -- 172.21.15.97:0/2961715535 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba819f8b0 con 0x7fcba8069920 2024-03-11T19:57:47.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.983+0000 7fcb9f400700 1 -- 172.21.15.97:0/2961715535 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98018680 con 0x7fcba8069920 2024-03-11T19:57:47.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.983+0000 7fcbadc00700 1 -- 172.21.15.97:0/2961715535 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba81a7110 con 0x7fcba8069920 2024-03-11T19:57:47.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.985+0000 7fcbadc00700 1 -- 172.21.15.97:0/2961715535 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba8068930 con 0x7fcba8069920 2024-03-11T19:57:47.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.987+0000 7fcb9f400700 1 -- 172.21.15.97:0/2961715535 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcb98010910 con 0x7fcba8069920 2024-03-11T19:57:47.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.988+0000 7fcb9f400700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb8c0a1ac0 0x7fcb8c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:47.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.988+0000 7fcb9f400700 1 -- 172.21.15.97:0/2961715535 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fcb98014070 con 0x7fcba8069920 2024-03-11T19:57:47.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.988+0000 7fcb9fe00700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb8c0a1ac0 0x7fcb8c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:47.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:47 smithi114 ceph-mon[36376]: pgmap v383: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-11T19:57:47.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:47 smithi114 ceph-mon[36376]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T19:57:47.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:47 smithi114 ceph-mon[36376]: osdmap e93: 12 total, 12 up, 12 in 2024-03-11T19:57:47.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.989+0000 7fcb9fe00700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb8c0a1ac0 0x7fcb8c0a3f80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcb900098e0 tx=0x7fcb90005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:47.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:47.993+0000 7fcb9f400700 1 -- 172.21.15.97:0/2961715535 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcb980967c0 con 0x7fcba8069920 2024-03-11T19:57:48.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:47 smithi161 ceph-mon[36332]: pgmap v383: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-11T19:57:48.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:47 smithi161 ceph-mon[36332]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T19:57:48.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:47 smithi161 ceph-mon[36332]: osdmap e93: 12 total, 12 up, 12 in 2024-03-11T19:57:48.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.206+0000 7fcbadc00700 1 -- 172.21.15.97:0/2961715535 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcba8111ca0 con 0x7fcb8c0a1ac0 2024-03-11T19:57:48.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.219+0000 7fcb9f400700 1 -- 172.21.15.97:0/2961715535 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340512 (secure 0 0 0) 0x7fcba8111ca0 con 0x7fcb8c0a1ac0 2024-03-11T19:57:48.227 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:57:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.227+0000 7fcbadc00700 1 -- 172.21.15.97:0/2961715535 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb8c0a1ac0 msgr2=0x7fcb8c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.227+0000 7fcbadc00700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb8c0a1ac0 0x7fcb8c0a3f80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcb900098e0 tx=0x7fcb90005960 comp rx=0 tx=0).stop 2024-03-11T19:57:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.227+0000 7fcbadc00700 1 -- 172.21.15.97:0/2961715535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba8069920 msgr2=0x7fcba819a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.227+0000 7fcbadc00700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba8069920 0x7fcba819a410 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800e9a0 tx=0x7fcb9800ecb0 comp rx=0 tx=0).stop 2024-03-11T19:57:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.227+0000 7fcbadc00700 1 -- 172.21.15.97:0/2961715535 shutdown_connections 2024-03-11T19:57:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.227+0000 7fcbadc00700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb8c0a1ac0 0x7fcb8c0a3f80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.227+0000 7fcbadc00700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcba8069920 0x7fcba819a410 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.227+0000 7fcbadc00700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8107cc0 0x7fcba819ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.227+0000 7fcbadc00700 1 --2- 172.21.15.97:0/2961715535 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcba806a2c0 0x7fcba819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.227+0000 7fcbadc00700 1 -- 172.21.15.97:0/2961715535 >> 172.21.15.97:0/2961715535 conn(0x7fcba8077aa0 msgr2=0x7fcba80ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.227+0000 7fcbadc00700 1 -- 172.21.15.97:0/2961715535 shutdown_connections 2024-03-11T19:57:48.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:48.228+0000 7fcbadc00700 1 -- 172.21.15.97:0/2961715535 wait complete. 2024-03-11T19:57:48.231 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-11T19:57:49.058 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-03-11T19:57:46.680491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659169,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1023,"num_read_kb":640,"num_write":427,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181318,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":569,"ondisk_log_size":569,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384276,"kb_used_data":14260,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557548,"statfs":{"total":1151940427776,"available":1151546929152,"internally_reserved":0,"allocated":14602240,"data_stored":7981230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":36,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.134898"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.688843+0000","last_change":"2024-03-11T19:55:45.725019+0000","last_active":"2024-03-11T19:56:45.688843+0000","last_peered":"2024-03-11T19:56:45.688843+0000","last_clean":"2024-03-11T19:56:45.688843+0000","last_became_active":"2024-03-11T19:55:45.724602+0000","last_became_peered":"2024-03-11T19:55:45.724602+0000","last_unstale":"2024-03-11T19:56:45.688843+0000","last_undegraded":"2024-03-11T19:56:45.688843+0000","last_fullsized":"2024-03-11T19:56:45.688843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:16:18.218381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.509479+0000","last_change":"2024-03-11T19:55:45.724589+0000","last_active":"2024-03-11T19:57:44.509479+0000","last_peered":"2024-03-11T19:57:44.509479+0000","last_clean":"2024-03-11T19:57:44.509479+0000","last_became_active":"2024-03-11T19:55:45.724109+0000","last_became_peered":"2024-03-11T19:55:45.724109+0000","last_unstale":"2024-03-11T19:57:44.509479+0000","last_undegraded":"2024-03-11T19:57:44.509479+0000","last_fullsized":"2024-03-11T19:57:44.509479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:58:12.510437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.519909+0000","last_change":"2024-03-11T19:55:45.718831+0000","last_active":"2024-03-11T19:57:44.519909+0000","last_peered":"2024-03-11T19:57:44.519909+0000","last_clean":"2024-03-11T19:57:44.519909+0000","last_became_active":"2024-03-11T19:55:45.717652+0000","last_became_peered":"2024-03-11T19:55:45.717652+0000","last_unstale":"2024-03-11T19:57:44.519909+0000","last_undegraded":"2024-03-11T19:57:44.519909+0000","last_fullsized":"2024-03-11T19:57:44.519909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:16.471754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.693407+0000","last_change":"2024-03-11T19:55:45.701640+0000","last_active":"2024-03-11T19:56:45.693407+0000","last_peered":"2024-03-11T19:56:45.693407+0000","last_clean":"2024-03-11T19:56:45.693407+0000","last_became_active":"2024-03-11T19:55:45.699207+0000","last_became_peered":"2024-03-11T19:55:45.699207+0000","last_unstale":"2024-03-11T19:56:45.693407+0000","last_undegraded":"2024-03-11T19:56:45.693407+0000","last_fullsized":"2024-03-11T19:56:45.693407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:01.900429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.692564+0000","last_change":"2024-03-11T19:55:45.732040+0000","last_active":"2024-03-11T19:56:45.692564+0000","last_peered":"2024-03-11T19:56:45.692564+0000","last_clean":"2024-03-11T19:56:45.692564+0000","last_became_active":"2024-03-11T19:55:45.731647+0000","last_became_peered":"2024-03-11T19:55:45.731647+0000","last_unstale":"2024-03-11T19:56:45.692564+0000","last_undegraded":"2024-03-11T19:56:45.692564+0000","last_fullsized":"2024-03-11T19:56:45.692564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:51:57.345329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.683589+0000","last_change":"2024-03-11T19:55:45.707986+0000","last_active":"2024-03-11T19:56:45.683589+0000","last_peered":"2024-03-11T19:56:45.683589+0000","last_clean":"2024-03-11T19:56:45.683589+0000","last_became_active":"2024-03-11T19:55:45.707641+0000","last_became_peered":"2024-03-11T19:55:45.707641+0000","last_unstale":"2024-03-11T19:56:45.683589+0000","last_undegraded":"2024-03-11T19:56:45.683589+0000","last_fullsized":"2024-03-11T19:56:45.683589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:15:50.635234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.705737+0000","last_change":"2024-03-11T19:56:45.705737+0000","last_active":"2024-03-11T19:56:45.705737+0000","last_peered":"2024-03-11T19:56:45.705737+0000","last_clean":"2024-03-11T19:56:45.705737+0000","last_became_active":"2024-03-11T19:56:45.705477+0000","last_became_peered":"2024-03-11T19:56:45.705477+0000","last_unstale":"2024-03-11T19:56:45.705737+0000","last_undegraded":"2024-03-11T19:56:45.705737+0000","last_fullsized":"2024-03-11T19:56:45.705737+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:01:14.068476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.520902+0000","last_change":"2024-03-11T19:55:45.729587+0000","last_active":"2024-03-11T19:57:44.520902+0000","last_peered":"2024-03-11T19:57:44.520902+0000","last_clean":"2024-03-11T19:57:44.520902+0000","last_became_active":"2024-03-11T19:55:45.729037+0000","last_became_peered":"2024-03-11T19:55:45.729037+0000","last_unstale":"2024-03-11T19:57:44.520902+0000","last_undegraded":"2024-03-11T19:57:44.520902+0000","last_fullsized":"2024-03-11T19:57:44.520902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:55:43.445192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.692048+0000","last_change":"2024-03-11T19:55:45.709793+0000","last_active":"2024-03-11T19:56:45.692048+0000","last_peered":"2024-03-11T19:56:45.692048+0000","last_clean":"2024-03-11T19:56:45.692048+0000","last_became_active":"2024-03-11T19:55:45.708631+0000","last_became_peered":"2024-03-11T19:55:45.708631+0000","last_unstale":"2024-03-11T19:56:45.692048+0000","last_undegraded":"2024-03-11T19:56:45.692048+0000","last_fullsized":"2024-03-11T19:56:45.692048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:02.144066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.684068+0000","last_change":"2024-03-11T19:55:45.704227+0000","last_active":"2024-03-11T19:56:45.684068+0000","last_peered":"2024-03-11T19:56:45.684068+0000","last_clean":"2024-03-11T19:56:45.684068+0000","last_became_active":"2024-03-11T19:55:45.703885+0000","last_became_peered":"2024-03-11T19:55:45.703885+0000","last_unstale":"2024-03-11T19:56:45.684068+0000","last_undegraded":"2024-03-11T19:56:45.684068+0000","last_fullsized":"2024-03-11T19:56:45.684068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:36:21.401988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.690765+0000","last_change":"2024-03-11T19:55:45.708950+0000","last_active":"2024-03-11T19:56:45.690765+0000","last_peered":"2024-03-11T19:56:45.690765+0000","last_clean":"2024-03-11T19:56:45.690765+0000","last_became_active":"2024-03-11T19:55:45.708544+0000","last_became_peered":"2024-03-11T19:55:45.708544+0000","last_unstale":"2024-03-11T19:56:45.690765+0000","last_undegraded":"2024-03-11T19:56:45.690765+0000","last_fullsized":"2024-03-11T19:56:45.690765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:06:34.244449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.693399+0000","last_change":"2024-03-11T19:55:45.703923+0000","last_active":"2024-03-11T19:56:45.693399+0000","last_peered":"2024-03-11T19:56:45.693399+0000","last_clean":"2024-03-11T19:56:45.693399+0000","last_became_active":"2024-03-11T19:55:45.703595+0000","last_became_peered":"2024-03-11T19:55:45.703595+0000","last_unstale":"2024-03-11T19:56:45.693399+0000","last_undegraded":"2024-03-11T19:56:45.693399+0000","last_fullsized":"2024-03-11T19:56:45.693399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:00:19.683907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692145+0000","last_change":"2024-03-11T19:55:45.707499+0000","last_active":"2024-03-11T19:57:45.692145+0000","last_peered":"2024-03-11T19:57:45.692145+0000","last_clean":"2024-03-11T19:57:45.692145+0000","last_became_active":"2024-03-11T19:55:45.706344+0000","last_became_peered":"2024-03-11T19:55:45.706344+0000","last_unstale":"2024-03-11T19:57:45.692145+0000","last_undegraded":"2024-03-11T19:57:45.692145+0000","last_fullsized":"2024-03-11T19:57:45.692145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:38:24.082332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.516834+0000","last_change":"2024-03-11T19:55:45.722182+0000","last_active":"2024-03-11T19:57:44.516834+0000","last_peered":"2024-03-11T19:57:44.516834+0000","last_clean":"2024-03-11T19:57:44.516834+0000","last_became_active":"2024-03-11T19:55:45.721783+0000","last_became_peered":"2024-03-11T19:55:45.721783+0000","last_unstale":"2024-03-11T19:57:44.516834+0000","last_undegraded":"2024-03-11T19:57:44.516834+0000","last_fullsized":"2024-03-11T19:57:44.516834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:41:08.559002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.519802+0000","last_change":"2024-03-11T19:55:45.718881+0000","last_active":"2024-03-11T19:57:44.519802+0000","last_peered":"2024-03-11T19:57:44.519802+0000","last_clean":"2024-03-11T19:57:44.519802+0000","last_became_active":"2024-03-11T19:55:45.718458+0000","last_became_peered":"2024-03-11T19:55:45.718458+0000","last_unstale":"2024-03-11T19:57:44.519802+0000","last_undegraded":"2024-03-11T19:57:44.519802+0000","last_fullsized":"2024-03-11T19:57:44.519802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:22.075892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.690339+0000","last_change":"2024-03-11T19:55:45.726163+0000","last_active":"2024-03-11T19:56:45.690339+0000","last_peered":"2024-03-11T19:56:45.690339+0000","last_clean":"2024-03-11T19:56:45.690339+0000","last_became_active":"2024-03-11T19:55:45.725772+0000","last_became_peered":"2024-03-11T19:55:45.725772+0000","last_unstale":"2024-03-11T19:56:45.690339+0000","last_undegraded":"2024-03-11T19:56:45.690339+0000","last_fullsized":"2024-03-11T19:56:45.690339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:36:19.057920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.688575+0000","last_change":"2024-03-11T19:55:45.724787+0000","last_active":"2024-03-11T19:56:45.688575+0000","last_peered":"2024-03-11T19:56:45.688575+0000","last_clean":"2024-03-11T19:56:45.688575+0000","last_became_active":"2024-03-11T19:55:45.724437+0000","last_became_peered":"2024-03-11T19:55:45.724437+0000","last_unstale":"2024-03-11T19:56:45.688575+0000","last_undegraded":"2024-03-11T19:56:45.688575+0000","last_fullsized":"2024-03-11T19:56:45.688575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:55:53.811679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.696946+0000","last_change":"2024-03-11T19:55:45.714934+0000","last_active":"2024-03-11T19:57:45.696946+0000","last_peered":"2024-03-11T19:57:45.696946+0000","last_clean":"2024-03-11T19:57:45.696946+0000","last_became_active":"2024-03-11T19:55:45.709595+0000","last_became_peered":"2024-03-11T19:55:45.709595+0000","last_unstale":"2024-03-11T19:57:45.696946+0000","last_undegraded":"2024-03-11T19:57:45.696946+0000","last_fullsized":"2024-03-11T19:57:45.696946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:48.319073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.686689+0000","last_change":"2024-03-11T19:55:45.725747+0000","last_active":"2024-03-11T19:56:45.686689+0000","last_peered":"2024-03-11T19:56:45.686689+0000","last_clean":"2024-03-11T19:56:45.686689+0000","last_became_active":"2024-03-11T19:55:45.725239+0000","last_became_peered":"2024-03-11T19:55:45.725239+0000","last_unstale":"2024-03-11T19:56:45.686689+0000","last_undegraded":"2024-03-11T19:56:45.686689+0000","last_fullsized":"2024-03-11T19:56:45.686689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:51:24.046681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.686300+0000","last_change":"2024-03-11T19:55:50.109913+0000","last_active":"2024-03-11T19:56:45.686300+0000","last_peered":"2024-03-11T19:56:45.686300+0000","last_clean":"2024-03-11T19:56:45.686300+0000","last_became_active":"2024-03-11T19:55:50.109471+0000","last_became_peered":"2024-03-11T19:55:50.109471+0000","last_unstale":"2024-03-11T19:56:45.686300+0000","last_undegraded":"2024-03-11T19:56:45.686300+0000","last_fullsized":"2024-03-11T19:56:45.686300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:40.258022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":129,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.697208+0000","last_change":"2024-03-11T19:55:44.680181+0000","last_active":"2024-03-11T19:57:45.697208+0000","last_peered":"2024-03-11T19:57:45.697208+0000","last_clean":"2024-03-11T19:57:45.697208+0000","last_became_active":"2024-03-11T19:55:44.679807+0000","last_became_peered":"2024-03-11T19:55:44.679807+0000","last_unstale":"2024-03-11T19:57:45.697208+0000","last_undegraded":"2024-03-11T19:57:45.697208+0000","last_fullsized":"2024-03-11T19:57:45.697208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:23:04.532836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.513547+0000","last_change":"2024-03-11T19:55:45.723547+0000","last_active":"2024-03-11T19:57:44.513547+0000","last_peered":"2024-03-11T19:57:44.513547+0000","last_clean":"2024-03-11T19:57:44.513547+0000","last_became_active":"2024-03-11T19:55:45.722516+0000","last_became_peered":"2024-03-11T19:55:45.722516+0000","last_unstale":"2024-03-11T19:57:44.513547+0000","last_undegraded":"2024-03-11T19:57:44.513547+0000","last_fullsized":"2024-03-11T19:57:44.513547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:40.271814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.692914+0000","last_change":"2024-03-11T19:55:50.102141+0000","last_active":"2024-03-11T19:56:45.692914+0000","last_peered":"2024-03-11T19:56:45.692914+0000","last_clean":"2024-03-11T19:56:45.692914+0000","last_became_active":"2024-03-11T19:55:50.101667+0000","last_became_peered":"2024-03-11T19:55:50.101667+0000","last_unstale":"2024-03-11T19:56:45.692914+0000","last_undegraded":"2024-03-11T19:56:45.692914+0000","last_fullsized":"2024-03-11T19:56:45.692914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:54:11.453059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.688908+0000","last_change":"2024-03-11T19:55:44.684623+0000","last_active":"2024-03-11T19:56:45.688908+0000","last_peered":"2024-03-11T19:56:45.688908+0000","last_clean":"2024-03-11T19:56:45.688908+0000","last_became_active":"2024-03-11T19:55:44.683611+0000","last_became_peered":"2024-03-11T19:55:44.683611+0000","last_unstale":"2024-03-11T19:56:45.688908+0000","last_undegraded":"2024-03-11T19:56:45.688908+0000","last_fullsized":"2024-03-11T19:56:45.688908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:22:50.954364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.523733+0000","last_change":"2024-03-11T19:55:45.723619+0000","last_active":"2024-03-11T19:57:44.523733+0000","last_peered":"2024-03-11T19:57:44.523733+0000","last_clean":"2024-03-11T19:57:44.523733+0000","last_became_active":"2024-03-11T19:55:45.722685+0000","last_became_peered":"2024-03-11T19:55:45.722685+0000","last_unstale":"2024-03-11T19:57:44.523733+0000","last_undegraded":"2024-03-11T19:57:44.523733+0000","last_fullsized":"2024-03-11T19:57:44.523733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:47:13.747776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.693091+0000","last_change":"2024-03-11T19:55:50.102898+0000","last_active":"2024-03-11T19:56:45.693091+0000","last_peered":"2024-03-11T19:56:45.693091+0000","last_clean":"2024-03-11T19:56:45.693091+0000","last_became_active":"2024-03-11T19:55:50.102491+0000","last_became_peered":"2024-03-11T19:55:50.102491+0000","last_unstale":"2024-03-11T19:56:45.693091+0000","last_undegraded":"2024-03-11T19:56:45.693091+0000","last_fullsized":"2024-03-11T19:56:45.693091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:41:25.389542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.714549+0000","last_change":"2024-03-11T19:56:45.714194+0000","last_active":"2024-03-11T19:56:45.714549+0000","last_peered":"2024-03-11T19:56:45.714549+0000","last_clean":"2024-03-11T19:56:45.714549+0000","last_became_active":"2024-03-11T19:56:45.694559+0000","last_became_peered":"2024-03-11T19:56:45.694559+0000","last_unstale":"2024-03-11T19:56:45.714549+0000","last_undegraded":"2024-03-11T19:56:45.714549+0000","last_fullsized":"2024-03-11T19:56:45.714549+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:01:20.307915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.691883+0000","last_change":"2024-03-11T19:55:45.715157+0000","last_active":"2024-03-11T19:56:45.691883+0000","last_peered":"2024-03-11T19:56:45.691883+0000","last_clean":"2024-03-11T19:56:45.691883+0000","last_became_active":"2024-03-11T19:55:45.714536+0000","last_became_peered":"2024-03-11T19:55:45.714536+0000","last_unstale":"2024-03-11T19:56:45.691883+0000","last_undegraded":"2024-03-11T19:56:45.691883+0000","last_fullsized":"2024-03-11T19:56:45.691883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:02.416736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.693070+0000","last_change":"2024-03-11T19:55:50.102157+0000","last_active":"2024-03-11T19:56:45.693070+0000","last_peered":"2024-03-11T19:56:45.693070+0000","last_clean":"2024-03-11T19:56:45.693070+0000","last_became_active":"2024-03-11T19:55:50.101715+0000","last_became_peered":"2024-03-11T19:55:50.101715+0000","last_unstale":"2024-03-11T19:56:45.693070+0000","last_undegraded":"2024-03-11T19:56:45.693070+0000","last_fullsized":"2024-03-11T19:56:45.693070+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:12:18.780708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.699180+0000","last_change":"2024-03-11T19:56:45.698585+0000","last_active":"2024-03-11T19:56:45.699180+0000","last_peered":"2024-03-11T19:56:45.699180+0000","last_clean":"2024-03-11T19:56:45.699180+0000","last_became_active":"2024-03-11T19:56:45.692929+0000","last_became_peered":"2024-03-11T19:56:45.692929+0000","last_unstale":"2024-03-11T19:56:45.699180+0000","last_undegraded":"2024-03-11T19:56:45.699180+0000","last_fullsized":"2024-03-11T19:56:45.699180+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:43:11.158934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.686907+0000","last_change":"2024-03-11T19:55:45.709973+0000","last_active":"2024-03-11T19:57:45.686907+0000","last_peered":"2024-03-11T19:57:45.686907+0000","last_clean":"2024-03-11T19:57:45.686907+0000","last_became_active":"2024-03-11T19:55:45.709645+0000","last_became_peered":"2024-03-11T19:55:45.709645+0000","last_unstale":"2024-03-11T19:57:45.686907+0000","last_undegraded":"2024-03-11T19:57:45.686907+0000","last_fullsized":"2024-03-11T19:57:45.686907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:16.516982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.686517+0000","last_change":"2024-03-11T19:55:50.094093+0000","last_active":"2024-03-11T19:56:45.686517+0000","last_peered":"2024-03-11T19:56:45.686517+0000","last_clean":"2024-03-11T19:56:45.686517+0000","last_became_active":"2024-03-11T19:55:50.093653+0000","last_became_peered":"2024-03-11T19:55:50.093653+0000","last_unstale":"2024-03-11T19:56:45.686517+0000","last_undegraded":"2024-03-11T19:56:45.686517+0000","last_fullsized":"2024-03-11T19:56:45.686517+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:28:41.668001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":127,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.686915+0000","last_change":"2024-03-11T19:55:44.671547+0000","last_active":"2024-03-11T19:57:45.686915+0000","last_peered":"2024-03-11T19:57:45.686915+0000","last_clean":"2024-03-11T19:57:45.686915+0000","last_became_active":"2024-03-11T19:55:44.671211+0000","last_became_peered":"2024-03-11T19:55:44.671211+0000","last_unstale":"2024-03-11T19:57:45.686915+0000","last_undegraded":"2024-03-11T19:57:45.686915+0000","last_fullsized":"2024-03-11T19:57:45.686915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:18:46.261250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.692356+0000","last_change":"2024-03-11T19:55:45.707968+0000","last_active":"2024-03-11T19:56:45.692356+0000","last_peered":"2024-03-11T19:56:45.692356+0000","last_clean":"2024-03-11T19:56:45.692356+0000","last_became_active":"2024-03-11T19:55:45.706998+0000","last_became_peered":"2024-03-11T19:55:45.706998+0000","last_unstale":"2024-03-11T19:56:45.692356+0000","last_undegraded":"2024-03-11T19:56:45.692356+0000","last_fullsized":"2024-03-11T19:56:45.692356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:49:35.711123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.686555+0000","last_change":"2024-03-11T19:55:50.104310+0000","last_active":"2024-03-11T19:56:45.686555+0000","last_peered":"2024-03-11T19:56:45.686555+0000","last_clean":"2024-03-11T19:56:45.686555+0000","last_became_active":"2024-03-11T19:55:50.103831+0000","last_became_peered":"2024-03-11T19:55:50.103831+0000","last_unstale":"2024-03-11T19:56:45.686555+0000","last_undegraded":"2024-03-11T19:56:45.686555+0000","last_fullsized":"2024-03-11T19:56:45.686555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:33.213987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.683589+0000","last_change":"2024-03-11T19:55:44.677615+0000","last_active":"2024-03-11T19:56:45.683589+0000","last_peered":"2024-03-11T19:56:45.683589+0000","last_clean":"2024-03-11T19:56:45.683589+0000","last_became_active":"2024-03-11T19:55:44.677153+0000","last_became_peered":"2024-03-11T19:55:44.677153+0000","last_unstale":"2024-03-11T19:56:45.683589+0000","last_undegraded":"2024-03-11T19:56:45.683589+0000","last_fullsized":"2024-03-11T19:56:45.683589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:52:59.153222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687012+0000","last_change":"2024-03-11T19:55:45.731827+0000","last_active":"2024-03-11T19:57:45.687012+0000","last_peered":"2024-03-11T19:57:45.687012+0000","last_clean":"2024-03-11T19:57:45.687012+0000","last_became_active":"2024-03-11T19:55:45.731460+0000","last_became_peered":"2024-03-11T19:55:45.731460+0000","last_unstale":"2024-03-11T19:57:45.687012+0000","last_undegraded":"2024-03-11T19:57:45.687012+0000","last_fullsized":"2024-03-11T19:57:45.687012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:12.277198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.688371+0000","last_change":"2024-03-11T19:55:50.109812+0000","last_active":"2024-03-11T19:56:45.688371+0000","last_peered":"2024-03-11T19:56:45.688371+0000","last_clean":"2024-03-11T19:56:45.688371+0000","last_became_active":"2024-03-11T19:55:50.109420+0000","last_became_peered":"2024-03-11T19:55:50.109420+0000","last_unstale":"2024-03-11T19:56:45.688371+0000","last_undegraded":"2024-03-11T19:56:45.688371+0000","last_fullsized":"2024-03-11T19:56:45.688371+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:08:30.105941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.691625+0000","last_change":"2024-03-11T19:55:44.683173+0000","last_active":"2024-03-11T19:57:45.691625+0000","last_peered":"2024-03-11T19:57:45.691625+0000","last_clean":"2024-03-11T19:57:45.691625+0000","last_became_active":"2024-03-11T19:55:44.682778+0000","last_became_peered":"2024-03-11T19:55:44.682778+0000","last_unstale":"2024-03-11T19:57:45.691625+0000","last_undegraded":"2024-03-11T19:57:45.691625+0000","last_fullsized":"2024-03-11T19:57:45.691625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:15.119864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.526800+0000","last_change":"2024-03-11T19:55:45.721833+0000","last_active":"2024-03-11T19:57:44.526800+0000","last_peered":"2024-03-11T19:57:44.526800+0000","last_clean":"2024-03-11T19:57:44.526800+0000","last_became_active":"2024-03-11T19:55:45.721490+0000","last_became_peered":"2024-03-11T19:55:45.721490+0000","last_unstale":"2024-03-11T19:57:44.526800+0000","last_undegraded":"2024-03-11T19:57:44.526800+0000","last_fullsized":"2024-03-11T19:57:44.526800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:09.824912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.700087+0000","last_change":"2024-03-11T19:55:50.108424+0000","last_active":"2024-03-11T19:56:45.700087+0000","last_peered":"2024-03-11T19:56:45.700087+0000","last_clean":"2024-03-11T19:56:45.700087+0000","last_became_active":"2024-03-11T19:55:50.107072+0000","last_became_peered":"2024-03-11T19:55:50.107072+0000","last_unstale":"2024-03-11T19:56:45.700087+0000","last_undegraded":"2024-03-11T19:56:45.700087+0000","last_fullsized":"2024-03-11T19:56:45.700087+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:31.751597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.686954+0000","last_change":"2024-03-11T19:55:44.671570+0000","last_active":"2024-03-11T19:57:45.686954+0000","last_peered":"2024-03-11T19:57:45.686954+0000","last_clean":"2024-03-11T19:57:45.686954+0000","last_became_active":"2024-03-11T19:55:44.671007+0000","last_became_peered":"2024-03-11T19:55:44.671007+0000","last_unstale":"2024-03-11T19:57:45.686954+0000","last_undegraded":"2024-03-11T19:57:45.686954+0000","last_fullsized":"2024-03-11T19:57:45.686954+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:23.865011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.695987+0000","last_change":"2024-03-11T19:55:45.733048+0000","last_active":"2024-03-11T19:57:45.695987+0000","last_peered":"2024-03-11T19:57:45.695987+0000","last_clean":"2024-03-11T19:57:45.695987+0000","last_became_active":"2024-03-11T19:55:45.731656+0000","last_became_peered":"2024-03-11T19:55:45.731656+0000","last_unstale":"2024-03-11T19:57:45.695987+0000","last_undegraded":"2024-03-11T19:57:45.695987+0000","last_fullsized":"2024-03-11T19:57:45.695987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:28:56.433327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.685508+0000","last_change":"2024-03-11T19:55:50.105532+0000","last_active":"2024-03-11T19:56:45.685508+0000","last_peered":"2024-03-11T19:56:45.685508+0000","last_clean":"2024-03-11T19:56:45.685508+0000","last_became_active":"2024-03-11T19:55:50.102200+0000","last_became_peered":"2024-03-11T19:55:50.102200+0000","last_unstale":"2024-03-11T19:56:45.685508+0000","last_undegraded":"2024-03-11T19:56:45.685508+0000","last_fullsized":"2024-03-11T19:56:45.685508+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:37.060472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":198,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:41.757494+0000","last_change":"2024-03-11T19:55:44.678224+0000","last_active":"2024-03-11T19:57:41.757494+0000","last_peered":"2024-03-11T19:57:41.757494+0000","last_clean":"2024-03-11T19:57:41.757494+0000","last_became_active":"2024-03-11T19:55:44.677936+0000","last_became_peered":"2024-03-11T19:55:44.677936+0000","last_unstale":"2024-03-11T19:57:41.757494+0000","last_undegraded":"2024-03-11T19:57:41.757494+0000","last_fullsized":"2024-03-11T19:57:41.757494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:20:42.486501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":168,"num_read_kb":168,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.691890+0000","last_change":"2024-03-11T19:55:45.709917+0000","last_active":"2024-03-11T19:56:45.691890+0000","last_peered":"2024-03-11T19:56:45.691890+0000","last_clean":"2024-03-11T19:56:45.691890+0000","last_became_active":"2024-03-11T19:55:45.709318+0000","last_became_peered":"2024-03-11T19:55:45.709318+0000","last_unstale":"2024-03-11T19:56:45.691890+0000","last_undegraded":"2024-03-11T19:56:45.691890+0000","last_fullsized":"2024-03-11T19:56:45.691890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:08.298158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.692937+0000","last_change":"2024-03-11T19:55:50.103384+0000","last_active":"2024-03-11T19:56:45.692937+0000","last_peered":"2024-03-11T19:56:45.692937+0000","last_clean":"2024-03-11T19:56:45.692937+0000","last_became_active":"2024-03-11T19:55:50.100641+0000","last_became_peered":"2024-03-11T19:55:50.100641+0000","last_unstale":"2024-03-11T19:56:45.692937+0000","last_undegraded":"2024-03-11T19:56:45.692937+0000","last_fullsized":"2024-03-11T19:56:45.692937+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:58.524369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.691040+0000","last_change":"2024-03-11T19:55:44.676004+0000","last_active":"2024-03-11T19:56:45.691040+0000","last_peered":"2024-03-11T19:56:45.691040+0000","last_clean":"2024-03-11T19:56:45.691040+0000","last_became_active":"2024-03-11T19:55:44.675640+0000","last_became_peered":"2024-03-11T19:55:44.675640+0000","last_unstale":"2024-03-11T19:56:45.691040+0000","last_undegraded":"2024-03-11T19:56:45.691040+0000","last_fullsized":"2024-03-11T19:56:45.691040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:15.096780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.522910+0000","last_change":"2024-03-11T19:56:45.702569+0000","last_active":"2024-03-11T19:57:44.522910+0000","last_peered":"2024-03-11T19:57:44.522910+0000","last_clean":"2024-03-11T19:57:44.522910+0000","last_became_active":"2024-03-11T19:56:45.701890+0000","last_became_peered":"2024-03-11T19:56:45.701890+0000","last_unstale":"2024-03-11T19:57:44.522910+0000","last_undegraded":"2024-03-11T19:57:44.522910+0000","last_fullsized":"2024-03-11T19:57:44.522910+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:40.434712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.693263+0000","last_change":"2024-03-11T19:55:50.096136+0000","last_active":"2024-03-11T19:56:45.693263+0000","last_peered":"2024-03-11T19:56:45.693263+0000","last_clean":"2024-03-11T19:56:45.693263+0000","last_became_active":"2024-03-11T19:55:50.095670+0000","last_became_peered":"2024-03-11T19:55:50.095670+0000","last_unstale":"2024-03-11T19:56:45.693263+0000","last_undegraded":"2024-03-11T19:56:45.693263+0000","last_fullsized":"2024-03-11T19:56:45.693263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:40.943273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.686791+0000","last_change":"2024-03-11T19:55:44.681547+0000","last_active":"2024-03-11T19:57:45.686791+0000","last_peered":"2024-03-11T19:57:45.686791+0000","last_clean":"2024-03-11T19:57:45.686791+0000","last_became_active":"2024-03-11T19:55:44.681237+0000","last_became_peered":"2024-03-11T19:55:44.681237+0000","last_unstale":"2024-03-11T19:57:45.686791+0000","last_undegraded":"2024-03-11T19:57:45.686791+0000","last_fullsized":"2024-03-11T19:57:45.686791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:08:09.509915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.692001+0000","last_change":"2024-03-11T19:55:45.705025+0000","last_active":"2024-03-11T19:56:45.692001+0000","last_peered":"2024-03-11T19:56:45.692001+0000","last_clean":"2024-03-11T19:56:45.692001+0000","last_became_active":"2024-03-11T19:55:45.702680+0000","last_became_peered":"2024-03-11T19:55:45.702680+0000","last_unstale":"2024-03-11T19:56:45.692001+0000","last_undegraded":"2024-03-11T19:56:45.692001+0000","last_fullsized":"2024-03-11T19:56:45.692001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:28.245002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.695555+0000","last_change":"2024-03-11T19:55:50.099648+0000","last_active":"2024-03-11T19:57:45.695555+0000","last_peered":"2024-03-11T19:57:45.695555+0000","last_clean":"2024-03-11T19:57:45.695555+0000","last_became_active":"2024-03-11T19:55:50.098516+0000","last_became_peered":"2024-03-11T19:55:50.098516+0000","last_unstale":"2024-03-11T19:57:45.695555+0000","last_undegraded":"2024-03-11T19:57:45.695555+0000","last_fullsized":"2024-03-11T19:57:45.695555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:19.644046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'32","reported_seq":232,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:41.757446+0000","last_change":"2024-03-11T19:55:44.672460+0000","last_active":"2024-03-11T19:57:41.757446+0000","last_peered":"2024-03-11T19:57:41.757446+0000","last_clean":"2024-03-11T19:57:41.757446+0000","last_became_active":"2024-03-11T19:55:44.672123+0000","last_became_peered":"2024-03-11T19:55:44.672123+0000","last_unstale":"2024-03-11T19:57:41.757446+0000","last_undegraded":"2024-03-11T19:57:41.757446+0000","last_fullsized":"2024-03-11T19:57:41.757446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:43.940638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61222,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":174,"num_read_kb":147,"num_write":34,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.693409+0000","last_change":"2024-03-11T19:55:45.702933+0000","last_active":"2024-03-11T19:56:45.693409+0000","last_peered":"2024-03-11T19:56:45.693409+0000","last_clean":"2024-03-11T19:56:45.693409+0000","last_became_active":"2024-03-11T19:55:45.702585+0000","last_became_peered":"2024-03-11T19:55:45.702585+0000","last_unstale":"2024-03-11T19:56:45.693409+0000","last_undegraded":"2024-03-11T19:56:45.693409+0000","last_fullsized":"2024-03-11T19:56:45.693409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:54:38.762156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687748+0000","last_change":"2024-03-11T19:55:50.102571+0000","last_active":"2024-03-11T19:57:45.687748+0000","last_peered":"2024-03-11T19:57:45.687748+0000","last_clean":"2024-03-11T19:57:45.687748+0000","last_became_active":"2024-03-11T19:55:50.102099+0000","last_became_peered":"2024-03-11T19:55:50.102099+0000","last_unstale":"2024-03-11T19:57:45.687748+0000","last_undegraded":"2024-03-11T19:57:45.687748+0000","last_fullsized":"2024-03-11T19:57:45.687748+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:17.631378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.686385+0000","last_change":"2024-03-11T19:55:44.671027+0000","last_active":"2024-03-11T19:56:45.686385+0000","last_peered":"2024-03-11T19:56:45.686385+0000","last_clean":"2024-03-11T19:56:45.686385+0000","last_became_active":"2024-03-11T19:55:44.669696+0000","last_became_peered":"2024-03-11T19:55:44.669696+0000","last_unstale":"2024-03-11T19:56:45.686385+0000","last_undegraded":"2024-03-11T19:56:45.686385+0000","last_fullsized":"2024-03-11T19:56:45.686385+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:48:20.169638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.691595+0000","last_change":"2024-03-11T19:55:45.729119+0000","last_active":"2024-03-11T19:57:45.691595+0000","last_peered":"2024-03-11T19:57:45.691595+0000","last_clean":"2024-03-11T19:57:45.691595+0000","last_became_active":"2024-03-11T19:55:45.727442+0000","last_became_peered":"2024-03-11T19:55:45.727442+0000","last_unstale":"2024-03-11T19:57:45.691595+0000","last_undegraded":"2024-03-11T19:57:45.691595+0000","last_fullsized":"2024-03-11T19:57:45.691595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:14:51.483199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.688270+0000","last_change":"2024-03-11T19:55:50.104326+0000","last_active":"2024-03-11T19:56:45.688270+0000","last_peered":"2024-03-11T19:56:45.688270+0000","last_clean":"2024-03-11T19:56:45.688270+0000","last_became_active":"2024-03-11T19:55:50.103881+0000","last_became_peered":"2024-03-11T19:55:50.103881+0000","last_unstale":"2024-03-11T19:56:45.688270+0000","last_undegraded":"2024-03-11T19:56:45.688270+0000","last_fullsized":"2024-03-11T19:56:45.688270+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:12:38.819591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.686622+0000","last_change":"2024-03-11T19:55:44.684900+0000","last_active":"2024-03-11T19:57:45.686622+0000","last_peered":"2024-03-11T19:57:45.686622+0000","last_clean":"2024-03-11T19:57:45.686622+0000","last_became_active":"2024-03-11T19:55:44.684590+0000","last_became_peered":"2024-03-11T19:55:44.684590+0000","last_unstale":"2024-03-11T19:57:45.686622+0000","last_undegraded":"2024-03-11T19:57:45.686622+0000","last_fullsized":"2024-03-11T19:57:45.686622+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:07.744366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.690674+0000","last_change":"2024-03-11T19:55:45.704787+0000","last_active":"2024-03-11T19:56:45.690674+0000","last_peered":"2024-03-11T19:56:45.690674+0000","last_clean":"2024-03-11T19:56:45.690674+0000","last_became_active":"2024-03-11T19:55:45.704405+0000","last_became_peered":"2024-03-11T19:55:45.704405+0000","last_unstale":"2024-03-11T19:56:45.690674+0000","last_undegraded":"2024-03-11T19:56:45.690674+0000","last_fullsized":"2024-03-11T19:56:45.690674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:58:05.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.697029+0000","last_change":"2024-03-11T19:55:50.104793+0000","last_active":"2024-03-11T19:57:45.697029+0000","last_peered":"2024-03-11T19:57:45.697029+0000","last_clean":"2024-03-11T19:57:45.697029+0000","last_became_active":"2024-03-11T19:55:50.104415+0000","last_became_peered":"2024-03-11T19:55:50.104415+0000","last_unstale":"2024-03-11T19:57:45.697029+0000","last_undegraded":"2024-03-11T19:57:45.697029+0000","last_fullsized":"2024-03-11T19:57:45.697029+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:53.307313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":117,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:40.779472+0000","last_change":"2024-03-11T19:55:44.669403+0000","last_active":"2024-03-11T19:57:40.779472+0000","last_peered":"2024-03-11T19:57:40.779472+0000","last_clean":"2024-03-11T19:57:40.779472+0000","last_became_active":"2024-03-11T19:55:44.668908+0000","last_became_peered":"2024-03-11T19:55:44.668908+0000","last_unstale":"2024-03-11T19:57:40.779472+0000","last_undegraded":"2024-03-11T19:57:40.779472+0000","last_fullsized":"2024-03-11T19:57:40.779472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:56:36.831506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":87,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.691012+0000","last_change":"2024-03-11T19:55:45.700029+0000","last_active":"2024-03-11T19:56:45.691012+0000","last_peered":"2024-03-11T19:56:45.691012+0000","last_clean":"2024-03-11T19:56:45.691012+0000","last_became_active":"2024-03-11T19:55:45.697296+0000","last_became_peered":"2024-03-11T19:55:45.697296+0000","last_unstale":"2024-03-11T19:56:45.691012+0000","last_undegraded":"2024-03-11T19:56:45.691012+0000","last_fullsized":"2024-03-11T19:56:45.691012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:59:35.025845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"peering","last_fresh":"2024-03-11T19:56:45.700253+0000","last_change":"2024-03-11T19:57:45.687743+0000","last_active":"2024-03-11T19:57:45.687743+0000","last_peered":"2024-03-11T19:56:45.700253+0000","last_clean":"2024-03-11T19:56:45.700253+0000","last_became_active":"2024-03-11T19:55:50.108182+0000","last_became_peered":"2024-03-11T19:55:50.108182+0000","last_unstale":"2024-03-11T19:56:45.700253+0000","last_undegraded":"2024-03-11T19:56:45.700253+0000","last_fullsized":"2024-03-11T19:56:45.700253+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,8,9,11],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.692347+0000","last_change":"2024-03-11T19:55:44.677491+0000","last_active":"2024-03-11T19:56:45.692347+0000","last_peered":"2024-03-11T19:56:45.692347+0000","last_clean":"2024-03-11T19:56:45.692347+0000","last_became_active":"2024-03-11T19:55:44.677176+0000","last_became_peered":"2024-03-11T19:55:44.677176+0000","last_unstale":"2024-03-11T19:56:45.692347+0000","last_undegraded":"2024-03-11T19:56:45.692347+0000","last_fullsized":"2024-03-11T19:56:45.692347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:51.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687518+0000","last_change":"2024-03-11T19:55:45.703310+0000","last_active":"2024-03-11T19:57:45.687518+0000","last_peered":"2024-03-11T19:57:45.687518+0000","last_clean":"2024-03-11T19:57:45.687518+0000","last_became_active":"2024-03-11T19:55:45.702933+0000","last_became_peered":"2024-03-11T19:55:45.702933+0000","last_unstale":"2024-03-11T19:57:45.687518+0000","last_undegraded":"2024-03-11T19:57:45.687518+0000","last_fullsized":"2024-03-11T19:57:45.687518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:07:58.787949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687431+0000","last_change":"2024-03-11T19:55:50.103849+0000","last_active":"2024-03-11T19:57:45.687431+0000","last_peered":"2024-03-11T19:57:45.687431+0000","last_clean":"2024-03-11T19:57:45.687431+0000","last_became_active":"2024-03-11T19:55:50.103360+0000","last_became_peered":"2024-03-11T19:55:50.103360+0000","last_unstale":"2024-03-11T19:57:45.687431+0000","last_undegraded":"2024-03-11T19:57:45.687431+0000","last_fullsized":"2024-03-11T19:57:45.687431+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:38:05.151041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.692943+0000","last_change":"2024-03-11T19:55:44.669839+0000","last_active":"2024-03-11T19:56:45.692943+0000","last_peered":"2024-03-11T19:56:45.692943+0000","last_clean":"2024-03-11T19:56:45.692943+0000","last_became_active":"2024-03-11T19:55:44.669494+0000","last_became_peered":"2024-03-11T19:55:44.669494+0000","last_unstale":"2024-03-11T19:56:45.692943+0000","last_undegraded":"2024-03-11T19:56:45.692943+0000","last_fullsized":"2024-03-11T19:56:45.692943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:00.313847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.693864+0000","last_change":"2024-03-11T19:56:45.693864+0000","last_active":"2024-03-11T19:56:45.693864+0000","last_peered":"2024-03-11T19:56:45.693864+0000","last_clean":"2024-03-11T19:56:45.693864+0000","last_became_active":"2024-03-11T19:56:45.693091+0000","last_became_peered":"2024-03-11T19:56:45.693091+0000","last_unstale":"2024-03-11T19:56:45.693864+0000","last_undegraded":"2024-03-11T19:56:45.693864+0000","last_fullsized":"2024-03-11T19:56:45.693864+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:26:37.233485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.692315+0000","last_change":"2024-03-11T19:55:50.143916+0000","last_active":"2024-03-11T19:56:45.692315+0000","last_peered":"2024-03-11T19:56:45.692315+0000","last_clean":"2024-03-11T19:56:45.692315+0000","last_became_active":"2024-03-11T19:55:50.143341+0000","last_became_peered":"2024-03-11T19:55:50.143341+0000","last_unstale":"2024-03-11T19:56:45.692315+0000","last_undegraded":"2024-03-11T19:56:45.692315+0000","last_fullsized":"2024-03-11T19:56:45.692315+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:40.637529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.696901+0000","last_change":"2024-03-11T19:55:44.688468+0000","last_active":"2024-03-11T19:57:45.696901+0000","last_peered":"2024-03-11T19:57:45.696901+0000","last_clean":"2024-03-11T19:57:45.696901+0000","last_became_active":"2024-03-11T19:55:44.688065+0000","last_became_peered":"2024-03-11T19:55:44.688065+0000","last_unstale":"2024-03-11T19:57:45.696901+0000","last_undegraded":"2024-03-11T19:57:45.696901+0000","last_fullsized":"2024-03-11T19:57:45.696901+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:29:23.392749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.683843+0000","last_change":"2024-03-11T19:55:45.728970+0000","last_active":"2024-03-11T19:56:45.683843+0000","last_peered":"2024-03-11T19:56:45.683843+0000","last_clean":"2024-03-11T19:56:45.683843+0000","last_became_active":"2024-03-11T19:55:45.728516+0000","last_became_peered":"2024-03-11T19:55:45.728516+0000","last_unstale":"2024-03-11T19:56:45.683843+0000","last_undegraded":"2024-03-11T19:56:45.683843+0000","last_fullsized":"2024-03-11T19:56:45.683843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:19.242610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692260+0000","last_change":"2024-03-11T19:55:50.111884+0000","last_active":"2024-03-11T19:57:45.692260+0000","last_peered":"2024-03-11T19:57:45.692260+0000","last_clean":"2024-03-11T19:57:45.692260+0000","last_became_active":"2024-03-11T19:55:50.110802+0000","last_became_peered":"2024-03-11T19:55:50.110802+0000","last_unstale":"2024-03-11T19:57:45.692260+0000","last_undegraded":"2024-03-11T19:57:45.692260+0000","last_fullsized":"2024-03-11T19:57:45.692260+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:35.866368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:40.824852+0000","last_change":"2024-03-11T19:55:44.677156+0000","last_active":"2024-03-11T19:57:40.824852+0000","last_peered":"2024-03-11T19:57:40.824852+0000","last_clean":"2024-03-11T19:57:40.824852+0000","last_became_active":"2024-03-11T19:55:44.676656+0000","last_became_peered":"2024-03-11T19:55:44.676656+0000","last_unstale":"2024-03-11T19:57:40.824852+0000","last_undegraded":"2024-03-11T19:57:40.824852+0000","last_fullsized":"2024-03-11T19:57:40.824852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:40:27.521669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.683589+0000","last_change":"2024-03-11T19:55:45.715545+0000","last_active":"2024-03-11T19:56:45.683589+0000","last_peered":"2024-03-11T19:56:45.683589+0000","last_clean":"2024-03-11T19:56:45.683589+0000","last_became_active":"2024-03-11T19:55:45.715042+0000","last_became_peered":"2024-03-11T19:55:45.715042+0000","last_unstale":"2024-03-11T19:56:45.683589+0000","last_undegraded":"2024-03-11T19:56:45.683589+0000","last_fullsized":"2024-03-11T19:56:45.683589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:06:33.362235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692138+0000","last_change":"2024-03-11T19:55:50.145603+0000","last_active":"2024-03-11T19:57:45.692138+0000","last_peered":"2024-03-11T19:57:45.692138+0000","last_clean":"2024-03-11T19:57:45.692138+0000","last_became_active":"2024-03-11T19:55:50.143689+0000","last_became_peered":"2024-03-11T19:55:50.143689+0000","last_unstale":"2024-03-11T19:57:45.692138+0000","last_undegraded":"2024-03-11T19:57:45.692138+0000","last_fullsized":"2024-03-11T19:57:45.692138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:57.289983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687250+0000","last_change":"2024-03-11T19:55:45.702011+0000","last_active":"2024-03-11T19:57:45.687250+0000","last_peered":"2024-03-11T19:57:45.687250+0000","last_clean":"2024-03-11T19:57:45.687250+0000","last_became_active":"2024-03-11T19:55:45.701673+0000","last_became_peered":"2024-03-11T19:55:45.701673+0000","last_unstale":"2024-03-11T19:57:45.687250+0000","last_undegraded":"2024-03-11T19:57:45.687250+0000","last_fullsized":"2024-03-11T19:57:45.687250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:17.028276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":123,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.691567+0000","last_change":"2024-03-11T19:55:44.689121+0000","last_active":"2024-03-11T19:57:45.691567+0000","last_peered":"2024-03-11T19:57:45.691567+0000","last_clean":"2024-03-11T19:57:45.691567+0000","last_became_active":"2024-03-11T19:55:44.688453+0000","last_became_peered":"2024-03-11T19:55:44.688453+0000","last_unstale":"2024-03-11T19:57:45.691567+0000","last_undegraded":"2024-03-11T19:57:45.691567+0000","last_fullsized":"2024-03-11T19:57:45.691567+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:31.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":89,"num_read_kb":89,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.690639+0000","last_change":"2024-03-11T19:55:50.098694+0000","last_active":"2024-03-11T19:56:45.690639+0000","last_peered":"2024-03-11T19:56:45.690639+0000","last_clean":"2024-03-11T19:56:45.690639+0000","last_became_active":"2024-03-11T19:55:50.098121+0000","last_became_peered":"2024-03-11T19:55:50.098121+0000","last_unstale":"2024-03-11T19:56:45.690639+0000","last_undegraded":"2024-03-11T19:56:45.690639+0000","last_fullsized":"2024-03-11T19:56:45.690639+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:29:11.046703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.696088+0000","last_change":"2024-03-11T19:56:45.703746+0000","last_active":"2024-03-11T19:57:45.696088+0000","last_peered":"2024-03-11T19:57:45.696088+0000","last_clean":"2024-03-11T19:57:45.696088+0000","last_became_active":"2024-03-11T19:56:45.701252+0000","last_became_peered":"2024-03-11T19:56:45.701252+0000","last_unstale":"2024-03-11T19:57:45.696088+0000","last_undegraded":"2024-03-11T19:57:45.696088+0000","last_fullsized":"2024-03-11T19:57:45.696088+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:16:50.219593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.693234+0000","last_change":"2024-03-11T19:55:44.672644+0000","last_active":"2024-03-11T19:56:45.693234+0000","last_peered":"2024-03-11T19:56:45.693234+0000","last_clean":"2024-03-11T19:56:45.693234+0000","last_became_active":"2024-03-11T19:55:44.671645+0000","last_became_peered":"2024-03-11T19:55:44.671645+0000","last_unstale":"2024-03-11T19:56:45.693234+0000","last_undegraded":"2024-03-11T19:56:45.693234+0000","last_fullsized":"2024-03-11T19:56:45.693234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:25:46.637768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":363,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.690732+0000","last_change":"2024-03-11T19:54:20.915891+0000","last_active":"2024-03-11T19:56:45.690732+0000","last_peered":"2024-03-11T19:56:45.690732+0000","last_clean":"2024-03-11T19:56:45.690732+0000","last_became_active":"2024-03-11T19:54:20.915450+0000","last_became_peered":"2024-03-11T19:54:20.915450+0000","last_unstale":"2024-03-11T19:56:45.690732+0000","last_undegraded":"2024-03-11T19:56:45.690732+0000","last_fullsized":"2024-03-11T19:56:45.690732+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_clean_scrub_stamp":"2024-03-11T19:50:29.052858+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:47.966076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687164+0000","last_change":"2024-03-11T19:55:50.144053+0000","last_active":"2024-03-11T19:57:45.687164+0000","last_peered":"2024-03-11T19:57:45.687164+0000","last_clean":"2024-03-11T19:57:45.687164+0000","last_became_active":"2024-03-11T19:55:50.143509+0000","last_became_peered":"2024-03-11T19:55:50.143509+0000","last_unstale":"2024-03-11T19:57:45.687164+0000","last_undegraded":"2024-03-11T19:57:45.687164+0000","last_fullsized":"2024-03-11T19:57:45.687164+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:57:15.448812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.691409+0000","last_change":"2024-03-11T19:55:45.712621+0000","last_active":"2024-03-11T19:57:45.691409+0000","last_peered":"2024-03-11T19:57:45.691409+0000","last_clean":"2024-03-11T19:57:45.691409+0000","last_became_active":"2024-03-11T19:55:45.712161+0000","last_became_peered":"2024-03-11T19:55:45.712161+0000","last_unstale":"2024-03-11T19:57:45.691409+0000","last_undegraded":"2024-03-11T19:57:45.691409+0000","last_fullsized":"2024-03-11T19:57:45.691409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:34:52.911944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.690863+0000","last_change":"2024-03-11T19:55:44.674606+0000","last_active":"2024-03-11T19:56:45.690863+0000","last_peered":"2024-03-11T19:56:45.690863+0000","last_clean":"2024-03-11T19:56:45.690863+0000","last_became_active":"2024-03-11T19:55:44.674355+0000","last_became_peered":"2024-03-11T19:55:44.674355+0000","last_unstale":"2024-03-11T19:56:45.690863+0000","last_undegraded":"2024-03-11T19:56:45.690863+0000","last_fullsized":"2024-03-11T19:56:45.690863+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:31:01.662582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.688395+0000","last_change":"2024-03-11T19:55:50.101468+0000","last_active":"2024-03-11T19:56:45.688395+0000","last_peered":"2024-03-11T19:56:45.688395+0000","last_clean":"2024-03-11T19:56:45.688395+0000","last_became_active":"2024-03-11T19:55:50.101040+0000","last_became_peered":"2024-03-11T19:55:50.101040+0000","last_unstale":"2024-03-11T19:56:45.688395+0000","last_undegraded":"2024-03-11T19:56:45.688395+0000","last_fullsized":"2024-03-11T19:56:45.688395+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:55.354176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.693422+0000","last_change":"2024-03-11T19:55:44.681467+0000","last_active":"2024-03-11T19:56:45.693422+0000","last_peered":"2024-03-11T19:56:45.693422+0000","last_clean":"2024-03-11T19:56:45.693422+0000","last_became_active":"2024-03-11T19:55:44.681176+0000","last_became_peered":"2024-03-11T19:55:44.681176+0000","last_unstale":"2024-03-11T19:56:45.693422+0000","last_undegraded":"2024-03-11T19:56:45.693422+0000","last_fullsized":"2024-03-11T19:56:45.693422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:27:38.019116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692424+0000","last_change":"2024-03-11T19:55:45.709798+0000","last_active":"2024-03-11T19:57:45.692424+0000","last_peered":"2024-03-11T19:57:45.692424+0000","last_clean":"2024-03-11T19:57:45.692424+0000","last_became_active":"2024-03-11T19:55:45.709277+0000","last_became_peered":"2024-03-11T19:55:45.709277+0000","last_unstale":"2024-03-11T19:57:45.692424+0000","last_undegraded":"2024-03-11T19:57:45.692424+0000","last_fullsized":"2024-03-11T19:57:45.692424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:58.852316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.685512+0000","last_change":"2024-03-11T19:55:50.146601+0000","last_active":"2024-03-11T19:56:45.685512+0000","last_peered":"2024-03-11T19:56:45.685512+0000","last_clean":"2024-03-11T19:56:45.685512+0000","last_became_active":"2024-03-11T19:55:50.144584+0000","last_became_peered":"2024-03-11T19:55:50.144584+0000","last_unstale":"2024-03-11T19:56:45.685512+0000","last_undegraded":"2024-03-11T19:56:45.685512+0000","last_fullsized":"2024-03-11T19:56:45.685512+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:00.365713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'14","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.696915+0000","last_change":"2024-03-11T19:55:44.689109+0000","last_active":"2024-03-11T19:57:45.696915+0000","last_peered":"2024-03-11T19:57:45.696915+0000","last_clean":"2024-03-11T19:57:45.696915+0000","last_became_active":"2024-03-11T19:55:44.687323+0000","last_became_peered":"2024-03-11T19:55:44.687323+0000","last_unstale":"2024-03-11T19:57:45.696915+0000","last_undegraded":"2024-03-11T19:57:45.696915+0000","last_fullsized":"2024-03-11T19:57:45.696915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:24.613901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.519164+0000","last_change":"2024-03-11T19:55:45.726033+0000","last_active":"2024-03-11T19:57:44.519164+0000","last_peered":"2024-03-11T19:57:44.519164+0000","last_clean":"2024-03-11T19:57:44.519164+0000","last_became_active":"2024-03-11T19:55:45.725353+0000","last_became_peered":"2024-03-11T19:55:45.725353+0000","last_unstale":"2024-03-11T19:57:44.519164+0000","last_undegraded":"2024-03-11T19:57:44.519164+0000","last_fullsized":"2024-03-11T19:57:44.519164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:36:24.341870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.686082+0000","last_change":"2024-03-11T19:55:50.089496+0000","last_active":"2024-03-11T19:56:45.686082+0000","last_peered":"2024-03-11T19:56:45.686082+0000","last_clean":"2024-03-11T19:56:45.686082+0000","last_became_active":"2024-03-11T19:55:50.089072+0000","last_became_peered":"2024-03-11T19:55:50.089072+0000","last_unstale":"2024-03-11T19:56:45.686082+0000","last_undegraded":"2024-03-11T19:56:45.686082+0000","last_fullsized":"2024-03-11T19:56:45.686082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:27:32.539333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.693606+0000","last_change":"2024-03-11T19:55:44.673147+0000","last_active":"2024-03-11T19:56:45.693606+0000","last_peered":"2024-03-11T19:56:45.693606+0000","last_clean":"2024-03-11T19:56:45.693606+0000","last_became_active":"2024-03-11T19:55:44.672787+0000","last_became_peered":"2024-03-11T19:55:44.672787+0000","last_unstale":"2024-03-11T19:56:45.693606+0000","last_undegraded":"2024-03-11T19:56:45.693606+0000","last_fullsized":"2024-03-11T19:56:45.693606+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:11:23.864419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.523344+0000","last_change":"2024-03-11T19:55:45.732376+0000","last_active":"2024-03-11T19:57:44.523344+0000","last_peered":"2024-03-11T19:57:44.523344+0000","last_clean":"2024-03-11T19:57:44.523344+0000","last_became_active":"2024-03-11T19:55:45.731956+0000","last_became_peered":"2024-03-11T19:55:45.731956+0000","last_unstale":"2024-03-11T19:57:44.523344+0000","last_undegraded":"2024-03-11T19:57:44.523344+0000","last_fullsized":"2024-03-11T19:57:44.523344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:43:48.423678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.693364+0000","last_change":"2024-03-11T19:55:50.144080+0000","last_active":"2024-03-11T19:56:45.693364+0000","last_peered":"2024-03-11T19:56:45.693364+0000","last_clean":"2024-03-11T19:56:45.693364+0000","last_became_active":"2024-03-11T19:55:50.143585+0000","last_became_peered":"2024-03-11T19:55:50.143585+0000","last_unstale":"2024-03-11T19:56:45.693364+0000","last_undegraded":"2024-03-11T19:56:45.693364+0000","last_fullsized":"2024-03-11T19:56:45.693364+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:47:51.388823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.690372+0000","last_change":"2024-03-11T19:55:44.677354+0000","last_active":"2024-03-11T19:56:45.690372+0000","last_peered":"2024-03-11T19:56:45.690372+0000","last_clean":"2024-03-11T19:56:45.690372+0000","last_became_active":"2024-03-11T19:55:44.677013+0000","last_became_peered":"2024-03-11T19:55:44.677013+0000","last_unstale":"2024-03-11T19:56:45.690372+0000","last_undegraded":"2024-03-11T19:56:45.690372+0000","last_fullsized":"2024-03-11T19:56:45.690372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:50:48.855320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.692065+0000","last_change":"2024-03-11T19:55:45.702278+0000","last_active":"2024-03-11T19:56:45.692065+0000","last_peered":"2024-03-11T19:56:45.692065+0000","last_clean":"2024-03-11T19:56:45.692065+0000","last_became_active":"2024-03-11T19:55:45.701843+0000","last_became_peered":"2024-03-11T19:55:45.701843+0000","last_unstale":"2024-03-11T19:56:45.692065+0000","last_undegraded":"2024-03-11T19:56:45.692065+0000","last_fullsized":"2024-03-11T19:56:45.692065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:55:28.351093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.697109+0000","last_change":"2024-03-11T19:55:50.106083+0000","last_active":"2024-03-11T19:57:45.697109+0000","last_peered":"2024-03-11T19:57:45.697109+0000","last_clean":"2024-03-11T19:57:45.697109+0000","last_became_active":"2024-03-11T19:55:50.105591+0000","last_became_peered":"2024-03-11T19:55:50.105591+0000","last_unstale":"2024-03-11T19:57:45.697109+0000","last_undegraded":"2024-03-11T19:57:45.697109+0000","last_fullsized":"2024-03-11T19:57:45.697109+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:34:58.713672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.686053+0000","last_change":"2024-03-11T19:55:44.688128+0000","last_active":"2024-03-11T19:56:45.686053+0000","last_peered":"2024-03-11T19:56:45.686053+0000","last_clean":"2024-03-11T19:56:45.686053+0000","last_became_active":"2024-03-11T19:55:44.684165+0000","last_became_peered":"2024-03-11T19:55:44.684165+0000","last_unstale":"2024-03-11T19:56:45.686053+0000","last_undegraded":"2024-03-11T19:56:45.686053+0000","last_fullsized":"2024-03-11T19:56:45.686053+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:37:19.415960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.688069+0000","last_change":"2024-03-11T19:55:45.718968+0000","last_active":"2024-03-11T19:56:45.688069+0000","last_peered":"2024-03-11T19:56:45.688069+0000","last_clean":"2024-03-11T19:56:45.688069+0000","last_became_active":"2024-03-11T19:55:45.718625+0000","last_became_peered":"2024-03-11T19:55:45.718625+0000","last_unstale":"2024-03-11T19:56:45.688069+0000","last_undegraded":"2024-03-11T19:56:45.688069+0000","last_fullsized":"2024-03-11T19:56:45.688069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:27.197227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.693192+0000","last_change":"2024-03-11T19:55:50.097617+0000","last_active":"2024-03-11T19:56:45.693192+0000","last_peered":"2024-03-11T19:56:45.693192+0000","last_clean":"2024-03-11T19:56:45.693192+0000","last_became_active":"2024-03-11T19:55:50.097137+0000","last_became_peered":"2024-03-11T19:55:50.097137+0000","last_unstale":"2024-03-11T19:56:45.693192+0000","last_undegraded":"2024-03-11T19:56:45.693192+0000","last_fullsized":"2024-03-11T19:56:45.693192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:19:20.995137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.686267+0000","last_change":"2024-03-11T19:55:44.669944+0000","last_active":"2024-03-11T19:56:45.686267+0000","last_peered":"2024-03-11T19:56:45.686267+0000","last_clean":"2024-03-11T19:56:45.686267+0000","last_became_active":"2024-03-11T19:55:44.669309+0000","last_became_peered":"2024-03-11T19:55:44.669309+0000","last_unstale":"2024-03-11T19:56:45.686267+0000","last_undegraded":"2024-03-11T19:56:45.686267+0000","last_fullsized":"2024-03-11T19:56:45.686267+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:06:41.489579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.520289+0000","last_change":"2024-03-11T19:55:45.731348+0000","last_active":"2024-03-11T19:57:44.520289+0000","last_peered":"2024-03-11T19:57:44.520289+0000","last_clean":"2024-03-11T19:57:44.520289+0000","last_became_active":"2024-03-11T19:55:45.730957+0000","last_became_peered":"2024-03-11T19:55:45.730957+0000","last_unstale":"2024-03-11T19:57:44.520289+0000","last_undegraded":"2024-03-11T19:57:44.520289+0000","last_fullsized":"2024-03-11T19:57:44.520289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:06:50.422511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.693626+0000","last_change":"2024-03-11T19:55:50.144050+0000","last_active":"2024-03-11T19:56:45.693626+0000","last_peered":"2024-03-11T19:56:45.693626+0000","last_clean":"2024-03-11T19:56:45.693626+0000","last_became_active":"2024-03-11T19:55:50.143591+0000","last_became_peered":"2024-03-11T19:55:50.143591+0000","last_unstale":"2024-03-11T19:56:45.693626+0000","last_undegraded":"2024-03-11T19:56:45.693626+0000","last_fullsized":"2024-03-11T19:56:45.693626+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:02.006665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.692666+0000","last_change":"2024-03-11T19:55:44.671973+0000","last_active":"2024-03-11T19:56:45.692666+0000","last_peered":"2024-03-11T19:56:45.692666+0000","last_clean":"2024-03-11T19:56:45.692666+0000","last_became_active":"2024-03-11T19:55:44.671661+0000","last_became_peered":"2024-03-11T19:55:44.671661+0000","last_unstale":"2024-03-11T19:56:45.692666+0000","last_undegraded":"2024-03-11T19:56:45.692666+0000","last_fullsized":"2024-03-11T19:56:45.692666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:35.747386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.690863+0000","last_change":"2024-03-11T19:55:45.705545+0000","last_active":"2024-03-11T19:56:45.690863+0000","last_peered":"2024-03-11T19:56:45.690863+0000","last_clean":"2024-03-11T19:56:45.690863+0000","last_became_active":"2024-03-11T19:55:45.705083+0000","last_became_peered":"2024-03-11T19:55:45.705083+0000","last_unstale":"2024-03-11T19:56:45.690863+0000","last_undegraded":"2024-03-11T19:56:45.690863+0000","last_fullsized":"2024-03-11T19:56:45.690863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:02:58.219026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.691556+0000","last_change":"2024-03-11T19:55:50.097951+0000","last_active":"2024-03-11T19:56:45.691556+0000","last_peered":"2024-03-11T19:56:45.691556+0000","last_clean":"2024-03-11T19:56:45.691556+0000","last_became_active":"2024-03-11T19:55:50.097446+0000","last_became_peered":"2024-03-11T19:55:50.097446+0000","last_unstale":"2024-03-11T19:56:45.691556+0000","last_undegraded":"2024-03-11T19:56:45.691556+0000","last_fullsized":"2024-03-11T19:56:45.691556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:12.105801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'6","reported_seq":120,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:40.786657+0000","last_change":"2024-03-11T19:55:44.673136+0000","last_active":"2024-03-11T19:57:40.786657+0000","last_peered":"2024-03-11T19:57:40.786657+0000","last_clean":"2024-03-11T19:57:40.786657+0000","last_became_active":"2024-03-11T19:55:44.672467+0000","last_became_peered":"2024-03-11T19:55:44.672467+0000","last_unstale":"2024-03-11T19:57:40.786657+0000","last_undegraded":"2024-03-11T19:57:40.786657+0000","last_fullsized":"2024-03-11T19:57:40.786657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:46:05.931253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":87,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.691478+0000","last_change":"2024-03-11T19:55:45.710561+0000","last_active":"2024-03-11T19:56:45.691478+0000","last_peered":"2024-03-11T19:56:45.691478+0000","last_clean":"2024-03-11T19:56:45.691478+0000","last_became_active":"2024-03-11T19:55:45.710164+0000","last_became_peered":"2024-03-11T19:55:45.710164+0000","last_unstale":"2024-03-11T19:56:45.691478+0000","last_undegraded":"2024-03-11T19:56:45.691478+0000","last_fullsized":"2024-03-11T19:56:45.691478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:00:33.977595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.698886+0000","last_change":"2024-03-11T19:55:50.113390+0000","last_active":"2024-03-11T19:56:45.698886+0000","last_peered":"2024-03-11T19:56:45.698886+0000","last_clean":"2024-03-11T19:56:45.698886+0000","last_became_active":"2024-03-11T19:55:50.112656+0000","last_became_peered":"2024-03-11T19:55:50.112656+0000","last_unstale":"2024-03-11T19:56:45.698886+0000","last_undegraded":"2024-03-11T19:56:45.698886+0000","last_fullsized":"2024-03-11T19:56:45.698886+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:11.952585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687033+0000","last_change":"2024-03-11T19:55:44.684907+0000","last_active":"2024-03-11T19:57:45.687033+0000","last_peered":"2024-03-11T19:57:45.687033+0000","last_clean":"2024-03-11T19:57:45.687033+0000","last_became_active":"2024-03-11T19:55:44.684600+0000","last_became_peered":"2024-03-11T19:55:44.684600+0000","last_unstale":"2024-03-11T19:57:45.687033+0000","last_undegraded":"2024-03-11T19:57:45.687033+0000","last_fullsized":"2024-03-11T19:57:45.687033+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:12:33.666756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.690710+0000","last_change":"2024-03-11T19:55:45.723431+0000","last_active":"2024-03-11T19:56:45.690710+0000","last_peered":"2024-03-11T19:56:45.690710+0000","last_clean":"2024-03-11T19:56:45.690710+0000","last_became_active":"2024-03-11T19:55:45.723135+0000","last_became_peered":"2024-03-11T19:55:45.723135+0000","last_unstale":"2024-03-11T19:56:45.690710+0000","last_undegraded":"2024-03-11T19:56:45.690710+0000","last_fullsized":"2024-03-11T19:56:45.690710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:12:35.654094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.686759+0000","last_change":"2024-03-11T19:55:50.092876+0000","last_active":"2024-03-11T19:57:45.686759+0000","last_peered":"2024-03-11T19:57:45.686759+0000","last_clean":"2024-03-11T19:57:45.686759+0000","last_became_active":"2024-03-11T19:55:50.092478+0000","last_became_peered":"2024-03-11T19:55:50.092478+0000","last_unstale":"2024-03-11T19:57:45.686759+0000","last_undegraded":"2024-03-11T19:57:45.686759+0000","last_fullsized":"2024-03-11T19:57:45.686759+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:39.248325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692817+0000","last_change":"2024-03-11T19:56:45.711269+0000","last_active":"2024-03-11T19:57:45.692817+0000","last_peered":"2024-03-11T19:57:45.692817+0000","last_clean":"2024-03-11T19:57:45.692817+0000","last_became_active":"2024-03-11T19:56:45.704657+0000","last_became_peered":"2024-03-11T19:56:45.704657+0000","last_unstale":"2024-03-11T19:57:45.692817+0000","last_undegraded":"2024-03-11T19:57:45.692817+0000","last_fullsized":"2024-03-11T19:57:45.692817+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:51.549965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.691269+0000","last_change":"2024-03-11T19:55:45.702924+0000","last_active":"2024-03-11T19:56:45.691269+0000","last_peered":"2024-03-11T19:56:45.691269+0000","last_clean":"2024-03-11T19:56:45.691269+0000","last_became_active":"2024-03-11T19:55:45.702530+0000","last_became_peered":"2024-03-11T19:55:45.702530+0000","last_unstale":"2024-03-11T19:56:45.691269+0000","last_undegraded":"2024-03-11T19:56:45.691269+0000","last_fullsized":"2024-03-11T19:56:45.691269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:14.679964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.688126+0000","last_change":"2024-03-11T19:55:45.706123+0000","last_active":"2024-03-11T19:56:45.688126+0000","last_peered":"2024-03-11T19:56:45.688126+0000","last_clean":"2024-03-11T19:56:45.688126+0000","last_became_active":"2024-03-11T19:55:45.705706+0000","last_became_peered":"2024-03-11T19:55:45.705706+0000","last_unstale":"2024-03-11T19:56:45.688126+0000","last_undegraded":"2024-03-11T19:56:45.688126+0000","last_fullsized":"2024-03-11T19:56:45.688126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:42:58.989823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.686448+0000","last_change":"2024-03-11T19:55:45.703188+0000","last_active":"2024-03-11T19:56:45.686448+0000","last_peered":"2024-03-11T19:56:45.686448+0000","last_clean":"2024-03-11T19:56:45.686448+0000","last_became_active":"2024-03-11T19:55:45.702803+0000","last_became_peered":"2024-03-11T19:55:45.702803+0000","last_unstale":"2024-03-11T19:56:45.686448+0000","last_undegraded":"2024-03-11T19:56:45.686448+0000","last_fullsized":"2024-03-11T19:56:45.686448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:20.922963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.686351+0000","last_change":"2024-03-11T19:55:45.728856+0000","last_active":"2024-03-11T19:56:45.686351+0000","last_peered":"2024-03-11T19:56:45.686351+0000","last_clean":"2024-03-11T19:56:45.686351+0000","last_became_active":"2024-03-11T19:55:45.728526+0000","last_became_peered":"2024-03-11T19:55:45.728526+0000","last_unstale":"2024-03-11T19:56:45.686351+0000","last_undegraded":"2024-03-11T19:56:45.686351+0000","last_fullsized":"2024-03-11T19:56:45.686351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:25:30.836073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.527515+0000","last_change":"2024-03-11T19:55:45.725075+0000","last_active":"2024-03-11T19:57:44.527515+0000","last_peered":"2024-03-11T19:57:44.527515+0000","last_clean":"2024-03-11T19:57:44.527515+0000","last_became_active":"2024-03-11T19:55:45.724627+0000","last_became_peered":"2024-03-11T19:55:45.724627+0000","last_unstale":"2024-03-11T19:57:44.527515+0000","last_undegraded":"2024-03-11T19:57:44.527515+0000","last_fullsized":"2024-03-11T19:57:44.527515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:01.039660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692385+0000","last_change":"2024-03-11T19:55:45.718879+0000","last_active":"2024-03-11T19:57:45.692385+0000","last_peered":"2024-03-11T19:57:45.692385+0000","last_clean":"2024-03-11T19:57:45.692385+0000","last_became_active":"2024-03-11T19:55:45.718535+0000","last_became_peered":"2024-03-11T19:55:45.718535+0000","last_unstale":"2024-03-11T19:57:45.692385+0000","last_undegraded":"2024-03-11T19:57:45.692385+0000","last_fullsized":"2024-03-11T19:57:45.692385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:48.236368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.689198+0000","last_change":"2024-03-11T19:55:45.703432+0000","last_active":"2024-03-11T19:56:45.689198+0000","last_peered":"2024-03-11T19:56:45.689198+0000","last_clean":"2024-03-11T19:56:45.689198+0000","last_became_active":"2024-03-11T19:55:45.702669+0000","last_became_peered":"2024-03-11T19:55:45.702669+0000","last_unstale":"2024-03-11T19:56:45.689198+0000","last_undegraded":"2024-03-11T19:56:45.689198+0000","last_fullsized":"2024-03-11T19:56:45.689198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:02.496981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.686821+0000","last_change":"2024-03-11T19:55:45.708733+0000","last_active":"2024-03-11T19:56:45.686821+0000","last_peered":"2024-03-11T19:56:45.686821+0000","last_clean":"2024-03-11T19:56:45.686821+0000","last_became_active":"2024-03-11T19:55:45.708369+0000","last_became_peered":"2024-03-11T19:55:45.708369+0000","last_unstale":"2024-03-11T19:56:45.686821+0000","last_undegraded":"2024-03-11T19:56:45.686821+0000","last_fullsized":"2024-03-11T19:56:45.686821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:36.619802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:57:44.491000+0000","last_change":"2024-03-11T19:55:45.725698+0000","last_active":"2024-03-11T19:57:44.491000+0000","last_peered":"2024-03-11T19:57:44.491000+0000","last_clean":"2024-03-11T19:57:44.491000+0000","last_became_active":"2024-03-11T19:55:45.724872+0000","last_became_peered":"2024-03-11T19:55:45.724872+0000","last_unstale":"2024-03-11T19:57:44.491000+0000","last_undegraded":"2024-03-11T19:57:44.491000+0000","last_fullsized":"2024-03-11T19:57:44.491000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.982969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687079+0000","last_change":"2024-03-11T19:55:45.726923+0000","last_active":"2024-03-11T19:57:45.687079+0000","last_peered":"2024-03-11T19:57:45.687079+0000","last_clean":"2024-03-11T19:57:45.687079+0000","last_became_active":"2024-03-11T19:55:45.726589+0000","last_became_peered":"2024-03-11T19:55:45.726589+0000","last_unstale":"2024-03-11T19:57:45.687079+0000","last_undegraded":"2024-03-11T19:57:45.687079+0000","last_fullsized":"2024-03-11T19:57:45.687079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:29:15.197359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.695652+0000","last_change":"2024-03-11T19:56:45.698623+0000","last_active":"2024-03-11T19:57:45.695652+0000","last_peered":"2024-03-11T19:57:45.695652+0000","last_clean":"2024-03-11T19:57:45.695652+0000","last_became_active":"2024-03-11T19:56:45.697805+0000","last_became_peered":"2024-03-11T19:56:45.697805+0000","last_unstale":"2024-03-11T19:57:45.695652+0000","last_undegraded":"2024-03-11T19:57:45.695652+0000","last_fullsized":"2024-03-11T19:57:45.695652+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:16:31.624856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.691897+0000","last_change":"2024-03-11T19:55:45.729141+0000","last_active":"2024-03-11T19:56:45.691897+0000","last_peered":"2024-03-11T19:56:45.691897+0000","last_clean":"2024-03-11T19:56:45.691897+0000","last_became_active":"2024-03-11T19:55:45.728781+0000","last_became_peered":"2024-03-11T19:55:45.728781+0000","last_unstale":"2024-03-11T19:56:45.691897+0000","last_undegraded":"2024-03-11T19:56:45.691897+0000","last_fullsized":"2024-03-11T19:56:45.691897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:02.939181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692179+0000","last_change":"2024-03-11T19:55:45.723892+0000","last_active":"2024-03-11T19:57:45.692179+0000","last_peered":"2024-03-11T19:57:45.692179+0000","last_clean":"2024-03-11T19:57:45.692179+0000","last_became_active":"2024-03-11T19:55:45.723544+0000","last_became_peered":"2024-03-11T19:55:45.723544+0000","last_unstale":"2024-03-11T19:57:45.692179+0000","last_undegraded":"2024-03-11T19:57:45.692179+0000","last_fullsized":"2024-03-11T19:57:45.692179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:09:08.720854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T19:56:45.699717+0000","last_change":"2024-03-11T19:56:45.699717+0000","last_active":"2024-03-11T19:56:45.699717+0000","last_peered":"2024-03-11T19:56:45.699717+0000","last_clean":"2024-03-11T19:56:45.699717+0000","last_became_active":"2024-03-11T19:56:45.699346+0000","last_became_peered":"2024-03-11T19:56:45.699346+0000","last_unstale":"2024-03-11T19:56:45.699717+0000","last_undegraded":"2024-03-11T19:56:45.699717+0000","last_fullsized":"2024-03-11T19:56:45.699717+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:52.920371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68567,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":887,"num_read_kb":521,"num_write":244,"num_write_kb":139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":205560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1036288,"data_stored":502327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32868,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712284,"statfs":{"total":95995035648,"available":95961378816,"internally_reserved":0,"allocated":1724416,"data_stored":1149392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1077248,"data_stored":514883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":501080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1650688,"data_stored":1098159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":506291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1097728,"data_stored":559040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":498776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36436,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708716,"statfs":{"total":95995035648,"available":95957725184,"internally_reserved":0,"allocated":1118208,"data_stored":560042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182484,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":496496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":500862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1658880,"data_stored":1093882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:57:49.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T19:57:49.491 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:49.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:49 smithi097 ceph-mon[31898]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:57:49.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:49 smithi097 ceph-mon[31898]: pgmap v385: 129 pgs: 1 activating, 1 peering, 127 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 895 B/s wr, 13 op/s; 14 B/s, 0 objects/s recovering 2024-03-11T19:57:49.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:49 smithi114 ceph-mon[36376]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:57:49.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:49 smithi114 ceph-mon[36376]: pgmap v385: 129 pgs: 1 activating, 1 peering, 127 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 895 B/s wr, 13 op/s; 14 B/s, 0 objects/s recovering 2024-03-11T19:57:50.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:49 smithi161 ceph-mon[36332]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:57:50.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:49 smithi161 ceph-mon[36332]: pgmap v385: 129 pgs: 1 activating, 1 peering, 127 active+clean; 644 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 895 B/s wr, 13 op/s; 14 B/s, 0 objects/s recovering 2024-03-11T19:57:51.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.903+0000 7f830c200700 1 -- 172.21.15.97:0/2523047355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83041091a0 msgr2=0x7f8304110d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:51.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.903+0000 7f830c200700 1 --2- 172.21.15.97:0/2523047355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83041091a0 0x7f8304110d70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f82fc009f90 tx=0x7f82fc00b330 comp rx=0 tx=0).stop 2024-03-11T19:57:51.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.903+0000 7f830c200700 1 -- 172.21.15.97:0/2523047355 shutdown_connections 2024-03-11T19:57:51.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.903+0000 7f830c200700 1 --2- 172.21.15.97:0/2523047355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f83041040c0 0x7f83041044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:51.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.903+0000 7f830c200700 1 --2- 172.21.15.97:0/2523047355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83041091a0 0x7f8304110d70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:51.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.903+0000 7f830c200700 1 --2- 172.21.15.97:0/2523047355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8304104a70 0x7f8304108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:51.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.903+0000 7f830c200700 1 -- 172.21.15.97:0/2523047355 >> 172.21.15.97:0/2523047355 conn(0x7f83040ff950 msgr2=0x7f8304101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:51.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.903+0000 7f830c200700 1 -- 172.21.15.97:0/2523047355 shutdown_connections 2024-03-11T19:57:51.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.903+0000 7f830c200700 1 -- 172.21.15.97:0/2523047355 wait complete. 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.904+0000 7f830c200700 1 Processor -- start 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.905+0000 7f830c200700 1 -- start start 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.905+0000 7f830c200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83041040c0 0x7f830419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.905+0000 7f830c200700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8304104a70 0x7f830419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.905+0000 7f830c200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83041091a0 0x7f830419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.905+0000 7f830c200700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830419f360 con 0x7f8304104a70 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.905+0000 7f830c200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830419f4d0 con 0x7f83041040c0 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.905+0000 7f830c200700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830419f690 con 0x7f83041091a0 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.905+0000 7f8308e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8304104a70 0x7f830419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.905+0000 7f8309800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83041040c0 0x7f830419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.906+0000 7f8309800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83041040c0 0x7f830419a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:37814/0 (socket says 172.21.15.97:37814) 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.906+0000 7f8309800700 1 -- 172.21.15.97:0/2119195221 learned_addr learned my addr 172.21.15.97:0/2119195221 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.906+0000 7f830a200700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83041091a0 0x7f830419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:51.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.906+0000 7f8309800700 1 -- 172.21.15.97:0/2119195221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83041091a0 msgr2=0x7f830419ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.906+0000 7f8309800700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83041091a0 0x7f830419ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.906+0000 7f8309800700 1 -- 172.21.15.97:0/2119195221 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8304104a70 msgr2=0x7f830419a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.906+0000 7f8309800700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8304104a70 0x7f830419a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.906+0000 7f8309800700 1 -- 172.21.15.97:0/2119195221 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc009c40 con 0x7f83041040c0 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.906+0000 7f830a200700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83041091a0 0x7f830419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.906+0000 7f8309800700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83041040c0 0x7f830419a480 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f82f800e9c0 tx=0x7f82f800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.906+0000 7f8302000700 1 -- 172.21.15.97:0/2119195221 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f800cbd0 con 0x7f83041040c0 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.907+0000 7f8302000700 1 -- 172.21.15.97:0/2119195221 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82f800eeb0 con 0x7f83041040c0 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.907+0000 7f830c200700 1 -- 172.21.15.97:0/2119195221 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830419f970 con 0x7f83041040c0 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.907+0000 7f8302000700 1 -- 172.21.15.97:0/2119195221 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8005680 con 0x7f83041040c0 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.907+0000 7f830c200700 1 -- 172.21.15.97:0/2119195221 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83041a71d0 con 0x7f83041040c0 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.908+0000 7f830c200700 1 -- 172.21.15.97:0/2119195221 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8304068930 con 0x7f83041040c0 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.910+0000 7f8302000700 1 -- 172.21.15.97:0/2119195221 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f82f8004750 con 0x7f83041040c0 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.911+0000 7f8302000700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f82ec0a1b40 0x7f82ec0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:51.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.911+0000 7f8302000700 1 -- 172.21.15.97:0/2119195221 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f82f8014070 con 0x7f83041040c0 2024-03-11T19:57:51.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.911+0000 7f8308e00700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f82ec0a1b40 0x7f82ec0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:51.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.912+0000 7f8308e00700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f82ec0a1b40 0x7f82ec0a4000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f830410a6a0 tx=0x7f82f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:51.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:51.913+0000 7f8302000700 1 -- 172.21.15.97:0/2119195221 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f82f80966d0 con 0x7f83041040c0 2024-03-11T19:57:51.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:51 smithi114 ceph-mon[36376]: pgmap v386: 129 pgs: 1 activating, 1 peering, 127 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 895 B/s wr, 8 op/s; 14 B/s, 0 objects/s recovering 2024-03-11T19:57:52.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:51 smithi161 ceph-mon[36332]: pgmap v386: 129 pgs: 1 activating, 1 peering, 127 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 895 B/s wr, 8 op/s; 14 B/s, 0 objects/s recovering 2024-03-11T19:57:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.128+0000 7f830c200700 1 -- 172.21.15.97:0/2119195221 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8304113f50 con 0x7f82ec0a1b40 2024-03-11T19:57:52.128 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:51 smithi097 ceph-mon[31898]: pgmap v386: 129 pgs: 1 activating, 1 peering, 127 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 895 B/s wr, 8 op/s; 14 B/s, 0 objects/s recovering 2024-03-11T19:57:52.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.157+0000 7f8302000700 1 -- 172.21.15.97:0/2119195221 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340653 (secure 0 0 0) 0x7f8304113f50 con 0x7f82ec0a1b40 2024-03-11T19:57:52.164 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:57:52.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.166+0000 7f830c200700 1 -- 172.21.15.97:0/2119195221 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f82ec0a1b40 msgr2=0x7f82ec0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:52.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.166+0000 7f830c200700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f82ec0a1b40 0x7f82ec0a4000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f830410a6a0 tx=0x7f82f0009260 comp rx=0 tx=0).stop 2024-03-11T19:57:52.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.166+0000 7f830c200700 1 -- 172.21.15.97:0/2119195221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83041040c0 msgr2=0x7f830419a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:52.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.166+0000 7f830c200700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83041040c0 0x7f830419a480 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f82f800e9c0 tx=0x7f82f800ecd0 comp rx=0 tx=0).stop 2024-03-11T19:57:52.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.166+0000 7f830c200700 1 -- 172.21.15.97:0/2119195221 shutdown_connections 2024-03-11T19:57:52.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.166+0000 7f830c200700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f82ec0a1b40 0x7f82ec0a4000 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:52.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.166+0000 7f830c200700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83041091a0 0x7f830419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:52.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.166+0000 7f830c200700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83041040c0 0x7f830419a480 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:52.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.166+0000 7f830c200700 1 --2- 172.21.15.97:0/2119195221 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8304104a70 0x7f830419a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:52.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.166+0000 7f830c200700 1 -- 172.21.15.97:0/2119195221 >> 172.21.15.97:0/2119195221 conn(0x7f83040ff950 msgr2=0x7f8304100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:52.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.167+0000 7f830c200700 1 -- 172.21.15.97:0/2119195221 shutdown_connections 2024-03-11T19:57:52.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:52.167+0000 7f830c200700 1 -- 172.21.15.97:0/2119195221 wait complete. 2024-03-11T19:57:52.168 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-11T19:57:53.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:52 smithi161 ceph-mon[36332]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T19:57:53.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:52 smithi161 ceph-mon[36332]: Cluster is now healthy 2024-03-11T19:57:53.035 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:52 smithi097 ceph-mon[31898]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T19:57:53.035 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:52 smithi097 ceph-mon[31898]: Cluster is now healthy 2024-03-11T19:57:53.039 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-03-11T19:57:50.550216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659559,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1060,"num_read_kb":657,"num_write":433,"num_write_kb":3085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1181435,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":856,"ondisk_log_size":856,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384560,"kb_used_data":14552,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557264,"statfs":{"total":1151940427776,"available":1151546638336,"internally_reserved":0,"allocated":14901248,"data_stored":8156935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":28,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002815"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686971+0000","last_change":"2024-03-11T19:55:45.725019+0000","last_active":"2024-03-11T19:57:46.686971+0000","last_peered":"2024-03-11T19:57:46.686971+0000","last_clean":"2024-03-11T19:57:46.686971+0000","last_became_active":"2024-03-11T19:55:45.724602+0000","last_became_peered":"2024-03-11T19:55:45.724602+0000","last_unstale":"2024-03-11T19:57:46.686971+0000","last_undegraded":"2024-03-11T19:57:46.686971+0000","last_fullsized":"2024-03-11T19:57:46.686971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:16:18.218381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881861+0000","last_change":"2024-03-11T19:55:45.724589+0000","last_active":"2024-03-11T19:57:46.881861+0000","last_peered":"2024-03-11T19:57:46.881861+0000","last_clean":"2024-03-11T19:57:46.881861+0000","last_became_active":"2024-03-11T19:55:45.724109+0000","last_became_peered":"2024-03-11T19:55:45.724109+0000","last_unstale":"2024-03-11T19:57:46.881861+0000","last_undegraded":"2024-03-11T19:57:46.881861+0000","last_fullsized":"2024-03-11T19:57:46.881861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:58:12.510437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881733+0000","last_change":"2024-03-11T19:55:45.718831+0000","last_active":"2024-03-11T19:57:46.881733+0000","last_peered":"2024-03-11T19:57:46.881733+0000","last_clean":"2024-03-11T19:57:46.881733+0000","last_became_active":"2024-03-11T19:55:45.717652+0000","last_became_peered":"2024-03-11T19:55:45.717652+0000","last_unstale":"2024-03-11T19:57:46.881733+0000","last_undegraded":"2024-03-11T19:57:46.881733+0000","last_fullsized":"2024-03-11T19:57:46.881733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:16.471754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685116+0000","last_change":"2024-03-11T19:55:45.701640+0000","last_active":"2024-03-11T19:57:46.685116+0000","last_peered":"2024-03-11T19:57:46.685116+0000","last_clean":"2024-03-11T19:57:46.685116+0000","last_became_active":"2024-03-11T19:55:45.699207+0000","last_became_peered":"2024-03-11T19:55:45.699207+0000","last_unstale":"2024-03-11T19:57:46.685116+0000","last_undegraded":"2024-03-11T19:57:46.685116+0000","last_fullsized":"2024-03-11T19:57:46.685116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:01.900429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685627+0000","last_change":"2024-03-11T19:55:45.732040+0000","last_active":"2024-03-11T19:57:46.685627+0000","last_peered":"2024-03-11T19:57:46.685627+0000","last_clean":"2024-03-11T19:57:46.685627+0000","last_became_active":"2024-03-11T19:55:45.731647+0000","last_became_peered":"2024-03-11T19:55:45.731647+0000","last_unstale":"2024-03-11T19:57:46.685627+0000","last_undegraded":"2024-03-11T19:57:46.685627+0000","last_fullsized":"2024-03-11T19:57:46.685627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:51:57.345329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683200+0000","last_change":"2024-03-11T19:55:45.707986+0000","last_active":"2024-03-11T19:57:46.683200+0000","last_peered":"2024-03-11T19:57:46.683200+0000","last_clean":"2024-03-11T19:57:46.683200+0000","last_became_active":"2024-03-11T19:55:45.707641+0000","last_became_peered":"2024-03-11T19:55:45.707641+0000","last_unstale":"2024-03-11T19:57:46.683200+0000","last_undegraded":"2024-03-11T19:57:46.683200+0000","last_fullsized":"2024-03-11T19:57:46.683200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:15:50.635234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683168+0000","last_change":"2024-03-11T19:56:45.705737+0000","last_active":"2024-03-11T19:57:46.683168+0000","last_peered":"2024-03-11T19:57:46.683168+0000","last_clean":"2024-03-11T19:57:46.683168+0000","last_became_active":"2024-03-11T19:56:45.705477+0000","last_became_peered":"2024-03-11T19:56:45.705477+0000","last_unstale":"2024-03-11T19:57:46.683168+0000","last_undegraded":"2024-03-11T19:57:46.683168+0000","last_fullsized":"2024-03-11T19:57:46.683168+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:01:14.068476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572051+0000","last_change":"2024-03-11T19:55:45.729587+0000","last_active":"2024-03-11T19:57:47.572051+0000","last_peered":"2024-03-11T19:57:47.572051+0000","last_clean":"2024-03-11T19:57:47.572051+0000","last_became_active":"2024-03-11T19:55:45.729037+0000","last_became_peered":"2024-03-11T19:55:45.729037+0000","last_unstale":"2024-03-11T19:57:47.572051+0000","last_undegraded":"2024-03-11T19:57:47.572051+0000","last_fullsized":"2024-03-11T19:57:47.572051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:55:43.445192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023626+0000","last_change":"2024-03-11T19:55:45.709793+0000","last_active":"2024-03-11T19:57:47.023626+0000","last_peered":"2024-03-11T19:57:47.023626+0000","last_clean":"2024-03-11T19:57:47.023626+0000","last_became_active":"2024-03-11T19:55:45.708631+0000","last_became_peered":"2024-03-11T19:55:45.708631+0000","last_unstale":"2024-03-11T19:57:47.023626+0000","last_undegraded":"2024-03-11T19:57:47.023626+0000","last_fullsized":"2024-03-11T19:57:47.023626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:02.144066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684105+0000","last_change":"2024-03-11T19:55:45.704227+0000","last_active":"2024-03-11T19:57:46.684105+0000","last_peered":"2024-03-11T19:57:46.684105+0000","last_clean":"2024-03-11T19:57:46.684105+0000","last_became_active":"2024-03-11T19:55:45.703885+0000","last_became_peered":"2024-03-11T19:55:45.703885+0000","last_unstale":"2024-03-11T19:57:46.684105+0000","last_undegraded":"2024-03-11T19:57:46.684105+0000","last_fullsized":"2024-03-11T19:57:46.684105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:36:21.401988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685372+0000","last_change":"2024-03-11T19:55:45.708950+0000","last_active":"2024-03-11T19:57:46.685372+0000","last_peered":"2024-03-11T19:57:46.685372+0000","last_clean":"2024-03-11T19:57:46.685372+0000","last_became_active":"2024-03-11T19:55:45.708544+0000","last_became_peered":"2024-03-11T19:55:45.708544+0000","last_unstale":"2024-03-11T19:57:46.685372+0000","last_undegraded":"2024-03-11T19:57:46.685372+0000","last_fullsized":"2024-03-11T19:57:46.685372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:06:34.244449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685013+0000","last_change":"2024-03-11T19:55:45.703923+0000","last_active":"2024-03-11T19:57:46.685013+0000","last_peered":"2024-03-11T19:57:46.685013+0000","last_clean":"2024-03-11T19:57:46.685013+0000","last_became_active":"2024-03-11T19:55:45.703595+0000","last_became_peered":"2024-03-11T19:55:45.703595+0000","last_unstale":"2024-03-11T19:57:46.685013+0000","last_undegraded":"2024-03-11T19:57:46.685013+0000","last_fullsized":"2024-03-11T19:57:46.685013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:00:19.683907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692145+0000","last_change":"2024-03-11T19:55:45.707499+0000","last_active":"2024-03-11T19:57:45.692145+0000","last_peered":"2024-03-11T19:57:45.692145+0000","last_clean":"2024-03-11T19:57:45.692145+0000","last_became_active":"2024-03-11T19:55:45.706344+0000","last_became_peered":"2024-03-11T19:55:45.706344+0000","last_unstale":"2024-03-11T19:57:45.692145+0000","last_undegraded":"2024-03-11T19:57:45.692145+0000","last_fullsized":"2024-03-11T19:57:45.692145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:38:24.082332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.880311+0000","last_change":"2024-03-11T19:55:45.722182+0000","last_active":"2024-03-11T19:57:46.880311+0000","last_peered":"2024-03-11T19:57:46.880311+0000","last_clean":"2024-03-11T19:57:46.880311+0000","last_became_active":"2024-03-11T19:55:45.721783+0000","last_became_peered":"2024-03-11T19:55:45.721783+0000","last_unstale":"2024-03-11T19:57:46.880311+0000","last_undegraded":"2024-03-11T19:57:46.880311+0000","last_fullsized":"2024-03-11T19:57:46.880311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:41:08.559002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.880277+0000","last_change":"2024-03-11T19:55:45.718881+0000","last_active":"2024-03-11T19:57:46.880277+0000","last_peered":"2024-03-11T19:57:46.880277+0000","last_clean":"2024-03-11T19:57:46.880277+0000","last_became_active":"2024-03-11T19:55:45.718458+0000","last_became_peered":"2024-03-11T19:55:45.718458+0000","last_unstale":"2024-03-11T19:57:46.880277+0000","last_undegraded":"2024-03-11T19:57:46.880277+0000","last_fullsized":"2024-03-11T19:57:46.880277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:22.075892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023120+0000","last_change":"2024-03-11T19:55:45.726163+0000","last_active":"2024-03-11T19:57:47.023120+0000","last_peered":"2024-03-11T19:57:47.023120+0000","last_clean":"2024-03-11T19:57:47.023120+0000","last_became_active":"2024-03-11T19:55:45.725772+0000","last_became_peered":"2024-03-11T19:55:45.725772+0000","last_unstale":"2024-03-11T19:57:47.023120+0000","last_undegraded":"2024-03-11T19:57:47.023120+0000","last_fullsized":"2024-03-11T19:57:47.023120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:36:19.057920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686772+0000","last_change":"2024-03-11T19:55:45.724787+0000","last_active":"2024-03-11T19:57:46.686772+0000","last_peered":"2024-03-11T19:57:46.686772+0000","last_clean":"2024-03-11T19:57:46.686772+0000","last_became_active":"2024-03-11T19:55:45.724437+0000","last_became_peered":"2024-03-11T19:55:45.724437+0000","last_unstale":"2024-03-11T19:57:46.686772+0000","last_undegraded":"2024-03-11T19:57:46.686772+0000","last_fullsized":"2024-03-11T19:57:46.686772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:55:53.811679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.696946+0000","last_change":"2024-03-11T19:55:45.714934+0000","last_active":"2024-03-11T19:57:45.696946+0000","last_peered":"2024-03-11T19:57:45.696946+0000","last_clean":"2024-03-11T19:57:45.696946+0000","last_became_active":"2024-03-11T19:55:45.709595+0000","last_became_peered":"2024-03-11T19:55:45.709595+0000","last_unstale":"2024-03-11T19:57:45.696946+0000","last_undegraded":"2024-03-11T19:57:45.696946+0000","last_fullsized":"2024-03-11T19:57:45.696946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:48.319073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802786+0000","last_change":"2024-03-11T19:55:45.725747+0000","last_active":"2024-03-11T19:57:46.802786+0000","last_peered":"2024-03-11T19:57:46.802786+0000","last_clean":"2024-03-11T19:57:46.802786+0000","last_became_active":"2024-03-11T19:55:45.725239+0000","last_became_peered":"2024-03-11T19:55:45.725239+0000","last_unstale":"2024-03-11T19:57:46.802786+0000","last_undegraded":"2024-03-11T19:57:46.802786+0000","last_fullsized":"2024-03-11T19:57:46.802786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:51:24.046681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802300+0000","last_change":"2024-03-11T19:55:50.109913+0000","last_active":"2024-03-11T19:57:46.802300+0000","last_peered":"2024-03-11T19:57:46.802300+0000","last_clean":"2024-03-11T19:57:46.802300+0000","last_became_active":"2024-03-11T19:55:50.109471+0000","last_became_peered":"2024-03-11T19:55:50.109471+0000","last_unstale":"2024-03-11T19:57:46.802300+0000","last_undegraded":"2024-03-11T19:57:46.802300+0000","last_fullsized":"2024-03-11T19:57:46.802300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:40.258022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":129,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.697208+0000","last_change":"2024-03-11T19:55:44.680181+0000","last_active":"2024-03-11T19:57:45.697208+0000","last_peered":"2024-03-11T19:57:45.697208+0000","last_clean":"2024-03-11T19:57:45.697208+0000","last_became_active":"2024-03-11T19:55:44.679807+0000","last_became_peered":"2024-03-11T19:55:44.679807+0000","last_unstale":"2024-03-11T19:57:45.697208+0000","last_undegraded":"2024-03-11T19:57:45.697208+0000","last_fullsized":"2024-03-11T19:57:45.697208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:23:04.532836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573255+0000","last_change":"2024-03-11T19:55:45.723547+0000","last_active":"2024-03-11T19:57:47.573255+0000","last_peered":"2024-03-11T19:57:47.573255+0000","last_clean":"2024-03-11T19:57:47.573255+0000","last_became_active":"2024-03-11T19:55:45.722516+0000","last_became_peered":"2024-03-11T19:55:45.722516+0000","last_unstale":"2024-03-11T19:57:47.573255+0000","last_undegraded":"2024-03-11T19:57:47.573255+0000","last_fullsized":"2024-03-11T19:57:47.573255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:40.271814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683243+0000","last_change":"2024-03-11T19:55:50.102141+0000","last_active":"2024-03-11T19:57:46.683243+0000","last_peered":"2024-03-11T19:57:46.683243+0000","last_clean":"2024-03-11T19:57:46.683243+0000","last_became_active":"2024-03-11T19:55:50.101667+0000","last_became_peered":"2024-03-11T19:55:50.101667+0000","last_unstale":"2024-03-11T19:57:46.683243+0000","last_undegraded":"2024-03-11T19:57:46.683243+0000","last_fullsized":"2024-03-11T19:57:46.683243+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:54:11.453059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573605+0000","last_change":"2024-03-11T19:55:44.684623+0000","last_active":"2024-03-11T19:57:47.573605+0000","last_peered":"2024-03-11T19:57:47.573605+0000","last_clean":"2024-03-11T19:57:47.573605+0000","last_became_active":"2024-03-11T19:55:44.683611+0000","last_became_peered":"2024-03-11T19:55:44.683611+0000","last_unstale":"2024-03-11T19:57:47.573605+0000","last_undegraded":"2024-03-11T19:57:47.573605+0000","last_fullsized":"2024-03-11T19:57:47.573605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:22:50.954364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881778+0000","last_change":"2024-03-11T19:55:45.723619+0000","last_active":"2024-03-11T19:57:46.881778+0000","last_peered":"2024-03-11T19:57:46.881778+0000","last_clean":"2024-03-11T19:57:46.881778+0000","last_became_active":"2024-03-11T19:55:45.722685+0000","last_became_peered":"2024-03-11T19:55:45.722685+0000","last_unstale":"2024-03-11T19:57:46.881778+0000","last_undegraded":"2024-03-11T19:57:46.881778+0000","last_fullsized":"2024-03-11T19:57:46.881778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:47:13.747776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"activating","last_fresh":"2024-03-11T19:57:46.685468+0000","last_change":"2024-03-11T19:57:46.683796+0000","last_active":"2024-03-11T19:57:45.680845+0000","last_peered":"2024-03-11T19:56:45.693139+0000","last_clean":"2024-03-11T19:56:45.693139+0000","last_became_active":"2024-03-11T19:55:50.102491+0000","last_became_peered":"2024-03-11T19:55:50.102491+0000","last_unstale":"2024-03-11T19:57:46.685468+0000","last_undegraded":"2024-03-11T19:57:46.685468+0000","last_fullsized":"2024-03-11T19:57:46.685468+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":["4(0)","3(1)","5(3)","8(2)","9(2)"],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687185+0000","last_change":"2024-03-11T19:56:45.714194+0000","last_active":"2024-03-11T19:57:46.687185+0000","last_peered":"2024-03-11T19:57:46.687185+0000","last_clean":"2024-03-11T19:57:46.687185+0000","last_became_active":"2024-03-11T19:56:45.694559+0000","last_became_peered":"2024-03-11T19:56:45.694559+0000","last_unstale":"2024-03-11T19:57:46.687185+0000","last_undegraded":"2024-03-11T19:57:46.687185+0000","last_fullsized":"2024-03-11T19:57:46.687185+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:01:20.307915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023254+0000","last_change":"2024-03-11T19:55:45.715157+0000","last_active":"2024-03-11T19:57:47.023254+0000","last_peered":"2024-03-11T19:57:47.023254+0000","last_clean":"2024-03-11T19:57:47.023254+0000","last_became_active":"2024-03-11T19:55:45.714536+0000","last_became_peered":"2024-03-11T19:55:45.714536+0000","last_unstale":"2024-03-11T19:57:47.023254+0000","last_undegraded":"2024-03-11T19:57:47.023254+0000","last_fullsized":"2024-03-11T19:57:47.023254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:02.416736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.682631+0000","last_change":"2024-03-11T19:55:50.102157+0000","last_active":"2024-03-11T19:57:46.682631+0000","last_peered":"2024-03-11T19:57:46.682631+0000","last_clean":"2024-03-11T19:57:46.682631+0000","last_became_active":"2024-03-11T19:55:50.101715+0000","last_became_peered":"2024-03-11T19:55:50.101715+0000","last_unstale":"2024-03-11T19:57:46.682631+0000","last_undegraded":"2024-03-11T19:57:46.682631+0000","last_fullsized":"2024-03-11T19:57:46.682631+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:12:18.780708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687126+0000","last_change":"2024-03-11T19:56:45.698585+0000","last_active":"2024-03-11T19:57:46.687126+0000","last_peered":"2024-03-11T19:57:46.687126+0000","last_clean":"2024-03-11T19:57:46.687126+0000","last_became_active":"2024-03-11T19:56:45.692929+0000","last_became_peered":"2024-03-11T19:56:45.692929+0000","last_unstale":"2024-03-11T19:57:46.687126+0000","last_undegraded":"2024-03-11T19:57:46.687126+0000","last_fullsized":"2024-03-11T19:57:46.687126+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:43:11.158934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.686907+0000","last_change":"2024-03-11T19:55:45.709973+0000","last_active":"2024-03-11T19:57:45.686907+0000","last_peered":"2024-03-11T19:57:45.686907+0000","last_clean":"2024-03-11T19:57:45.686907+0000","last_became_active":"2024-03-11T19:55:45.709645+0000","last_became_peered":"2024-03-11T19:55:45.709645+0000","last_unstale":"2024-03-11T19:57:45.686907+0000","last_undegraded":"2024-03-11T19:57:45.686907+0000","last_fullsized":"2024-03-11T19:57:45.686907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:16.516982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802538+0000","last_change":"2024-03-11T19:55:50.094093+0000","last_active":"2024-03-11T19:57:46.802538+0000","last_peered":"2024-03-11T19:57:46.802538+0000","last_clean":"2024-03-11T19:57:46.802538+0000","last_became_active":"2024-03-11T19:55:50.093653+0000","last_became_peered":"2024-03-11T19:55:50.093653+0000","last_unstale":"2024-03-11T19:57:46.802538+0000","last_undegraded":"2024-03-11T19:57:46.802538+0000","last_fullsized":"2024-03-11T19:57:46.802538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:28:41.668001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":127,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.686915+0000","last_change":"2024-03-11T19:55:44.671547+0000","last_active":"2024-03-11T19:57:45.686915+0000","last_peered":"2024-03-11T19:57:45.686915+0000","last_clean":"2024-03-11T19:57:45.686915+0000","last_became_active":"2024-03-11T19:55:44.671211+0000","last_became_peered":"2024-03-11T19:55:44.671211+0000","last_unstale":"2024-03-11T19:57:45.686915+0000","last_undegraded":"2024-03-11T19:57:45.686915+0000","last_fullsized":"2024-03-11T19:57:45.686915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:18:46.261250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685626+0000","last_change":"2024-03-11T19:55:45.707968+0000","last_active":"2024-03-11T19:57:46.685626+0000","last_peered":"2024-03-11T19:57:46.685626+0000","last_clean":"2024-03-11T19:57:46.685626+0000","last_became_active":"2024-03-11T19:55:45.706998+0000","last_became_peered":"2024-03-11T19:55:45.706998+0000","last_unstale":"2024-03-11T19:57:46.685626+0000","last_undegraded":"2024-03-11T19:57:46.685626+0000","last_fullsized":"2024-03-11T19:57:46.685626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:49:35.711123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802348+0000","last_change":"2024-03-11T19:55:50.104310+0000","last_active":"2024-03-11T19:57:46.802348+0000","last_peered":"2024-03-11T19:57:46.802348+0000","last_clean":"2024-03-11T19:57:46.802348+0000","last_became_active":"2024-03-11T19:55:50.103831+0000","last_became_peered":"2024-03-11T19:55:50.103831+0000","last_unstale":"2024-03-11T19:57:46.802348+0000","last_undegraded":"2024-03-11T19:57:46.802348+0000","last_fullsized":"2024-03-11T19:57:46.802348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:33.213987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683083+0000","last_change":"2024-03-11T19:55:44.677615+0000","last_active":"2024-03-11T19:57:46.683083+0000","last_peered":"2024-03-11T19:57:46.683083+0000","last_clean":"2024-03-11T19:57:46.683083+0000","last_became_active":"2024-03-11T19:55:44.677153+0000","last_became_peered":"2024-03-11T19:55:44.677153+0000","last_unstale":"2024-03-11T19:57:46.683083+0000","last_undegraded":"2024-03-11T19:57:46.683083+0000","last_fullsized":"2024-03-11T19:57:46.683083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:52:59.153222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687012+0000","last_change":"2024-03-11T19:55:45.731827+0000","last_active":"2024-03-11T19:57:45.687012+0000","last_peered":"2024-03-11T19:57:45.687012+0000","last_clean":"2024-03-11T19:57:45.687012+0000","last_became_active":"2024-03-11T19:55:45.731460+0000","last_became_peered":"2024-03-11T19:55:45.731460+0000","last_unstale":"2024-03-11T19:57:45.687012+0000","last_undegraded":"2024-03-11T19:57:45.687012+0000","last_fullsized":"2024-03-11T19:57:45.687012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:12.277198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686597+0000","last_change":"2024-03-11T19:55:50.109812+0000","last_active":"2024-03-11T19:57:46.686597+0000","last_peered":"2024-03-11T19:57:46.686597+0000","last_clean":"2024-03-11T19:57:46.686597+0000","last_became_active":"2024-03-11T19:55:50.109420+0000","last_became_peered":"2024-03-11T19:55:50.109420+0000","last_unstale":"2024-03-11T19:57:46.686597+0000","last_undegraded":"2024-03-11T19:57:46.686597+0000","last_fullsized":"2024-03-11T19:57:46.686597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:08:30.105941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.691625+0000","last_change":"2024-03-11T19:55:44.683173+0000","last_active":"2024-03-11T19:57:45.691625+0000","last_peered":"2024-03-11T19:57:45.691625+0000","last_clean":"2024-03-11T19:57:45.691625+0000","last_became_active":"2024-03-11T19:55:44.682778+0000","last_became_peered":"2024-03-11T19:55:44.682778+0000","last_unstale":"2024-03-11T19:57:45.691625+0000","last_undegraded":"2024-03-11T19:57:45.691625+0000","last_fullsized":"2024-03-11T19:57:45.691625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:15.119864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572517+0000","last_change":"2024-03-11T19:55:45.721833+0000","last_active":"2024-03-11T19:57:47.572517+0000","last_peered":"2024-03-11T19:57:47.572517+0000","last_clean":"2024-03-11T19:57:47.572517+0000","last_became_active":"2024-03-11T19:55:45.721490+0000","last_became_peered":"2024-03-11T19:55:45.721490+0000","last_unstale":"2024-03-11T19:57:47.572517+0000","last_undegraded":"2024-03-11T19:57:47.572517+0000","last_fullsized":"2024-03-11T19:57:47.572517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:09.824912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881479+0000","last_change":"2024-03-11T19:55:50.108424+0000","last_active":"2024-03-11T19:57:46.881479+0000","last_peered":"2024-03-11T19:57:46.881479+0000","last_clean":"2024-03-11T19:57:46.881479+0000","last_became_active":"2024-03-11T19:55:50.107072+0000","last_became_peered":"2024-03-11T19:55:50.107072+0000","last_unstale":"2024-03-11T19:57:46.881479+0000","last_undegraded":"2024-03-11T19:57:46.881479+0000","last_fullsized":"2024-03-11T19:57:46.881479+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:31.751597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.686954+0000","last_change":"2024-03-11T19:55:44.671570+0000","last_active":"2024-03-11T19:57:45.686954+0000","last_peered":"2024-03-11T19:57:45.686954+0000","last_clean":"2024-03-11T19:57:45.686954+0000","last_became_active":"2024-03-11T19:55:44.671007+0000","last_became_peered":"2024-03-11T19:55:44.671007+0000","last_unstale":"2024-03-11T19:57:45.686954+0000","last_undegraded":"2024-03-11T19:57:45.686954+0000","last_fullsized":"2024-03-11T19:57:45.686954+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:23.865011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.695987+0000","last_change":"2024-03-11T19:55:45.733048+0000","last_active":"2024-03-11T19:57:45.695987+0000","last_peered":"2024-03-11T19:57:45.695987+0000","last_clean":"2024-03-11T19:57:45.695987+0000","last_became_active":"2024-03-11T19:55:45.731656+0000","last_became_peered":"2024-03-11T19:55:45.731656+0000","last_unstale":"2024-03-11T19:57:45.695987+0000","last_undegraded":"2024-03-11T19:57:45.695987+0000","last_fullsized":"2024-03-11T19:57:45.695987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:28:56.433327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572169+0000","last_change":"2024-03-11T19:55:50.105532+0000","last_active":"2024-03-11T19:57:47.572169+0000","last_peered":"2024-03-11T19:57:47.572169+0000","last_clean":"2024-03-11T19:57:47.572169+0000","last_became_active":"2024-03-11T19:55:50.102200+0000","last_became_peered":"2024-03-11T19:55:50.102200+0000","last_unstale":"2024-03-11T19:57:47.572169+0000","last_undegraded":"2024-03-11T19:57:47.572169+0000","last_fullsized":"2024-03-11T19:57:47.572169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:37.060472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":214,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.765984+0000","last_change":"2024-03-11T19:55:44.678224+0000","last_active":"2024-03-11T19:57:46.765984+0000","last_peered":"2024-03-11T19:57:46.765984+0000","last_clean":"2024-03-11T19:57:46.765984+0000","last_became_active":"2024-03-11T19:55:44.677936+0000","last_became_peered":"2024-03-11T19:55:44.677936+0000","last_unstale":"2024-03-11T19:57:46.765984+0000","last_undegraded":"2024-03-11T19:57:46.765984+0000","last_fullsized":"2024-03-11T19:57:46.765984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:20:42.486501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":178,"num_read_kb":178,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023625+0000","last_change":"2024-03-11T19:55:45.709917+0000","last_active":"2024-03-11T19:57:47.023625+0000","last_peered":"2024-03-11T19:57:47.023625+0000","last_clean":"2024-03-11T19:57:47.023625+0000","last_became_active":"2024-03-11T19:55:45.709318+0000","last_became_peered":"2024-03-11T19:55:45.709318+0000","last_unstale":"2024-03-11T19:57:47.023625+0000","last_undegraded":"2024-03-11T19:57:47.023625+0000","last_fullsized":"2024-03-11T19:57:47.023625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:08.298158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.883428+0000","last_change":"2024-03-11T19:57:46.882937+0000","last_active":"2024-03-11T19:57:46.883428+0000","last_peered":"2024-03-11T19:57:46.883428+0000","last_clean":"2024-03-11T19:57:46.883428+0000","last_became_active":"2024-03-11T19:57:46.882408+0000","last_became_peered":"2024-03-11T19:57:46.882408+0000","last_unstale":"2024-03-11T19:57:46.883428+0000","last_undegraded":"2024-03-11T19:57:46.883428+0000","last_fullsized":"2024-03-11T19:57:46.883428+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:58:27.842775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685774+0000","last_change":"2024-03-11T19:55:44.676004+0000","last_active":"2024-03-11T19:57:46.685774+0000","last_peered":"2024-03-11T19:57:46.685774+0000","last_clean":"2024-03-11T19:57:46.685774+0000","last_became_active":"2024-03-11T19:55:44.675640+0000","last_became_peered":"2024-03-11T19:55:44.675640+0000","last_unstale":"2024-03-11T19:57:46.685774+0000","last_undegraded":"2024-03-11T19:57:46.685774+0000","last_fullsized":"2024-03-11T19:57:46.685774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:15.096780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881928+0000","last_change":"2024-03-11T19:56:45.702569+0000","last_active":"2024-03-11T19:57:46.881928+0000","last_peered":"2024-03-11T19:57:46.881928+0000","last_clean":"2024-03-11T19:57:46.881928+0000","last_became_active":"2024-03-11T19:56:45.701890+0000","last_became_peered":"2024-03-11T19:56:45.701890+0000","last_unstale":"2024-03-11T19:57:46.881928+0000","last_undegraded":"2024-03-11T19:57:46.881928+0000","last_fullsized":"2024-03-11T19:57:46.881928+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:40.434712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684317+0000","last_change":"2024-03-11T19:55:50.096136+0000","last_active":"2024-03-11T19:57:46.684317+0000","last_peered":"2024-03-11T19:57:46.684317+0000","last_clean":"2024-03-11T19:57:46.684317+0000","last_became_active":"2024-03-11T19:55:50.095670+0000","last_became_peered":"2024-03-11T19:55:50.095670+0000","last_unstale":"2024-03-11T19:57:46.684317+0000","last_undegraded":"2024-03-11T19:57:46.684317+0000","last_fullsized":"2024-03-11T19:57:46.684317+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:40.943273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.686791+0000","last_change":"2024-03-11T19:55:44.681547+0000","last_active":"2024-03-11T19:57:45.686791+0000","last_peered":"2024-03-11T19:57:45.686791+0000","last_clean":"2024-03-11T19:57:45.686791+0000","last_became_active":"2024-03-11T19:55:44.681237+0000","last_became_peered":"2024-03-11T19:55:44.681237+0000","last_unstale":"2024-03-11T19:57:45.686791+0000","last_undegraded":"2024-03-11T19:57:45.686791+0000","last_fullsized":"2024-03-11T19:57:45.686791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:08:09.509915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023380+0000","last_change":"2024-03-11T19:55:45.705025+0000","last_active":"2024-03-11T19:57:47.023380+0000","last_peered":"2024-03-11T19:57:47.023380+0000","last_clean":"2024-03-11T19:57:47.023380+0000","last_became_active":"2024-03-11T19:55:45.702680+0000","last_became_peered":"2024-03-11T19:55:45.702680+0000","last_unstale":"2024-03-11T19:57:47.023380+0000","last_undegraded":"2024-03-11T19:57:47.023380+0000","last_fullsized":"2024-03-11T19:57:47.023380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:28.245002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.695555+0000","last_change":"2024-03-11T19:55:50.099648+0000","last_active":"2024-03-11T19:57:45.695555+0000","last_peered":"2024-03-11T19:57:45.695555+0000","last_clean":"2024-03-11T19:57:45.695555+0000","last_became_active":"2024-03-11T19:55:50.098516+0000","last_became_peered":"2024-03-11T19:55:50.098516+0000","last_unstale":"2024-03-11T19:57:45.695555+0000","last_undegraded":"2024-03-11T19:57:45.695555+0000","last_fullsized":"2024-03-11T19:57:45.695555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:19.644046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":250,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.765992+0000","last_change":"2024-03-11T19:55:44.672460+0000","last_active":"2024-03-11T19:57:46.765992+0000","last_peered":"2024-03-11T19:57:46.765992+0000","last_clean":"2024-03-11T19:57:46.765992+0000","last_became_active":"2024-03-11T19:55:44.672123+0000","last_became_peered":"2024-03-11T19:55:44.672123+0000","last_unstale":"2024-03-11T19:57:46.765992+0000","last_undegraded":"2024-03-11T19:57:46.765992+0000","last_fullsized":"2024-03-11T19:57:46.765992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:43.940638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61471,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":185,"num_read_kb":153,"num_write":37,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685900+0000","last_change":"2024-03-11T19:55:45.702933+0000","last_active":"2024-03-11T19:57:46.685900+0000","last_peered":"2024-03-11T19:57:46.685900+0000","last_clean":"2024-03-11T19:57:46.685900+0000","last_became_active":"2024-03-11T19:55:45.702585+0000","last_became_peered":"2024-03-11T19:55:45.702585+0000","last_unstale":"2024-03-11T19:57:46.685900+0000","last_undegraded":"2024-03-11T19:57:46.685900+0000","last_fullsized":"2024-03-11T19:57:46.685900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:54:38.762156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687748+0000","last_change":"2024-03-11T19:55:50.102571+0000","last_active":"2024-03-11T19:57:45.687748+0000","last_peered":"2024-03-11T19:57:45.687748+0000","last_clean":"2024-03-11T19:57:45.687748+0000","last_became_active":"2024-03-11T19:55:50.102099+0000","last_became_peered":"2024-03-11T19:55:50.102099+0000","last_unstale":"2024-03-11T19:57:45.687748+0000","last_undegraded":"2024-03-11T19:57:45.687748+0000","last_fullsized":"2024-03-11T19:57:45.687748+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:17.631378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802401+0000","last_change":"2024-03-11T19:55:44.671027+0000","last_active":"2024-03-11T19:57:46.802401+0000","last_peered":"2024-03-11T19:57:46.802401+0000","last_clean":"2024-03-11T19:57:46.802401+0000","last_became_active":"2024-03-11T19:55:44.669696+0000","last_became_peered":"2024-03-11T19:55:44.669696+0000","last_unstale":"2024-03-11T19:57:46.802401+0000","last_undegraded":"2024-03-11T19:57:46.802401+0000","last_fullsized":"2024-03-11T19:57:46.802401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:48:20.169638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.691595+0000","last_change":"2024-03-11T19:55:45.729119+0000","last_active":"2024-03-11T19:57:45.691595+0000","last_peered":"2024-03-11T19:57:45.691595+0000","last_clean":"2024-03-11T19:57:45.691595+0000","last_became_active":"2024-03-11T19:55:45.727442+0000","last_became_peered":"2024-03-11T19:55:45.727442+0000","last_unstale":"2024-03-11T19:57:45.691595+0000","last_undegraded":"2024-03-11T19:57:45.691595+0000","last_fullsized":"2024-03-11T19:57:45.691595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:14:51.483199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686945+0000","last_change":"2024-03-11T19:55:50.104326+0000","last_active":"2024-03-11T19:57:46.686945+0000","last_peered":"2024-03-11T19:57:46.686945+0000","last_clean":"2024-03-11T19:57:46.686945+0000","last_became_active":"2024-03-11T19:55:50.103881+0000","last_became_peered":"2024-03-11T19:55:50.103881+0000","last_unstale":"2024-03-11T19:57:46.686945+0000","last_undegraded":"2024-03-11T19:57:46.686945+0000","last_fullsized":"2024-03-11T19:57:46.686945+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:12:38.819591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.686622+0000","last_change":"2024-03-11T19:55:44.684900+0000","last_active":"2024-03-11T19:57:45.686622+0000","last_peered":"2024-03-11T19:57:45.686622+0000","last_clean":"2024-03-11T19:57:45.686622+0000","last_became_active":"2024-03-11T19:55:44.684590+0000","last_became_peered":"2024-03-11T19:55:44.684590+0000","last_unstale":"2024-03-11T19:57:45.686622+0000","last_undegraded":"2024-03-11T19:57:45.686622+0000","last_fullsized":"2024-03-11T19:57:45.686622+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:07.744366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684928+0000","last_change":"2024-03-11T19:55:45.704787+0000","last_active":"2024-03-11T19:57:46.684928+0000","last_peered":"2024-03-11T19:57:46.684928+0000","last_clean":"2024-03-11T19:57:46.684928+0000","last_became_active":"2024-03-11T19:55:45.704405+0000","last_became_peered":"2024-03-11T19:55:45.704405+0000","last_unstale":"2024-03-11T19:57:46.684928+0000","last_undegraded":"2024-03-11T19:57:46.684928+0000","last_fullsized":"2024-03-11T19:57:46.684928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:58:05.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.697029+0000","last_change":"2024-03-11T19:55:50.104793+0000","last_active":"2024-03-11T19:57:45.697029+0000","last_peered":"2024-03-11T19:57:45.697029+0000","last_clean":"2024-03-11T19:57:45.697029+0000","last_became_active":"2024-03-11T19:55:50.104415+0000","last_became_peered":"2024-03-11T19:55:50.104415+0000","last_unstale":"2024-03-11T19:57:45.697029+0000","last_undegraded":"2024-03-11T19:57:45.697029+0000","last_fullsized":"2024-03-11T19:57:45.697029+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:53.307313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":126,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683177+0000","last_change":"2024-03-11T19:55:44.669403+0000","last_active":"2024-03-11T19:57:46.683177+0000","last_peered":"2024-03-11T19:57:46.683177+0000","last_clean":"2024-03-11T19:57:46.683177+0000","last_became_active":"2024-03-11T19:55:44.668908+0000","last_became_peered":"2024-03-11T19:55:44.668908+0000","last_unstale":"2024-03-11T19:57:46.683177+0000","last_undegraded":"2024-03-11T19:57:46.683177+0000","last_fullsized":"2024-03-11T19:57:46.683177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:56:36.831506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":92,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684333+0000","last_change":"2024-03-11T19:55:45.700029+0000","last_active":"2024-03-11T19:57:46.684333+0000","last_peered":"2024-03-11T19:57:46.684333+0000","last_clean":"2024-03-11T19:57:46.684333+0000","last_became_active":"2024-03-11T19:55:45.697296+0000","last_became_peered":"2024-03-11T19:55:45.697296+0000","last_unstale":"2024-03-11T19:57:46.684333+0000","last_undegraded":"2024-03-11T19:57:46.684333+0000","last_fullsized":"2024-03-11T19:57:46.684333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:59:35.025845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"peering","last_fresh":"2024-03-11T19:56:45.700253+0000","last_change":"2024-03-11T19:57:45.687743+0000","last_active":"2024-03-11T19:57:45.687743+0000","last_peered":"2024-03-11T19:56:45.700253+0000","last_clean":"2024-03-11T19:56:45.700253+0000","last_became_active":"2024-03-11T19:55:50.108182+0000","last_became_peered":"2024-03-11T19:55:50.108182+0000","last_unstale":"2024-03-11T19:56:45.700253+0000","last_undegraded":"2024-03-11T19:56:45.700253+0000","last_fullsized":"2024-03-11T19:56:45.700253+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,8,9,11],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685341+0000","last_change":"2024-03-11T19:55:44.677491+0000","last_active":"2024-03-11T19:57:46.685341+0000","last_peered":"2024-03-11T19:57:46.685341+0000","last_clean":"2024-03-11T19:57:46.685341+0000","last_became_active":"2024-03-11T19:55:44.677176+0000","last_became_peered":"2024-03-11T19:55:44.677176+0000","last_unstale":"2024-03-11T19:57:46.685341+0000","last_undegraded":"2024-03-11T19:57:46.685341+0000","last_fullsized":"2024-03-11T19:57:46.685341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:51.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687518+0000","last_change":"2024-03-11T19:55:45.703310+0000","last_active":"2024-03-11T19:57:45.687518+0000","last_peered":"2024-03-11T19:57:45.687518+0000","last_clean":"2024-03-11T19:57:45.687518+0000","last_became_active":"2024-03-11T19:55:45.702933+0000","last_became_peered":"2024-03-11T19:55:45.702933+0000","last_unstale":"2024-03-11T19:57:45.687518+0000","last_undegraded":"2024-03-11T19:57:45.687518+0000","last_fullsized":"2024-03-11T19:57:45.687518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:07:58.787949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687431+0000","last_change":"2024-03-11T19:55:50.103849+0000","last_active":"2024-03-11T19:57:45.687431+0000","last_peered":"2024-03-11T19:57:45.687431+0000","last_clean":"2024-03-11T19:57:45.687431+0000","last_became_active":"2024-03-11T19:55:50.103360+0000","last_became_peered":"2024-03-11T19:55:50.103360+0000","last_unstale":"2024-03-11T19:57:45.687431+0000","last_undegraded":"2024-03-11T19:57:45.687431+0000","last_fullsized":"2024-03-11T19:57:45.687431+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:38:05.151041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685858+0000","last_change":"2024-03-11T19:55:44.669839+0000","last_active":"2024-03-11T19:57:46.685858+0000","last_peered":"2024-03-11T19:57:46.685858+0000","last_clean":"2024-03-11T19:57:46.685858+0000","last_became_active":"2024-03-11T19:55:44.669494+0000","last_became_peered":"2024-03-11T19:55:44.669494+0000","last_unstale":"2024-03-11T19:57:46.685858+0000","last_undegraded":"2024-03-11T19:57:46.685858+0000","last_fullsized":"2024-03-11T19:57:46.685858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:00.313847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687005+0000","last_change":"2024-03-11T19:56:45.693864+0000","last_active":"2024-03-11T19:57:46.687005+0000","last_peered":"2024-03-11T19:57:46.687005+0000","last_clean":"2024-03-11T19:57:46.687005+0000","last_became_active":"2024-03-11T19:56:45.693091+0000","last_became_peered":"2024-03-11T19:56:45.693091+0000","last_unstale":"2024-03-11T19:57:46.687005+0000","last_undegraded":"2024-03-11T19:57:46.687005+0000","last_fullsized":"2024-03-11T19:57:46.687005+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:26:37.233485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685431+0000","last_change":"2024-03-11T19:55:50.143916+0000","last_active":"2024-03-11T19:57:46.685431+0000","last_peered":"2024-03-11T19:57:46.685431+0000","last_clean":"2024-03-11T19:57:46.685431+0000","last_became_active":"2024-03-11T19:55:50.143341+0000","last_became_peered":"2024-03-11T19:55:50.143341+0000","last_unstale":"2024-03-11T19:57:46.685431+0000","last_undegraded":"2024-03-11T19:57:46.685431+0000","last_fullsized":"2024-03-11T19:57:46.685431+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:40.637529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.696901+0000","last_change":"2024-03-11T19:55:44.688468+0000","last_active":"2024-03-11T19:57:45.696901+0000","last_peered":"2024-03-11T19:57:45.696901+0000","last_clean":"2024-03-11T19:57:45.696901+0000","last_became_active":"2024-03-11T19:55:44.688065+0000","last_became_peered":"2024-03-11T19:55:44.688065+0000","last_unstale":"2024-03-11T19:57:45.696901+0000","last_undegraded":"2024-03-11T19:57:45.696901+0000","last_fullsized":"2024-03-11T19:57:45.696901+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:29:23.392749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683888+0000","last_change":"2024-03-11T19:55:45.728970+0000","last_active":"2024-03-11T19:57:46.683888+0000","last_peered":"2024-03-11T19:57:46.683888+0000","last_clean":"2024-03-11T19:57:46.683888+0000","last_became_active":"2024-03-11T19:55:45.728516+0000","last_became_peered":"2024-03-11T19:55:45.728516+0000","last_unstale":"2024-03-11T19:57:46.683888+0000","last_undegraded":"2024-03-11T19:57:46.683888+0000","last_fullsized":"2024-03-11T19:57:46.683888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:19.242610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692260+0000","last_change":"2024-03-11T19:55:50.111884+0000","last_active":"2024-03-11T19:57:45.692260+0000","last_peered":"2024-03-11T19:57:45.692260+0000","last_clean":"2024-03-11T19:57:45.692260+0000","last_became_active":"2024-03-11T19:55:50.110802+0000","last_became_peered":"2024-03-11T19:55:50.110802+0000","last_unstale":"2024-03-11T19:57:45.692260+0000","last_undegraded":"2024-03-11T19:57:45.692260+0000","last_fullsized":"2024-03-11T19:57:45.692260+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:35.866368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":125,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684613+0000","last_change":"2024-03-11T19:55:44.677156+0000","last_active":"2024-03-11T19:57:46.684613+0000","last_peered":"2024-03-11T19:57:46.684613+0000","last_clean":"2024-03-11T19:57:46.684613+0000","last_became_active":"2024-03-11T19:55:44.676656+0000","last_became_peered":"2024-03-11T19:55:44.676656+0000","last_unstale":"2024-03-11T19:57:46.684613+0000","last_undegraded":"2024-03-11T19:57:46.684613+0000","last_fullsized":"2024-03-11T19:57:46.684613+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:40:27.521669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683177+0000","last_change":"2024-03-11T19:55:45.715545+0000","last_active":"2024-03-11T19:57:46.683177+0000","last_peered":"2024-03-11T19:57:46.683177+0000","last_clean":"2024-03-11T19:57:46.683177+0000","last_became_active":"2024-03-11T19:55:45.715042+0000","last_became_peered":"2024-03-11T19:55:45.715042+0000","last_unstale":"2024-03-11T19:57:46.683177+0000","last_undegraded":"2024-03-11T19:57:46.683177+0000","last_fullsized":"2024-03-11T19:57:46.683177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:06:33.362235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692138+0000","last_change":"2024-03-11T19:55:50.145603+0000","last_active":"2024-03-11T19:57:45.692138+0000","last_peered":"2024-03-11T19:57:45.692138+0000","last_clean":"2024-03-11T19:57:45.692138+0000","last_became_active":"2024-03-11T19:55:50.143689+0000","last_became_peered":"2024-03-11T19:55:50.143689+0000","last_unstale":"2024-03-11T19:57:45.692138+0000","last_undegraded":"2024-03-11T19:57:45.692138+0000","last_fullsized":"2024-03-11T19:57:45.692138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:57.289983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687250+0000","last_change":"2024-03-11T19:55:45.702011+0000","last_active":"2024-03-11T19:57:45.687250+0000","last_peered":"2024-03-11T19:57:45.687250+0000","last_clean":"2024-03-11T19:57:45.687250+0000","last_became_active":"2024-03-11T19:55:45.701673+0000","last_became_peered":"2024-03-11T19:55:45.701673+0000","last_unstale":"2024-03-11T19:57:45.687250+0000","last_undegraded":"2024-03-11T19:57:45.687250+0000","last_fullsized":"2024-03-11T19:57:45.687250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:17.028276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":123,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.691567+0000","last_change":"2024-03-11T19:55:44.689121+0000","last_active":"2024-03-11T19:57:45.691567+0000","last_peered":"2024-03-11T19:57:45.691567+0000","last_clean":"2024-03-11T19:57:45.691567+0000","last_became_active":"2024-03-11T19:55:44.688453+0000","last_became_peered":"2024-03-11T19:55:44.688453+0000","last_unstale":"2024-03-11T19:57:45.691567+0000","last_undegraded":"2024-03-11T19:57:45.691567+0000","last_fullsized":"2024-03-11T19:57:45.691567+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:31.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":89,"num_read_kb":89,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.698213+0000","last_change":"2024-03-11T19:57:46.698213+0000","last_active":"2024-03-11T19:57:46.698213+0000","last_peered":"2024-03-11T19:57:46.698213+0000","last_clean":"2024-03-11T19:57:46.698213+0000","last_became_active":"2024-03-11T19:57:46.693710+0000","last_became_peered":"2024-03-11T19:57:46.693710+0000","last_unstale":"2024-03-11T19:57:46.698213+0000","last_undegraded":"2024-03-11T19:57:46.698213+0000","last_fullsized":"2024-03-11T19:57:46.698213+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:06:00.214092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.696088+0000","last_change":"2024-03-11T19:56:45.703746+0000","last_active":"2024-03-11T19:57:45.696088+0000","last_peered":"2024-03-11T19:57:45.696088+0000","last_clean":"2024-03-11T19:57:45.696088+0000","last_became_active":"2024-03-11T19:56:45.701252+0000","last_became_peered":"2024-03-11T19:56:45.701252+0000","last_unstale":"2024-03-11T19:57:45.696088+0000","last_undegraded":"2024-03-11T19:57:45.696088+0000","last_fullsized":"2024-03-11T19:57:45.696088+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:16:50.219593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684191+0000","last_change":"2024-03-11T19:55:44.672644+0000","last_active":"2024-03-11T19:57:46.684191+0000","last_peered":"2024-03-11T19:57:46.684191+0000","last_clean":"2024-03-11T19:57:46.684191+0000","last_became_active":"2024-03-11T19:55:44.671645+0000","last_became_peered":"2024-03-11T19:55:44.671645+0000","last_unstale":"2024-03-11T19:57:46.684191+0000","last_undegraded":"2024-03-11T19:57:46.684191+0000","last_fullsized":"2024-03-11T19:57:46.684191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:25:46.637768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":367,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684193+0000","last_change":"2024-03-11T19:54:20.915891+0000","last_active":"2024-03-11T19:57:46.684193+0000","last_peered":"2024-03-11T19:57:46.684193+0000","last_clean":"2024-03-11T19:57:46.684193+0000","last_became_active":"2024-03-11T19:54:20.915450+0000","last_became_peered":"2024-03-11T19:54:20.915450+0000","last_unstale":"2024-03-11T19:57:46.684193+0000","last_undegraded":"2024-03-11T19:57:46.684193+0000","last_fullsized":"2024-03-11T19:57:46.684193+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_clean_scrub_stamp":"2024-03-11T19:50:29.052858+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:47.966076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687164+0000","last_change":"2024-03-11T19:55:50.144053+0000","last_active":"2024-03-11T19:57:45.687164+0000","last_peered":"2024-03-11T19:57:45.687164+0000","last_clean":"2024-03-11T19:57:45.687164+0000","last_became_active":"2024-03-11T19:55:50.143509+0000","last_became_peered":"2024-03-11T19:55:50.143509+0000","last_unstale":"2024-03-11T19:57:45.687164+0000","last_undegraded":"2024-03-11T19:57:45.687164+0000","last_fullsized":"2024-03-11T19:57:45.687164+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:57:15.448812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.691409+0000","last_change":"2024-03-11T19:55:45.712621+0000","last_active":"2024-03-11T19:57:45.691409+0000","last_peered":"2024-03-11T19:57:45.691409+0000","last_clean":"2024-03-11T19:57:45.691409+0000","last_became_active":"2024-03-11T19:55:45.712161+0000","last_became_peered":"2024-03-11T19:55:45.712161+0000","last_unstale":"2024-03-11T19:57:45.691409+0000","last_undegraded":"2024-03-11T19:57:45.691409+0000","last_fullsized":"2024-03-11T19:57:45.691409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:34:52.911944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684355+0000","last_change":"2024-03-11T19:55:44.674606+0000","last_active":"2024-03-11T19:57:46.684355+0000","last_peered":"2024-03-11T19:57:46.684355+0000","last_clean":"2024-03-11T19:57:46.684355+0000","last_became_active":"2024-03-11T19:55:44.674355+0000","last_became_peered":"2024-03-11T19:55:44.674355+0000","last_unstale":"2024-03-11T19:57:46.684355+0000","last_undegraded":"2024-03-11T19:57:46.684355+0000","last_fullsized":"2024-03-11T19:57:46.684355+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:31:01.662582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686690+0000","last_change":"2024-03-11T19:55:50.101468+0000","last_active":"2024-03-11T19:57:46.686690+0000","last_peered":"2024-03-11T19:57:46.686690+0000","last_clean":"2024-03-11T19:57:46.686690+0000","last_became_active":"2024-03-11T19:55:50.101040+0000","last_became_peered":"2024-03-11T19:55:50.101040+0000","last_unstale":"2024-03-11T19:57:46.686690+0000","last_undegraded":"2024-03-11T19:57:46.686690+0000","last_fullsized":"2024-03-11T19:57:46.686690+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:55.354176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685982+0000","last_change":"2024-03-11T19:55:44.681467+0000","last_active":"2024-03-11T19:57:46.685982+0000","last_peered":"2024-03-11T19:57:46.685982+0000","last_clean":"2024-03-11T19:57:46.685982+0000","last_became_active":"2024-03-11T19:55:44.681176+0000","last_became_peered":"2024-03-11T19:55:44.681176+0000","last_unstale":"2024-03-11T19:57:46.685982+0000","last_undegraded":"2024-03-11T19:57:46.685982+0000","last_fullsized":"2024-03-11T19:57:46.685982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:27:38.019116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692424+0000","last_change":"2024-03-11T19:55:45.709798+0000","last_active":"2024-03-11T19:57:45.692424+0000","last_peered":"2024-03-11T19:57:45.692424+0000","last_clean":"2024-03-11T19:57:45.692424+0000","last_became_active":"2024-03-11T19:55:45.709277+0000","last_became_peered":"2024-03-11T19:55:45.709277+0000","last_unstale":"2024-03-11T19:57:45.692424+0000","last_undegraded":"2024-03-11T19:57:45.692424+0000","last_fullsized":"2024-03-11T19:57:45.692424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:58.852316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572588+0000","last_change":"2024-03-11T19:55:50.146601+0000","last_active":"2024-03-11T19:57:47.572588+0000","last_peered":"2024-03-11T19:57:47.572588+0000","last_clean":"2024-03-11T19:57:47.572588+0000","last_became_active":"2024-03-11T19:55:50.144584+0000","last_became_peered":"2024-03-11T19:55:50.144584+0000","last_unstale":"2024-03-11T19:57:47.572588+0000","last_undegraded":"2024-03-11T19:57:47.572588+0000","last_fullsized":"2024-03-11T19:57:47.572588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:00.365713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'14","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.696915+0000","last_change":"2024-03-11T19:55:44.689109+0000","last_active":"2024-03-11T19:57:45.696915+0000","last_peered":"2024-03-11T19:57:45.696915+0000","last_clean":"2024-03-11T19:57:45.696915+0000","last_became_active":"2024-03-11T19:55:44.687323+0000","last_became_peered":"2024-03-11T19:55:44.687323+0000","last_unstale":"2024-03-11T19:57:45.696915+0000","last_undegraded":"2024-03-11T19:57:45.696915+0000","last_fullsized":"2024-03-11T19:57:45.696915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:24.613901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573714+0000","last_change":"2024-03-11T19:55:45.726033+0000","last_active":"2024-03-11T19:57:47.573714+0000","last_peered":"2024-03-11T19:57:47.573714+0000","last_clean":"2024-03-11T19:57:47.573714+0000","last_became_active":"2024-03-11T19:55:45.725353+0000","last_became_peered":"2024-03-11T19:55:45.725353+0000","last_unstale":"2024-03-11T19:57:47.573714+0000","last_undegraded":"2024-03-11T19:57:47.573714+0000","last_fullsized":"2024-03-11T19:57:47.573714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:36:24.341870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802087+0000","last_change":"2024-03-11T19:55:50.089496+0000","last_active":"2024-03-11T19:57:46.802087+0000","last_peered":"2024-03-11T19:57:46.802087+0000","last_clean":"2024-03-11T19:57:46.802087+0000","last_became_active":"2024-03-11T19:55:50.089072+0000","last_became_peered":"2024-03-11T19:55:50.089072+0000","last_unstale":"2024-03-11T19:57:46.802087+0000","last_undegraded":"2024-03-11T19:57:46.802087+0000","last_fullsized":"2024-03-11T19:57:46.802087+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:27:32.539333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684876+0000","last_change":"2024-03-11T19:55:44.673147+0000","last_active":"2024-03-11T19:57:46.684876+0000","last_peered":"2024-03-11T19:57:46.684876+0000","last_clean":"2024-03-11T19:57:46.684876+0000","last_became_active":"2024-03-11T19:55:44.672787+0000","last_became_peered":"2024-03-11T19:55:44.672787+0000","last_unstale":"2024-03-11T19:57:46.684876+0000","last_undegraded":"2024-03-11T19:57:46.684876+0000","last_fullsized":"2024-03-11T19:57:46.684876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:11:23.864419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881937+0000","last_change":"2024-03-11T19:55:45.732376+0000","last_active":"2024-03-11T19:57:46.881937+0000","last_peered":"2024-03-11T19:57:46.881937+0000","last_clean":"2024-03-11T19:57:46.881937+0000","last_became_active":"2024-03-11T19:55:45.731956+0000","last_became_peered":"2024-03-11T19:55:45.731956+0000","last_unstale":"2024-03-11T19:57:46.881937+0000","last_undegraded":"2024-03-11T19:57:46.881937+0000","last_fullsized":"2024-03-11T19:57:46.881937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:43:48.423678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684605+0000","last_change":"2024-03-11T19:55:50.144080+0000","last_active":"2024-03-11T19:57:46.684605+0000","last_peered":"2024-03-11T19:57:46.684605+0000","last_clean":"2024-03-11T19:57:46.684605+0000","last_became_active":"2024-03-11T19:55:50.143585+0000","last_became_peered":"2024-03-11T19:55:50.143585+0000","last_unstale":"2024-03-11T19:57:46.684605+0000","last_undegraded":"2024-03-11T19:57:46.684605+0000","last_fullsized":"2024-03-11T19:57:46.684605+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:47:51.388823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023090+0000","last_change":"2024-03-11T19:55:44.677354+0000","last_active":"2024-03-11T19:57:47.023090+0000","last_peered":"2024-03-11T19:57:47.023090+0000","last_clean":"2024-03-11T19:57:47.023090+0000","last_became_active":"2024-03-11T19:55:44.677013+0000","last_became_peered":"2024-03-11T19:55:44.677013+0000","last_unstale":"2024-03-11T19:57:47.023090+0000","last_undegraded":"2024-03-11T19:57:47.023090+0000","last_fullsized":"2024-03-11T19:57:47.023090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:50:48.855320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685353+0000","last_change":"2024-03-11T19:55:45.702278+0000","last_active":"2024-03-11T19:57:46.685353+0000","last_peered":"2024-03-11T19:57:46.685353+0000","last_clean":"2024-03-11T19:57:46.685353+0000","last_became_active":"2024-03-11T19:55:45.701843+0000","last_became_peered":"2024-03-11T19:55:45.701843+0000","last_unstale":"2024-03-11T19:57:46.685353+0000","last_undegraded":"2024-03-11T19:57:46.685353+0000","last_fullsized":"2024-03-11T19:57:46.685353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:55:28.351093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.697109+0000","last_change":"2024-03-11T19:55:50.106083+0000","last_active":"2024-03-11T19:57:45.697109+0000","last_peered":"2024-03-11T19:57:45.697109+0000","last_clean":"2024-03-11T19:57:45.697109+0000","last_became_active":"2024-03-11T19:55:50.105591+0000","last_became_peered":"2024-03-11T19:55:50.105591+0000","last_unstale":"2024-03-11T19:57:45.697109+0000","last_undegraded":"2024-03-11T19:57:45.697109+0000","last_fullsized":"2024-03-11T19:57:45.697109+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:34:58.713672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572978+0000","last_change":"2024-03-11T19:55:44.688128+0000","last_active":"2024-03-11T19:57:47.572978+0000","last_peered":"2024-03-11T19:57:47.572978+0000","last_clean":"2024-03-11T19:57:47.572978+0000","last_became_active":"2024-03-11T19:55:44.684165+0000","last_became_peered":"2024-03-11T19:55:44.684165+0000","last_unstale":"2024-03-11T19:57:47.572978+0000","last_undegraded":"2024-03-11T19:57:47.572978+0000","last_fullsized":"2024-03-11T19:57:47.572978+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:37:19.415960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686707+0000","last_change":"2024-03-11T19:55:45.718968+0000","last_active":"2024-03-11T19:57:46.686707+0000","last_peered":"2024-03-11T19:57:46.686707+0000","last_clean":"2024-03-11T19:57:46.686707+0000","last_became_active":"2024-03-11T19:55:45.718625+0000","last_became_peered":"2024-03-11T19:55:45.718625+0000","last_unstale":"2024-03-11T19:57:46.686707+0000","last_undegraded":"2024-03-11T19:57:46.686707+0000","last_fullsized":"2024-03-11T19:57:46.686707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:27.197227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684464+0000","last_change":"2024-03-11T19:55:50.097617+0000","last_active":"2024-03-11T19:57:46.684464+0000","last_peered":"2024-03-11T19:57:46.684464+0000","last_clean":"2024-03-11T19:57:46.684464+0000","last_became_active":"2024-03-11T19:55:50.097137+0000","last_became_peered":"2024-03-11T19:55:50.097137+0000","last_unstale":"2024-03-11T19:57:46.684464+0000","last_undegraded":"2024-03-11T19:57:46.684464+0000","last_fullsized":"2024-03-11T19:57:46.684464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:19:20.995137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802186+0000","last_change":"2024-03-11T19:55:44.669944+0000","last_active":"2024-03-11T19:57:46.802186+0000","last_peered":"2024-03-11T19:57:46.802186+0000","last_clean":"2024-03-11T19:57:46.802186+0000","last_became_active":"2024-03-11T19:55:44.669309+0000","last_became_peered":"2024-03-11T19:55:44.669309+0000","last_unstale":"2024-03-11T19:57:46.802186+0000","last_undegraded":"2024-03-11T19:57:46.802186+0000","last_fullsized":"2024-03-11T19:57:46.802186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:06:41.489579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.574440+0000","last_change":"2024-03-11T19:55:45.731348+0000","last_active":"2024-03-11T19:57:47.574440+0000","last_peered":"2024-03-11T19:57:47.574440+0000","last_clean":"2024-03-11T19:57:47.574440+0000","last_became_active":"2024-03-11T19:55:45.730957+0000","last_became_peered":"2024-03-11T19:55:45.730957+0000","last_unstale":"2024-03-11T19:57:47.574440+0000","last_undegraded":"2024-03-11T19:57:47.574440+0000","last_fullsized":"2024-03-11T19:57:47.574440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:06:50.422511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684428+0000","last_change":"2024-03-11T19:55:50.144050+0000","last_active":"2024-03-11T19:57:46.684428+0000","last_peered":"2024-03-11T19:57:46.684428+0000","last_clean":"2024-03-11T19:57:46.684428+0000","last_became_active":"2024-03-11T19:55:50.143591+0000","last_became_peered":"2024-03-11T19:55:50.143591+0000","last_unstale":"2024-03-11T19:57:46.684428+0000","last_undegraded":"2024-03-11T19:57:46.684428+0000","last_fullsized":"2024-03-11T19:57:46.684428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:02.006665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685506+0000","last_change":"2024-03-11T19:55:44.671973+0000","last_active":"2024-03-11T19:57:46.685506+0000","last_peered":"2024-03-11T19:57:46.685506+0000","last_clean":"2024-03-11T19:57:46.685506+0000","last_became_active":"2024-03-11T19:55:44.671661+0000","last_became_peered":"2024-03-11T19:55:44.671661+0000","last_unstale":"2024-03-11T19:57:46.685506+0000","last_undegraded":"2024-03-11T19:57:46.685506+0000","last_fullsized":"2024-03-11T19:57:46.685506+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:35.747386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684764+0000","last_change":"2024-03-11T19:55:45.705545+0000","last_active":"2024-03-11T19:57:46.684764+0000","last_peered":"2024-03-11T19:57:46.684764+0000","last_clean":"2024-03-11T19:57:46.684764+0000","last_became_active":"2024-03-11T19:55:45.705083+0000","last_became_peered":"2024-03-11T19:55:45.705083+0000","last_unstale":"2024-03-11T19:57:46.684764+0000","last_undegraded":"2024-03-11T19:57:46.684764+0000","last_fullsized":"2024-03-11T19:57:46.684764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:02:58.219026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.022984+0000","last_change":"2024-03-11T19:55:50.097951+0000","last_active":"2024-03-11T19:57:47.022984+0000","last_peered":"2024-03-11T19:57:47.022984+0000","last_clean":"2024-03-11T19:57:47.022984+0000","last_became_active":"2024-03-11T19:55:50.097446+0000","last_became_peered":"2024-03-11T19:55:50.097446+0000","last_unstale":"2024-03-11T19:57:47.022984+0000","last_undegraded":"2024-03-11T19:57:47.022984+0000","last_fullsized":"2024-03-11T19:57:47.022984+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:12.105801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'7","reported_seq":131,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802616+0000","last_change":"2024-03-11T19:55:44.673136+0000","last_active":"2024-03-11T19:57:46.802616+0000","last_peered":"2024-03-11T19:57:46.802616+0000","last_clean":"2024-03-11T19:57:46.802616+0000","last_became_active":"2024-03-11T19:55:44.672467+0000","last_became_peered":"2024-03-11T19:55:44.672467+0000","last_unstale":"2024-03-11T19:57:46.802616+0000","last_undegraded":"2024-03-11T19:57:46.802616+0000","last_fullsized":"2024-03-11T19:57:46.802616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:46:05.931253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":93,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685595+0000","last_change":"2024-03-11T19:55:45.710561+0000","last_active":"2024-03-11T19:57:46.685595+0000","last_peered":"2024-03-11T19:57:46.685595+0000","last_clean":"2024-03-11T19:57:46.685595+0000","last_became_active":"2024-03-11T19:55:45.710164+0000","last_became_peered":"2024-03-11T19:55:45.710164+0000","last_unstale":"2024-03-11T19:57:46.685595+0000","last_undegraded":"2024-03-11T19:57:46.685595+0000","last_fullsized":"2024-03-11T19:57:46.685595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:00:33.977595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.880197+0000","last_change":"2024-03-11T19:55:50.113390+0000","last_active":"2024-03-11T19:57:46.880197+0000","last_peered":"2024-03-11T19:57:46.880197+0000","last_clean":"2024-03-11T19:57:46.880197+0000","last_became_active":"2024-03-11T19:55:50.112656+0000","last_became_peered":"2024-03-11T19:55:50.112656+0000","last_unstale":"2024-03-11T19:57:46.880197+0000","last_undegraded":"2024-03-11T19:57:46.880197+0000","last_fullsized":"2024-03-11T19:57:46.880197+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:11.952585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687033+0000","last_change":"2024-03-11T19:55:44.684907+0000","last_active":"2024-03-11T19:57:45.687033+0000","last_peered":"2024-03-11T19:57:45.687033+0000","last_clean":"2024-03-11T19:57:45.687033+0000","last_became_active":"2024-03-11T19:55:44.684600+0000","last_became_peered":"2024-03-11T19:55:44.684600+0000","last_unstale":"2024-03-11T19:57:45.687033+0000","last_undegraded":"2024-03-11T19:57:45.687033+0000","last_fullsized":"2024-03-11T19:57:45.687033+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:12:33.666756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685169+0000","last_change":"2024-03-11T19:55:45.723431+0000","last_active":"2024-03-11T19:57:46.685169+0000","last_peered":"2024-03-11T19:57:46.685169+0000","last_clean":"2024-03-11T19:57:46.685169+0000","last_became_active":"2024-03-11T19:55:45.723135+0000","last_became_peered":"2024-03-11T19:55:45.723135+0000","last_unstale":"2024-03-11T19:57:46.685169+0000","last_undegraded":"2024-03-11T19:57:46.685169+0000","last_fullsized":"2024-03-11T19:57:46.685169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:12:35.654094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.686759+0000","last_change":"2024-03-11T19:55:50.092876+0000","last_active":"2024-03-11T19:57:45.686759+0000","last_peered":"2024-03-11T19:57:45.686759+0000","last_clean":"2024-03-11T19:57:45.686759+0000","last_became_active":"2024-03-11T19:55:50.092478+0000","last_became_peered":"2024-03-11T19:55:50.092478+0000","last_unstale":"2024-03-11T19:57:45.686759+0000","last_undegraded":"2024-03-11T19:57:45.686759+0000","last_fullsized":"2024-03-11T19:57:45.686759+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:39.248325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692817+0000","last_change":"2024-03-11T19:56:45.711269+0000","last_active":"2024-03-11T19:57:45.692817+0000","last_peered":"2024-03-11T19:57:45.692817+0000","last_clean":"2024-03-11T19:57:45.692817+0000","last_became_active":"2024-03-11T19:56:45.704657+0000","last_became_peered":"2024-03-11T19:56:45.704657+0000","last_unstale":"2024-03-11T19:57:45.692817+0000","last_undegraded":"2024-03-11T19:57:45.692817+0000","last_fullsized":"2024-03-11T19:57:45.692817+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:51.549965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685910+0000","last_change":"2024-03-11T19:55:45.702924+0000","last_active":"2024-03-11T19:57:46.685910+0000","last_peered":"2024-03-11T19:57:46.685910+0000","last_clean":"2024-03-11T19:57:46.685910+0000","last_became_active":"2024-03-11T19:55:45.702530+0000","last_became_peered":"2024-03-11T19:55:45.702530+0000","last_unstale":"2024-03-11T19:57:46.685910+0000","last_undegraded":"2024-03-11T19:57:46.685910+0000","last_fullsized":"2024-03-11T19:57:46.685910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:14.679964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686739+0000","last_change":"2024-03-11T19:55:45.706123+0000","last_active":"2024-03-11T19:57:46.686739+0000","last_peered":"2024-03-11T19:57:46.686739+0000","last_clean":"2024-03-11T19:57:46.686739+0000","last_became_active":"2024-03-11T19:55:45.705706+0000","last_became_peered":"2024-03-11T19:55:45.705706+0000","last_unstale":"2024-03-11T19:57:46.686739+0000","last_undegraded":"2024-03-11T19:57:46.686739+0000","last_fullsized":"2024-03-11T19:57:46.686739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:42:58.989823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802576+0000","last_change":"2024-03-11T19:55:45.703188+0000","last_active":"2024-03-11T19:57:46.802576+0000","last_peered":"2024-03-11T19:57:46.802576+0000","last_clean":"2024-03-11T19:57:46.802576+0000","last_became_active":"2024-03-11T19:55:45.702803+0000","last_became_peered":"2024-03-11T19:55:45.702803+0000","last_unstale":"2024-03-11T19:57:46.802576+0000","last_undegraded":"2024-03-11T19:57:46.802576+0000","last_fullsized":"2024-03-11T19:57:46.802576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:20.922963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802163+0000","last_change":"2024-03-11T19:55:45.728856+0000","last_active":"2024-03-11T19:57:46.802163+0000","last_peered":"2024-03-11T19:57:46.802163+0000","last_clean":"2024-03-11T19:57:46.802163+0000","last_became_active":"2024-03-11T19:55:45.728526+0000","last_became_peered":"2024-03-11T19:55:45.728526+0000","last_unstale":"2024-03-11T19:57:46.802163+0000","last_undegraded":"2024-03-11T19:57:46.802163+0000","last_fullsized":"2024-03-11T19:57:46.802163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:25:30.836073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881301+0000","last_change":"2024-03-11T19:55:45.725075+0000","last_active":"2024-03-11T19:57:46.881301+0000","last_peered":"2024-03-11T19:57:46.881301+0000","last_clean":"2024-03-11T19:57:46.881301+0000","last_became_active":"2024-03-11T19:55:45.724627+0000","last_became_peered":"2024-03-11T19:55:45.724627+0000","last_unstale":"2024-03-11T19:57:46.881301+0000","last_undegraded":"2024-03-11T19:57:46.881301+0000","last_fullsized":"2024-03-11T19:57:46.881301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:01.039660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692385+0000","last_change":"2024-03-11T19:55:45.718879+0000","last_active":"2024-03-11T19:57:45.692385+0000","last_peered":"2024-03-11T19:57:45.692385+0000","last_clean":"2024-03-11T19:57:45.692385+0000","last_became_active":"2024-03-11T19:55:45.718535+0000","last_became_peered":"2024-03-11T19:55:45.718535+0000","last_unstale":"2024-03-11T19:57:45.692385+0000","last_undegraded":"2024-03-11T19:57:45.692385+0000","last_fullsized":"2024-03-11T19:57:45.692385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:48.236368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687310+0000","last_change":"2024-03-11T19:55:45.703432+0000","last_active":"2024-03-11T19:57:46.687310+0000","last_peered":"2024-03-11T19:57:46.687310+0000","last_clean":"2024-03-11T19:57:46.687310+0000","last_became_active":"2024-03-11T19:55:45.702669+0000","last_became_peered":"2024-03-11T19:55:45.702669+0000","last_unstale":"2024-03-11T19:57:46.687310+0000","last_undegraded":"2024-03-11T19:57:46.687310+0000","last_fullsized":"2024-03-11T19:57:46.687310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:02.496981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802680+0000","last_change":"2024-03-11T19:55:45.708733+0000","last_active":"2024-03-11T19:57:46.802680+0000","last_peered":"2024-03-11T19:57:46.802680+0000","last_clean":"2024-03-11T19:57:46.802680+0000","last_became_active":"2024-03-11T19:55:45.708369+0000","last_became_peered":"2024-03-11T19:55:45.708369+0000","last_unstale":"2024-03-11T19:57:46.802680+0000","last_undegraded":"2024-03-11T19:57:46.802680+0000","last_fullsized":"2024-03-11T19:57:46.802680+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:36.619802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573994+0000","last_change":"2024-03-11T19:55:45.725698+0000","last_active":"2024-03-11T19:57:47.573994+0000","last_peered":"2024-03-11T19:57:47.573994+0000","last_clean":"2024-03-11T19:57:47.573994+0000","last_became_active":"2024-03-11T19:55:45.724872+0000","last_became_peered":"2024-03-11T19:55:45.724872+0000","last_unstale":"2024-03-11T19:57:47.573994+0000","last_undegraded":"2024-03-11T19:57:47.573994+0000","last_fullsized":"2024-03-11T19:57:47.573994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.982969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.687079+0000","last_change":"2024-03-11T19:55:45.726923+0000","last_active":"2024-03-11T19:57:45.687079+0000","last_peered":"2024-03-11T19:57:45.687079+0000","last_clean":"2024-03-11T19:57:45.687079+0000","last_became_active":"2024-03-11T19:55:45.726589+0000","last_became_peered":"2024-03-11T19:55:45.726589+0000","last_unstale":"2024-03-11T19:57:45.687079+0000","last_undegraded":"2024-03-11T19:57:45.687079+0000","last_fullsized":"2024-03-11T19:57:45.687079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:29:15.197359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.695652+0000","last_change":"2024-03-11T19:56:45.698623+0000","last_active":"2024-03-11T19:57:45.695652+0000","last_peered":"2024-03-11T19:57:45.695652+0000","last_clean":"2024-03-11T19:57:45.695652+0000","last_became_active":"2024-03-11T19:56:45.697805+0000","last_became_peered":"2024-03-11T19:56:45.697805+0000","last_unstale":"2024-03-11T19:57:45.695652+0000","last_undegraded":"2024-03-11T19:57:45.695652+0000","last_fullsized":"2024-03-11T19:57:45.695652+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:16:31.624856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023161+0000","last_change":"2024-03-11T19:55:45.729141+0000","last_active":"2024-03-11T19:57:47.023161+0000","last_peered":"2024-03-11T19:57:47.023161+0000","last_clean":"2024-03-11T19:57:47.023161+0000","last_became_active":"2024-03-11T19:55:45.728781+0000","last_became_peered":"2024-03-11T19:55:45.728781+0000","last_unstale":"2024-03-11T19:57:47.023161+0000","last_undegraded":"2024-03-11T19:57:47.023161+0000","last_fullsized":"2024-03-11T19:57:47.023161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:02.939181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-11T19:57:45.692179+0000","last_change":"2024-03-11T19:55:45.723892+0000","last_active":"2024-03-11T19:57:45.692179+0000","last_peered":"2024-03-11T19:57:45.692179+0000","last_clean":"2024-03-11T19:57:45.692179+0000","last_became_active":"2024-03-11T19:55:45.723544+0000","last_became_peered":"2024-03-11T19:55:45.723544+0000","last_unstale":"2024-03-11T19:57:45.692179+0000","last_undegraded":"2024-03-11T19:57:45.692179+0000","last_fullsized":"2024-03-11T19:57:45.692179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:09:08.720854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687440+0000","last_change":"2024-03-11T19:56:45.699717+0000","last_active":"2024-03-11T19:57:46.687440+0000","last_peered":"2024-03-11T19:57:46.687440+0000","last_clean":"2024-03-11T19:57:46.687440+0000","last_became_active":"2024-03-11T19:56:45.699346+0000","last_became_peered":"2024-03-11T19:56:45.699346+0000","last_unstale":"2024-03-11T19:57:46.687440+0000","last_undegraded":"2024-03-11T19:57:46.687440+0000","last_fullsized":"2024-03-11T19:57:46.687440+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:52.920371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68957,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":924,"num_read_kb":538,"num_write":250,"num_write_kb":145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":206871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":217,"ondisk_log_size":217,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1069056,"data_stored":521390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645356,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32904,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712248,"statfs":{"total":95995035648,"available":95961341952,"internally_reserved":0,"allocated":1761280,"data_stored":1172659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1077248,"data_stored":514883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":520002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36984,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708168,"statfs":{"total":95995035648,"available":95957164032,"internally_reserved":0,"allocated":1683456,"data_stored":1117222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":506291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1130496,"data_stored":578211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":517698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36464,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708688,"statfs":{"total":95995035648,"available":95957696512,"internally_reserved":0,"allocated":1150976,"data_stored":579495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":515418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":519784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1658880,"data_stored":1093882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:57:53.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T19:57:53.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:52 smithi114 ceph-mon[36376]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T19:57:53.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:52 smithi114 ceph-mon[36376]: Cluster is now healthy 2024-03-11T19:57:53.540 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:54.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:53 smithi161 ceph-mon[36332]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:57:54.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:53 smithi161 ceph-mon[36332]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 895 B/s wr, 12 op/s; 14 B/s, 0 objects/s recovering 2024-03-11T19:57:54.057 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:53 smithi097 ceph-mon[31898]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:57:54.057 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:53 smithi097 ceph-mon[31898]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 895 B/s wr, 12 op/s; 14 B/s, 0 objects/s recovering 2024-03-11T19:57:54.241 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:53 smithi114 ceph-mon[36376]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:57:54.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:53 smithi114 ceph-mon[36376]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 895 B/s wr, 12 op/s; 14 B/s, 0 objects/s recovering 2024-03-11T19:57:55.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.874+0000 7fdba2000700 1 -- 172.21.15.97:0/3398275925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb9c107cc0 msgr2=0x7fdb9c1080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:55.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.874+0000 7fdba2000700 1 --2- 172.21.15.97:0/3398275925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb9c107cc0 0x7fdb9c1080a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c009990 tx=0x7fdb8c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:57:55.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.874+0000 7fdba2000700 1 -- 172.21.15.97:0/3398275925 shutdown_connections 2024-03-11T19:57:55.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.874+0000 7fdba2000700 1 --2- 172.21.15.97:0/3398275925 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb9c06a2c0 0x7fdb9c10eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:55.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.874+0000 7fdba2000700 1 --2- 172.21.15.97:0/3398275925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb9c107cc0 0x7fdb9c1080a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:55.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.874+0000 7fdba2000700 1 --2- 172.21.15.97:0/3398275925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb9c069920 0x7fdb9c069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:55.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.874+0000 7fdba2000700 1 -- 172.21.15.97:0/3398275925 >> 172.21.15.97:0/3398275925 conn(0x7fdb9c077aa0 msgr2=0x7fdb9c077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:55.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.874+0000 7fdba2000700 1 -- 172.21.15.97:0/3398275925 shutdown_connections 2024-03-11T19:57:55.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.874+0000 7fdba2000700 1 -- 172.21.15.97:0/3398275925 wait complete. 2024-03-11T19:57:55.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.875+0000 7fdba2000700 1 Processor -- start 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.875+0000 7fdba2000700 1 -- start start 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.876+0000 7fdba2000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb9c069920 0x7fdb9c19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.876+0000 7fdba2000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb9c06a2c0 0x7fdb9c19a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.876+0000 7fdba2000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb9c107cc0 0x7fdb9c19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.876+0000 7fdba2000700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c19f200 con 0x7fdb9c107cc0 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.876+0000 7fdba2000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c19f370 con 0x7fdb9c06a2c0 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.876+0000 7fdba2000700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c19f530 con 0x7fdb9c069920 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.876+0000 7fdb9be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb9c107cc0 0x7fdb9c19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.876+0000 7fdb93e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb9c06a2c0 0x7fdb9c19a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.876+0000 7fdb9b400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb9c069920 0x7fdb9c19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.876+0000 7fdb9be00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb9c107cc0 0x7fdb9c19ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39362/0 (socket says 172.21.15.97:39362) 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.876+0000 7fdb9be00700 1 -- 172.21.15.97:0/2421426149 learned_addr learned my addr 172.21.15.97:0/2421426149 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.877+0000 7fdb9b400700 1 -- 172.21.15.97:0/2421426149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb9c06a2c0 msgr2=0x7fdb9c19a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.877+0000 7fdb9b400700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb9c06a2c0 0x7fdb9c19a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:55.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.877+0000 7fdb9b400700 1 -- 172.21.15.97:0/2421426149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb9c107cc0 msgr2=0x7fdb9c19ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:55.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.877+0000 7fdb9b400700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb9c107cc0 0x7fdb9c19ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:55.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.877+0000 7fdb9b400700 1 -- 172.21.15.97:0/2421426149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c0095f0 con 0x7fdb9c069920 2024-03-11T19:57:55.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.877+0000 7fdb93e00700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb9c06a2c0 0x7fdb9c19a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:55.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.877+0000 7fdb9be00700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb9c107cc0 0x7fdb9c19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.877+0000 7fdb9b400700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb9c069920 0x7fdb9c19a320 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c006010 tx=0x7fdb8c005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.877+0000 7fdb98c00700 1 -- 172.21.15.97:0/2421426149 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c01d070 con 0x7fdb9c069920 2024-03-11T19:57:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.877+0000 7fdb98c00700 1 -- 172.21.15.97:0/2421426149 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb8c00b610 con 0x7fdb9c069920 2024-03-11T19:57:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.877+0000 7fdb98c00700 1 -- 172.21.15.97:0/2421426149 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c022e50 con 0x7fdb9c069920 2024-03-11T19:57:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.877+0000 7fdba2000700 1 -- 172.21.15.97:0/2421426149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9c19f7b0 con 0x7fdb9c069920 2024-03-11T19:57:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.878+0000 7fdba2000700 1 -- 172.21.15.97:0/2421426149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9c1a6ea0 con 0x7fdb9c069920 2024-03-11T19:57:55.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.879+0000 7fdba2000700 1 -- 172.21.15.97:0/2421426149 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb9c068930 con 0x7fdb9c069920 2024-03-11T19:57:55.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.880+0000 7fdb98c00700 1 -- 172.21.15.97:0/2421426149 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fdb8c022500 con 0x7fdb9c069920 2024-03-11T19:57:55.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.882+0000 7fdb98c00700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdb800a1b40 0x7fdb800a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:55.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.882+0000 7fdb98c00700 1 -- 172.21.15.97:0/2421426149 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fdb8c0cea30 con 0x7fdb9c069920 2024-03-11T19:57:55.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.882+0000 7fdb93e00700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdb800a1b40 0x7fdb800a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:55.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.883+0000 7fdb93e00700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdb800a1b40 0x7fdb800a4000 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c10ef40 tx=0x7fdb84009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:55.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:55.884+0000 7fdb98c00700 1 -- 172.21.15.97:0/2421426149 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdb8c0978b0 con 0x7fdb9c069920 2024-03-11T19:57:56.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:55 smithi161 ceph-mon[36332]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 807 B/s wr, 12 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T19:57:56.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.098+0000 7fdba2000700 1 -- 172.21.15.97:0/2421426149 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdb9c111dc0 con 0x7fdb800a1b40 2024-03-11T19:57:56.097 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:55 smithi097 ceph-mon[31898]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 807 B/s wr, 12 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T19:57:56.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.109+0000 7fdb98c00700 1 -- 172.21.15.97:0/2421426149 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340582 (secure 0 0 0) 0x7fdb9c111dc0 con 0x7fdb800a1b40 2024-03-11T19:57:56.113 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:57:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.114+0000 7fdba2000700 1 -- 172.21.15.97:0/2421426149 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdb800a1b40 msgr2=0x7fdb800a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.114+0000 7fdba2000700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdb800a1b40 0x7fdb800a4000 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c10ef40 tx=0x7fdb84009260 comp rx=0 tx=0).stop 2024-03-11T19:57:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.114+0000 7fdba2000700 1 -- 172.21.15.97:0/2421426149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb9c069920 msgr2=0x7fdb9c19a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.114+0000 7fdba2000700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb9c069920 0x7fdb9c19a320 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c006010 tx=0x7fdb8c005740 comp rx=0 tx=0).stop 2024-03-11T19:57:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.114+0000 7fdba2000700 1 -- 172.21.15.97:0/2421426149 shutdown_connections 2024-03-11T19:57:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.114+0000 7fdba2000700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fdb800a1b40 0x7fdb800a4000 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.114+0000 7fdba2000700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb9c107cc0 0x7fdb9c19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.114+0000 7fdba2000700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb9c069920 0x7fdb9c19a320 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.114+0000 7fdba2000700 1 --2- 172.21.15.97:0/2421426149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb9c06a2c0 0x7fdb9c19a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.114+0000 7fdba2000700 1 -- 172.21.15.97:0/2421426149 >> 172.21.15.97:0/2421426149 conn(0x7fdb9c077aa0 msgr2=0x7fdb9c0ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.115+0000 7fdba2000700 1 -- 172.21.15.97:0/2421426149 shutdown_connections 2024-03-11T19:57:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:56.115+0000 7fdba2000700 1 -- 172.21.15.97:0/2421426149 wait complete. 2024-03-11T19:57:56.118 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-11T19:57:56.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:55 smithi114 ceph-mon[36376]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 807 B/s wr, 12 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T19:57:56.960 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-03-11T19:57:54.552277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659559,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1110,"num_read_kb":682,"num_write":433,"num_write_kb":3085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1181435,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":856,"ondisk_log_size":856,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384628,"kb_used_data":14612,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557196,"statfs":{"total":1151940427776,"available":1151546568704,"internally_reserved":0,"allocated":14962688,"data_stored":8190599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":53,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.876981"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687052+0000","last_change":"2024-03-11T19:55:45.725019+0000","last_active":"2024-03-11T19:57:46.687052+0000","last_peered":"2024-03-11T19:57:46.687052+0000","last_clean":"2024-03-11T19:57:46.687052+0000","last_became_active":"2024-03-11T19:55:45.724602+0000","last_became_peered":"2024-03-11T19:55:45.724602+0000","last_unstale":"2024-03-11T19:57:46.687052+0000","last_undegraded":"2024-03-11T19:57:46.687052+0000","last_fullsized":"2024-03-11T19:57:46.687052+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:16:18.218381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881861+0000","last_change":"2024-03-11T19:55:45.724589+0000","last_active":"2024-03-11T19:57:46.881861+0000","last_peered":"2024-03-11T19:57:46.881861+0000","last_clean":"2024-03-11T19:57:46.881861+0000","last_became_active":"2024-03-11T19:55:45.724109+0000","last_became_peered":"2024-03-11T19:55:45.724109+0000","last_unstale":"2024-03-11T19:57:46.881861+0000","last_undegraded":"2024-03-11T19:57:46.881861+0000","last_fullsized":"2024-03-11T19:57:46.881861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:58:12.510437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881733+0000","last_change":"2024-03-11T19:55:45.718831+0000","last_active":"2024-03-11T19:57:46.881733+0000","last_peered":"2024-03-11T19:57:46.881733+0000","last_clean":"2024-03-11T19:57:46.881733+0000","last_became_active":"2024-03-11T19:55:45.717652+0000","last_became_peered":"2024-03-11T19:55:45.717652+0000","last_unstale":"2024-03-11T19:57:46.881733+0000","last_undegraded":"2024-03-11T19:57:46.881733+0000","last_fullsized":"2024-03-11T19:57:46.881733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:16.471754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685116+0000","last_change":"2024-03-11T19:55:45.701640+0000","last_active":"2024-03-11T19:57:46.685116+0000","last_peered":"2024-03-11T19:57:46.685116+0000","last_clean":"2024-03-11T19:57:46.685116+0000","last_became_active":"2024-03-11T19:55:45.699207+0000","last_became_peered":"2024-03-11T19:55:45.699207+0000","last_unstale":"2024-03-11T19:57:46.685116+0000","last_undegraded":"2024-03-11T19:57:46.685116+0000","last_fullsized":"2024-03-11T19:57:46.685116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:01.900429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685627+0000","last_change":"2024-03-11T19:55:45.732040+0000","last_active":"2024-03-11T19:57:46.685627+0000","last_peered":"2024-03-11T19:57:46.685627+0000","last_clean":"2024-03-11T19:57:46.685627+0000","last_became_active":"2024-03-11T19:55:45.731647+0000","last_became_peered":"2024-03-11T19:55:45.731647+0000","last_unstale":"2024-03-11T19:57:46.685627+0000","last_undegraded":"2024-03-11T19:57:46.685627+0000","last_fullsized":"2024-03-11T19:57:46.685627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:51:57.345329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683200+0000","last_change":"2024-03-11T19:55:45.707986+0000","last_active":"2024-03-11T19:57:46.683200+0000","last_peered":"2024-03-11T19:57:46.683200+0000","last_clean":"2024-03-11T19:57:46.683200+0000","last_became_active":"2024-03-11T19:55:45.707641+0000","last_became_peered":"2024-03-11T19:55:45.707641+0000","last_unstale":"2024-03-11T19:57:46.683200+0000","last_undegraded":"2024-03-11T19:57:46.683200+0000","last_fullsized":"2024-03-11T19:57:46.683200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:15:50.635234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683168+0000","last_change":"2024-03-11T19:56:45.705737+0000","last_active":"2024-03-11T19:57:46.683168+0000","last_peered":"2024-03-11T19:57:46.683168+0000","last_clean":"2024-03-11T19:57:46.683168+0000","last_became_active":"2024-03-11T19:56:45.705477+0000","last_became_peered":"2024-03-11T19:56:45.705477+0000","last_unstale":"2024-03-11T19:57:46.683168+0000","last_undegraded":"2024-03-11T19:57:46.683168+0000","last_fullsized":"2024-03-11T19:57:46.683168+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:01:14.068476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572051+0000","last_change":"2024-03-11T19:55:45.729587+0000","last_active":"2024-03-11T19:57:47.572051+0000","last_peered":"2024-03-11T19:57:47.572051+0000","last_clean":"2024-03-11T19:57:47.572051+0000","last_became_active":"2024-03-11T19:55:45.729037+0000","last_became_peered":"2024-03-11T19:55:45.729037+0000","last_unstale":"2024-03-11T19:57:47.572051+0000","last_undegraded":"2024-03-11T19:57:47.572051+0000","last_fullsized":"2024-03-11T19:57:47.572051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:55:43.445192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023626+0000","last_change":"2024-03-11T19:55:45.709793+0000","last_active":"2024-03-11T19:57:47.023626+0000","last_peered":"2024-03-11T19:57:47.023626+0000","last_clean":"2024-03-11T19:57:47.023626+0000","last_became_active":"2024-03-11T19:55:45.708631+0000","last_became_peered":"2024-03-11T19:55:45.708631+0000","last_unstale":"2024-03-11T19:57:47.023626+0000","last_undegraded":"2024-03-11T19:57:47.023626+0000","last_fullsized":"2024-03-11T19:57:47.023626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:02.144066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684105+0000","last_change":"2024-03-11T19:55:45.704227+0000","last_active":"2024-03-11T19:57:46.684105+0000","last_peered":"2024-03-11T19:57:46.684105+0000","last_clean":"2024-03-11T19:57:46.684105+0000","last_became_active":"2024-03-11T19:55:45.703885+0000","last_became_peered":"2024-03-11T19:55:45.703885+0000","last_unstale":"2024-03-11T19:57:46.684105+0000","last_undegraded":"2024-03-11T19:57:46.684105+0000","last_fullsized":"2024-03-11T19:57:46.684105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:36:21.401988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685372+0000","last_change":"2024-03-11T19:55:45.708950+0000","last_active":"2024-03-11T19:57:46.685372+0000","last_peered":"2024-03-11T19:57:46.685372+0000","last_clean":"2024-03-11T19:57:46.685372+0000","last_became_active":"2024-03-11T19:55:45.708544+0000","last_became_peered":"2024-03-11T19:55:45.708544+0000","last_unstale":"2024-03-11T19:57:46.685372+0000","last_undegraded":"2024-03-11T19:57:46.685372+0000","last_fullsized":"2024-03-11T19:57:46.685372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:06:34.244449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685013+0000","last_change":"2024-03-11T19:55:45.703923+0000","last_active":"2024-03-11T19:57:46.685013+0000","last_peered":"2024-03-11T19:57:46.685013+0000","last_clean":"2024-03-11T19:57:46.685013+0000","last_became_active":"2024-03-11T19:55:45.703595+0000","last_became_peered":"2024-03-11T19:55:45.703595+0000","last_unstale":"2024-03-11T19:57:46.685013+0000","last_undegraded":"2024-03-11T19:57:46.685013+0000","last_fullsized":"2024-03-11T19:57:46.685013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:00:19.683907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683892+0000","last_change":"2024-03-11T19:55:45.707499+0000","last_active":"2024-03-11T19:57:46.683892+0000","last_peered":"2024-03-11T19:57:46.683892+0000","last_clean":"2024-03-11T19:57:46.683892+0000","last_became_active":"2024-03-11T19:55:45.706344+0000","last_became_peered":"2024-03-11T19:55:45.706344+0000","last_unstale":"2024-03-11T19:57:46.683892+0000","last_undegraded":"2024-03-11T19:57:46.683892+0000","last_fullsized":"2024-03-11T19:57:46.683892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:38:24.082332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.880311+0000","last_change":"2024-03-11T19:55:45.722182+0000","last_active":"2024-03-11T19:57:46.880311+0000","last_peered":"2024-03-11T19:57:46.880311+0000","last_clean":"2024-03-11T19:57:46.880311+0000","last_became_active":"2024-03-11T19:55:45.721783+0000","last_became_peered":"2024-03-11T19:55:45.721783+0000","last_unstale":"2024-03-11T19:57:46.880311+0000","last_undegraded":"2024-03-11T19:57:46.880311+0000","last_fullsized":"2024-03-11T19:57:46.880311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:41:08.559002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.880277+0000","last_change":"2024-03-11T19:55:45.718881+0000","last_active":"2024-03-11T19:57:46.880277+0000","last_peered":"2024-03-11T19:57:46.880277+0000","last_clean":"2024-03-11T19:57:46.880277+0000","last_became_active":"2024-03-11T19:55:45.718458+0000","last_became_peered":"2024-03-11T19:55:45.718458+0000","last_unstale":"2024-03-11T19:57:46.880277+0000","last_undegraded":"2024-03-11T19:57:46.880277+0000","last_fullsized":"2024-03-11T19:57:46.880277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:22.075892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023120+0000","last_change":"2024-03-11T19:55:45.726163+0000","last_active":"2024-03-11T19:57:47.023120+0000","last_peered":"2024-03-11T19:57:47.023120+0000","last_clean":"2024-03-11T19:57:47.023120+0000","last_became_active":"2024-03-11T19:55:45.725772+0000","last_became_peered":"2024-03-11T19:55:45.725772+0000","last_unstale":"2024-03-11T19:57:47.023120+0000","last_undegraded":"2024-03-11T19:57:47.023120+0000","last_fullsized":"2024-03-11T19:57:47.023120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:36:19.057920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686772+0000","last_change":"2024-03-11T19:55:45.724787+0000","last_active":"2024-03-11T19:57:46.686772+0000","last_peered":"2024-03-11T19:57:46.686772+0000","last_clean":"2024-03-11T19:57:46.686772+0000","last_became_active":"2024-03-11T19:55:45.724437+0000","last_became_peered":"2024-03-11T19:55:45.724437+0000","last_unstale":"2024-03-11T19:57:46.686772+0000","last_undegraded":"2024-03-11T19:57:46.686772+0000","last_fullsized":"2024-03-11T19:57:46.686772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:55:53.811679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.691041+0000","last_change":"2024-03-11T19:55:45.714934+0000","last_active":"2024-03-11T19:57:46.691041+0000","last_peered":"2024-03-11T19:57:46.691041+0000","last_clean":"2024-03-11T19:57:46.691041+0000","last_became_active":"2024-03-11T19:55:45.709595+0000","last_became_peered":"2024-03-11T19:55:45.709595+0000","last_unstale":"2024-03-11T19:57:46.691041+0000","last_undegraded":"2024-03-11T19:57:46.691041+0000","last_fullsized":"2024-03-11T19:57:46.691041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:48.319073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802786+0000","last_change":"2024-03-11T19:55:45.725747+0000","last_active":"2024-03-11T19:57:46.802786+0000","last_peered":"2024-03-11T19:57:46.802786+0000","last_clean":"2024-03-11T19:57:46.802786+0000","last_became_active":"2024-03-11T19:55:45.725239+0000","last_became_peered":"2024-03-11T19:55:45.725239+0000","last_unstale":"2024-03-11T19:57:46.802786+0000","last_undegraded":"2024-03-11T19:57:46.802786+0000","last_fullsized":"2024-03-11T19:57:46.802786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:51:24.046681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802300+0000","last_change":"2024-03-11T19:55:50.109913+0000","last_active":"2024-03-11T19:57:46.802300+0000","last_peered":"2024-03-11T19:57:46.802300+0000","last_clean":"2024-03-11T19:57:46.802300+0000","last_became_active":"2024-03-11T19:55:50.109471+0000","last_became_peered":"2024-03-11T19:55:50.109471+0000","last_unstale":"2024-03-11T19:57:46.802300+0000","last_undegraded":"2024-03-11T19:57:46.802300+0000","last_fullsized":"2024-03-11T19:57:46.802300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:40.258022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":137,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:50.845313+0000","last_change":"2024-03-11T19:55:44.680181+0000","last_active":"2024-03-11T19:57:50.845313+0000","last_peered":"2024-03-11T19:57:50.845313+0000","last_clean":"2024-03-11T19:57:50.845313+0000","last_became_active":"2024-03-11T19:55:44.679807+0000","last_became_peered":"2024-03-11T19:55:44.679807+0000","last_unstale":"2024-03-11T19:57:50.845313+0000","last_undegraded":"2024-03-11T19:57:50.845313+0000","last_fullsized":"2024-03-11T19:57:50.845313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:23:04.532836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":93,"num_read_kb":93,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573255+0000","last_change":"2024-03-11T19:55:45.723547+0000","last_active":"2024-03-11T19:57:47.573255+0000","last_peered":"2024-03-11T19:57:47.573255+0000","last_clean":"2024-03-11T19:57:47.573255+0000","last_became_active":"2024-03-11T19:55:45.722516+0000","last_became_peered":"2024-03-11T19:55:45.722516+0000","last_unstale":"2024-03-11T19:57:47.573255+0000","last_undegraded":"2024-03-11T19:57:47.573255+0000","last_fullsized":"2024-03-11T19:57:47.573255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:40.271814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683243+0000","last_change":"2024-03-11T19:55:50.102141+0000","last_active":"2024-03-11T19:57:46.683243+0000","last_peered":"2024-03-11T19:57:46.683243+0000","last_clean":"2024-03-11T19:57:46.683243+0000","last_became_active":"2024-03-11T19:55:50.101667+0000","last_became_peered":"2024-03-11T19:55:50.101667+0000","last_unstale":"2024-03-11T19:57:46.683243+0000","last_undegraded":"2024-03-11T19:57:46.683243+0000","last_fullsized":"2024-03-11T19:57:46.683243+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:54:11.453059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573605+0000","last_change":"2024-03-11T19:55:44.684623+0000","last_active":"2024-03-11T19:57:47.573605+0000","last_peered":"2024-03-11T19:57:47.573605+0000","last_clean":"2024-03-11T19:57:47.573605+0000","last_became_active":"2024-03-11T19:55:44.683611+0000","last_became_peered":"2024-03-11T19:55:44.683611+0000","last_unstale":"2024-03-11T19:57:47.573605+0000","last_undegraded":"2024-03-11T19:57:47.573605+0000","last_fullsized":"2024-03-11T19:57:47.573605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:22:50.954364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881778+0000","last_change":"2024-03-11T19:55:45.723619+0000","last_active":"2024-03-11T19:57:46.881778+0000","last_peered":"2024-03-11T19:57:46.881778+0000","last_clean":"2024-03-11T19:57:46.881778+0000","last_became_active":"2024-03-11T19:55:45.722685+0000","last_became_peered":"2024-03-11T19:55:45.722685+0000","last_unstale":"2024-03-11T19:57:46.881778+0000","last_undegraded":"2024-03-11T19:57:46.881778+0000","last_fullsized":"2024-03-11T19:57:46.881778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:47:13.747776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.576246+0000","last_change":"2024-03-11T19:57:47.576246+0000","last_active":"2024-03-11T19:57:47.576246+0000","last_peered":"2024-03-11T19:57:47.576246+0000","last_clean":"2024-03-11T19:57:47.576246+0000","last_became_active":"2024-03-11T19:57:47.575837+0000","last_became_peered":"2024-03-11T19:57:47.575837+0000","last_unstale":"2024-03-11T19:57:47.576246+0000","last_undegraded":"2024-03-11T19:57:47.576246+0000","last_fullsized":"2024-03-11T19:57:47.576246+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:00.914062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687185+0000","last_change":"2024-03-11T19:56:45.714194+0000","last_active":"2024-03-11T19:57:46.687185+0000","last_peered":"2024-03-11T19:57:46.687185+0000","last_clean":"2024-03-11T19:57:46.687185+0000","last_became_active":"2024-03-11T19:56:45.694559+0000","last_became_peered":"2024-03-11T19:56:45.694559+0000","last_unstale":"2024-03-11T19:57:46.687185+0000","last_undegraded":"2024-03-11T19:57:46.687185+0000","last_fullsized":"2024-03-11T19:57:46.687185+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:01:20.307915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023254+0000","last_change":"2024-03-11T19:55:45.715157+0000","last_active":"2024-03-11T19:57:47.023254+0000","last_peered":"2024-03-11T19:57:47.023254+0000","last_clean":"2024-03-11T19:57:47.023254+0000","last_became_active":"2024-03-11T19:55:45.714536+0000","last_became_peered":"2024-03-11T19:55:45.714536+0000","last_unstale":"2024-03-11T19:57:47.023254+0000","last_undegraded":"2024-03-11T19:57:47.023254+0000","last_fullsized":"2024-03-11T19:57:47.023254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:02.416736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.682631+0000","last_change":"2024-03-11T19:55:50.102157+0000","last_active":"2024-03-11T19:57:46.682631+0000","last_peered":"2024-03-11T19:57:46.682631+0000","last_clean":"2024-03-11T19:57:46.682631+0000","last_became_active":"2024-03-11T19:55:50.101715+0000","last_became_peered":"2024-03-11T19:55:50.101715+0000","last_unstale":"2024-03-11T19:57:46.682631+0000","last_undegraded":"2024-03-11T19:57:46.682631+0000","last_fullsized":"2024-03-11T19:57:46.682631+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:12:18.780708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687126+0000","last_change":"2024-03-11T19:56:45.698585+0000","last_active":"2024-03-11T19:57:46.687126+0000","last_peered":"2024-03-11T19:57:46.687126+0000","last_clean":"2024-03-11T19:57:46.687126+0000","last_became_active":"2024-03-11T19:56:45.692929+0000","last_became_peered":"2024-03-11T19:56:45.692929+0000","last_unstale":"2024-03-11T19:57:46.687126+0000","last_undegraded":"2024-03-11T19:57:46.687126+0000","last_fullsized":"2024-03-11T19:57:46.687126+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:43:11.158934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680889+0000","last_change":"2024-03-11T19:55:45.709973+0000","last_active":"2024-03-11T19:57:46.680889+0000","last_peered":"2024-03-11T19:57:46.680889+0000","last_clean":"2024-03-11T19:57:46.680889+0000","last_became_active":"2024-03-11T19:55:45.709645+0000","last_became_peered":"2024-03-11T19:55:45.709645+0000","last_unstale":"2024-03-11T19:57:46.680889+0000","last_undegraded":"2024-03-11T19:57:46.680889+0000","last_fullsized":"2024-03-11T19:57:46.680889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:16.516982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802538+0000","last_change":"2024-03-11T19:55:50.094093+0000","last_active":"2024-03-11T19:57:46.802538+0000","last_peered":"2024-03-11T19:57:46.802538+0000","last_clean":"2024-03-11T19:57:46.802538+0000","last_became_active":"2024-03-11T19:55:50.093653+0000","last_became_peered":"2024-03-11T19:55:50.093653+0000","last_unstale":"2024-03-11T19:57:46.802538+0000","last_undegraded":"2024-03-11T19:57:46.802538+0000","last_fullsized":"2024-03-11T19:57:46.802538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:28:41.668001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":134,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:49.771696+0000","last_change":"2024-03-11T19:55:44.671547+0000","last_active":"2024-03-11T19:57:49.771696+0000","last_peered":"2024-03-11T19:57:49.771696+0000","last_clean":"2024-03-11T19:57:49.771696+0000","last_became_active":"2024-03-11T19:55:44.671211+0000","last_became_peered":"2024-03-11T19:55:44.671211+0000","last_unstale":"2024-03-11T19:57:49.771696+0000","last_undegraded":"2024-03-11T19:57:49.771696+0000","last_fullsized":"2024-03-11T19:57:49.771696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:18:46.261250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685626+0000","last_change":"2024-03-11T19:55:45.707968+0000","last_active":"2024-03-11T19:57:46.685626+0000","last_peered":"2024-03-11T19:57:46.685626+0000","last_clean":"2024-03-11T19:57:46.685626+0000","last_became_active":"2024-03-11T19:55:45.706998+0000","last_became_peered":"2024-03-11T19:55:45.706998+0000","last_unstale":"2024-03-11T19:57:46.685626+0000","last_undegraded":"2024-03-11T19:57:46.685626+0000","last_fullsized":"2024-03-11T19:57:46.685626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:49:35.711123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802348+0000","last_change":"2024-03-11T19:55:50.104310+0000","last_active":"2024-03-11T19:57:46.802348+0000","last_peered":"2024-03-11T19:57:46.802348+0000","last_clean":"2024-03-11T19:57:46.802348+0000","last_became_active":"2024-03-11T19:55:50.103831+0000","last_became_peered":"2024-03-11T19:55:50.103831+0000","last_unstale":"2024-03-11T19:57:46.802348+0000","last_undegraded":"2024-03-11T19:57:46.802348+0000","last_fullsized":"2024-03-11T19:57:46.802348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:33.213987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683083+0000","last_change":"2024-03-11T19:55:44.677615+0000","last_active":"2024-03-11T19:57:46.683083+0000","last_peered":"2024-03-11T19:57:46.683083+0000","last_clean":"2024-03-11T19:57:46.683083+0000","last_became_active":"2024-03-11T19:55:44.677153+0000","last_became_peered":"2024-03-11T19:55:44.677153+0000","last_unstale":"2024-03-11T19:57:46.683083+0000","last_undegraded":"2024-03-11T19:57:46.683083+0000","last_fullsized":"2024-03-11T19:57:46.683083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:52:59.153222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680692+0000","last_change":"2024-03-11T19:55:45.731827+0000","last_active":"2024-03-11T19:57:46.680692+0000","last_peered":"2024-03-11T19:57:46.680692+0000","last_clean":"2024-03-11T19:57:46.680692+0000","last_became_active":"2024-03-11T19:55:45.731460+0000","last_became_peered":"2024-03-11T19:55:45.731460+0000","last_unstale":"2024-03-11T19:57:46.680692+0000","last_undegraded":"2024-03-11T19:57:46.680692+0000","last_fullsized":"2024-03-11T19:57:46.680692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:12.277198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686597+0000","last_change":"2024-03-11T19:55:50.109812+0000","last_active":"2024-03-11T19:57:46.686597+0000","last_peered":"2024-03-11T19:57:46.686597+0000","last_clean":"2024-03-11T19:57:46.686597+0000","last_became_active":"2024-03-11T19:55:50.109420+0000","last_became_peered":"2024-03-11T19:55:50.109420+0000","last_unstale":"2024-03-11T19:57:46.686597+0000","last_undegraded":"2024-03-11T19:57:46.686597+0000","last_fullsized":"2024-03-11T19:57:46.686597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:08:30.105941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683574+0000","last_change":"2024-03-11T19:55:44.683173+0000","last_active":"2024-03-11T19:57:46.683574+0000","last_peered":"2024-03-11T19:57:46.683574+0000","last_clean":"2024-03-11T19:57:46.683574+0000","last_became_active":"2024-03-11T19:55:44.682778+0000","last_became_peered":"2024-03-11T19:55:44.682778+0000","last_unstale":"2024-03-11T19:57:46.683574+0000","last_undegraded":"2024-03-11T19:57:46.683574+0000","last_fullsized":"2024-03-11T19:57:46.683574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:15.119864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572517+0000","last_change":"2024-03-11T19:55:45.721833+0000","last_active":"2024-03-11T19:57:47.572517+0000","last_peered":"2024-03-11T19:57:47.572517+0000","last_clean":"2024-03-11T19:57:47.572517+0000","last_became_active":"2024-03-11T19:55:45.721490+0000","last_became_peered":"2024-03-11T19:55:45.721490+0000","last_unstale":"2024-03-11T19:57:47.572517+0000","last_undegraded":"2024-03-11T19:57:47.572517+0000","last_fullsized":"2024-03-11T19:57:47.572517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:09.824912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881479+0000","last_change":"2024-03-11T19:55:50.108424+0000","last_active":"2024-03-11T19:57:46.881479+0000","last_peered":"2024-03-11T19:57:46.881479+0000","last_clean":"2024-03-11T19:57:46.881479+0000","last_became_active":"2024-03-11T19:55:50.107072+0000","last_became_peered":"2024-03-11T19:55:50.107072+0000","last_unstale":"2024-03-11T19:57:46.881479+0000","last_undegraded":"2024-03-11T19:57:46.881479+0000","last_fullsized":"2024-03-11T19:57:46.881479+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:31.751597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680227+0000","last_change":"2024-03-11T19:55:44.671570+0000","last_active":"2024-03-11T19:57:46.680227+0000","last_peered":"2024-03-11T19:57:46.680227+0000","last_clean":"2024-03-11T19:57:46.680227+0000","last_became_active":"2024-03-11T19:55:44.671007+0000","last_became_peered":"2024-03-11T19:55:44.671007+0000","last_unstale":"2024-03-11T19:57:46.680227+0000","last_undegraded":"2024-03-11T19:57:46.680227+0000","last_fullsized":"2024-03-11T19:57:46.680227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:23.865011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.691569+0000","last_change":"2024-03-11T19:55:45.733048+0000","last_active":"2024-03-11T19:57:46.691569+0000","last_peered":"2024-03-11T19:57:46.691569+0000","last_clean":"2024-03-11T19:57:46.691569+0000","last_became_active":"2024-03-11T19:55:45.731656+0000","last_became_peered":"2024-03-11T19:55:45.731656+0000","last_unstale":"2024-03-11T19:57:46.691569+0000","last_undegraded":"2024-03-11T19:57:46.691569+0000","last_fullsized":"2024-03-11T19:57:46.691569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:28:56.433327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572169+0000","last_change":"2024-03-11T19:55:50.105532+0000","last_active":"2024-03-11T19:57:47.572169+0000","last_peered":"2024-03-11T19:57:47.572169+0000","last_clean":"2024-03-11T19:57:47.572169+0000","last_became_active":"2024-03-11T19:55:50.102200+0000","last_became_peered":"2024-03-11T19:55:50.102200+0000","last_unstale":"2024-03-11T19:57:47.572169+0000","last_undegraded":"2024-03-11T19:57:47.572169+0000","last_fullsized":"2024-03-11T19:57:47.572169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:37.060472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":224,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:51.774744+0000","last_change":"2024-03-11T19:55:44.678224+0000","last_active":"2024-03-11T19:57:51.774744+0000","last_peered":"2024-03-11T19:57:51.774744+0000","last_clean":"2024-03-11T19:57:51.774744+0000","last_became_active":"2024-03-11T19:55:44.677936+0000","last_became_peered":"2024-03-11T19:55:44.677936+0000","last_unstale":"2024-03-11T19:57:51.774744+0000","last_undegraded":"2024-03-11T19:57:51.774744+0000","last_fullsized":"2024-03-11T19:57:51.774744+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:20:42.486501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":188,"num_read_kb":188,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023625+0000","last_change":"2024-03-11T19:55:45.709917+0000","last_active":"2024-03-11T19:57:47.023625+0000","last_peered":"2024-03-11T19:57:47.023625+0000","last_clean":"2024-03-11T19:57:47.023625+0000","last_became_active":"2024-03-11T19:55:45.709318+0000","last_became_peered":"2024-03-11T19:55:45.709318+0000","last_unstale":"2024-03-11T19:57:47.023625+0000","last_undegraded":"2024-03-11T19:57:47.023625+0000","last_fullsized":"2024-03-11T19:57:47.023625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:08.298158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.883428+0000","last_change":"2024-03-11T19:57:46.882937+0000","last_active":"2024-03-11T19:57:46.883428+0000","last_peered":"2024-03-11T19:57:46.883428+0000","last_clean":"2024-03-11T19:57:46.883428+0000","last_became_active":"2024-03-11T19:57:46.882408+0000","last_became_peered":"2024-03-11T19:57:46.882408+0000","last_unstale":"2024-03-11T19:57:46.883428+0000","last_undegraded":"2024-03-11T19:57:46.883428+0000","last_fullsized":"2024-03-11T19:57:46.883428+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:58:27.842775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685774+0000","last_change":"2024-03-11T19:55:44.676004+0000","last_active":"2024-03-11T19:57:46.685774+0000","last_peered":"2024-03-11T19:57:46.685774+0000","last_clean":"2024-03-11T19:57:46.685774+0000","last_became_active":"2024-03-11T19:55:44.675640+0000","last_became_peered":"2024-03-11T19:55:44.675640+0000","last_unstale":"2024-03-11T19:57:46.685774+0000","last_undegraded":"2024-03-11T19:57:46.685774+0000","last_fullsized":"2024-03-11T19:57:46.685774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:15.096780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881928+0000","last_change":"2024-03-11T19:56:45.702569+0000","last_active":"2024-03-11T19:57:46.881928+0000","last_peered":"2024-03-11T19:57:46.881928+0000","last_clean":"2024-03-11T19:57:46.881928+0000","last_became_active":"2024-03-11T19:56:45.701890+0000","last_became_peered":"2024-03-11T19:56:45.701890+0000","last_unstale":"2024-03-11T19:57:46.881928+0000","last_undegraded":"2024-03-11T19:57:46.881928+0000","last_fullsized":"2024-03-11T19:57:46.881928+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:40.434712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684317+0000","last_change":"2024-03-11T19:55:50.096136+0000","last_active":"2024-03-11T19:57:46.684317+0000","last_peered":"2024-03-11T19:57:46.684317+0000","last_clean":"2024-03-11T19:57:46.684317+0000","last_became_active":"2024-03-11T19:55:50.095670+0000","last_became_peered":"2024-03-11T19:55:50.095670+0000","last_unstale":"2024-03-11T19:57:46.684317+0000","last_undegraded":"2024-03-11T19:57:46.684317+0000","last_fullsized":"2024-03-11T19:57:46.684317+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:40.943273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680839+0000","last_change":"2024-03-11T19:55:44.681547+0000","last_active":"2024-03-11T19:57:46.680839+0000","last_peered":"2024-03-11T19:57:46.680839+0000","last_clean":"2024-03-11T19:57:46.680839+0000","last_became_active":"2024-03-11T19:55:44.681237+0000","last_became_peered":"2024-03-11T19:55:44.681237+0000","last_unstale":"2024-03-11T19:57:46.680839+0000","last_undegraded":"2024-03-11T19:57:46.680839+0000","last_fullsized":"2024-03-11T19:57:46.680839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:08:09.509915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023380+0000","last_change":"2024-03-11T19:55:45.705025+0000","last_active":"2024-03-11T19:57:47.023380+0000","last_peered":"2024-03-11T19:57:47.023380+0000","last_clean":"2024-03-11T19:57:47.023380+0000","last_became_active":"2024-03-11T19:55:45.702680+0000","last_became_peered":"2024-03-11T19:55:45.702680+0000","last_unstale":"2024-03-11T19:57:47.023380+0000","last_undegraded":"2024-03-11T19:57:47.023380+0000","last_fullsized":"2024-03-11T19:57:47.023380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:28.245002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.689568+0000","last_change":"2024-03-11T19:55:50.099648+0000","last_active":"2024-03-11T19:57:46.689568+0000","last_peered":"2024-03-11T19:57:46.689568+0000","last_clean":"2024-03-11T19:57:46.689568+0000","last_became_active":"2024-03-11T19:55:50.098516+0000","last_became_peered":"2024-03-11T19:55:50.098516+0000","last_unstale":"2024-03-11T19:57:46.689568+0000","last_undegraded":"2024-03-11T19:57:46.689568+0000","last_fullsized":"2024-03-11T19:57:46.689568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:19.644046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":260,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:51.774692+0000","last_change":"2024-03-11T19:55:44.672460+0000","last_active":"2024-03-11T19:57:51.774692+0000","last_peered":"2024-03-11T19:57:51.774692+0000","last_clean":"2024-03-11T19:57:51.774692+0000","last_became_active":"2024-03-11T19:55:44.672123+0000","last_became_peered":"2024-03-11T19:55:44.672123+0000","last_unstale":"2024-03-11T19:57:51.774692+0000","last_undegraded":"2024-03-11T19:57:51.774692+0000","last_fullsized":"2024-03-11T19:57:51.774692+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:43.940638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61471,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":195,"num_read_kb":158,"num_write":37,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685900+0000","last_change":"2024-03-11T19:55:45.702933+0000","last_active":"2024-03-11T19:57:46.685900+0000","last_peered":"2024-03-11T19:57:46.685900+0000","last_clean":"2024-03-11T19:57:46.685900+0000","last_became_active":"2024-03-11T19:55:45.702585+0000","last_became_peered":"2024-03-11T19:55:45.702585+0000","last_unstale":"2024-03-11T19:57:46.685900+0000","last_undegraded":"2024-03-11T19:57:46.685900+0000","last_fullsized":"2024-03-11T19:57:46.685900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:54:38.762156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.681155+0000","last_change":"2024-03-11T19:55:50.102571+0000","last_active":"2024-03-11T19:57:46.681155+0000","last_peered":"2024-03-11T19:57:46.681155+0000","last_clean":"2024-03-11T19:57:46.681155+0000","last_became_active":"2024-03-11T19:55:50.102099+0000","last_became_peered":"2024-03-11T19:55:50.102099+0000","last_unstale":"2024-03-11T19:57:46.681155+0000","last_undegraded":"2024-03-11T19:57:46.681155+0000","last_fullsized":"2024-03-11T19:57:46.681155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:17.631378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802401+0000","last_change":"2024-03-11T19:55:44.671027+0000","last_active":"2024-03-11T19:57:46.802401+0000","last_peered":"2024-03-11T19:57:46.802401+0000","last_clean":"2024-03-11T19:57:46.802401+0000","last_became_active":"2024-03-11T19:55:44.669696+0000","last_became_peered":"2024-03-11T19:55:44.669696+0000","last_unstale":"2024-03-11T19:57:46.802401+0000","last_undegraded":"2024-03-11T19:57:46.802401+0000","last_fullsized":"2024-03-11T19:57:46.802401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:48:20.169638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683919+0000","last_change":"2024-03-11T19:55:45.729119+0000","last_active":"2024-03-11T19:57:46.683919+0000","last_peered":"2024-03-11T19:57:46.683919+0000","last_clean":"2024-03-11T19:57:46.683919+0000","last_became_active":"2024-03-11T19:55:45.727442+0000","last_became_peered":"2024-03-11T19:55:45.727442+0000","last_unstale":"2024-03-11T19:57:46.683919+0000","last_undegraded":"2024-03-11T19:57:46.683919+0000","last_fullsized":"2024-03-11T19:57:46.683919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:14:51.483199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686945+0000","last_change":"2024-03-11T19:55:50.104326+0000","last_active":"2024-03-11T19:57:46.686945+0000","last_peered":"2024-03-11T19:57:46.686945+0000","last_clean":"2024-03-11T19:57:46.686945+0000","last_became_active":"2024-03-11T19:55:50.103881+0000","last_became_peered":"2024-03-11T19:55:50.103881+0000","last_unstale":"2024-03-11T19:57:46.686945+0000","last_undegraded":"2024-03-11T19:57:46.686945+0000","last_fullsized":"2024-03-11T19:57:46.686945+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:12:38.819591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680603+0000","last_change":"2024-03-11T19:55:44.684900+0000","last_active":"2024-03-11T19:57:46.680603+0000","last_peered":"2024-03-11T19:57:46.680603+0000","last_clean":"2024-03-11T19:57:46.680603+0000","last_became_active":"2024-03-11T19:55:44.684590+0000","last_became_peered":"2024-03-11T19:55:44.684590+0000","last_unstale":"2024-03-11T19:57:46.680603+0000","last_undegraded":"2024-03-11T19:57:46.680603+0000","last_fullsized":"2024-03-11T19:57:46.680603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:07.744366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684928+0000","last_change":"2024-03-11T19:55:45.704787+0000","last_active":"2024-03-11T19:57:46.684928+0000","last_peered":"2024-03-11T19:57:46.684928+0000","last_clean":"2024-03-11T19:57:46.684928+0000","last_became_active":"2024-03-11T19:55:45.704405+0000","last_became_peered":"2024-03-11T19:55:45.704405+0000","last_unstale":"2024-03-11T19:57:46.684928+0000","last_undegraded":"2024-03-11T19:57:46.684928+0000","last_fullsized":"2024-03-11T19:57:46.684928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:58:05.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.690120+0000","last_change":"2024-03-11T19:55:50.104793+0000","last_active":"2024-03-11T19:57:46.690120+0000","last_peered":"2024-03-11T19:57:46.690120+0000","last_clean":"2024-03-11T19:57:46.690120+0000","last_became_active":"2024-03-11T19:55:50.104415+0000","last_became_peered":"2024-03-11T19:55:50.104415+0000","last_unstale":"2024-03-11T19:57:46.690120+0000","last_undegraded":"2024-03-11T19:57:46.690120+0000","last_fullsized":"2024-03-11T19:57:46.690120+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:53.307313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":131,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:50.797863+0000","last_change":"2024-03-11T19:55:44.669403+0000","last_active":"2024-03-11T19:57:50.797863+0000","last_peered":"2024-03-11T19:57:50.797863+0000","last_clean":"2024-03-11T19:57:50.797863+0000","last_became_active":"2024-03-11T19:55:44.668908+0000","last_became_peered":"2024-03-11T19:55:44.668908+0000","last_unstale":"2024-03-11T19:57:50.797863+0000","last_undegraded":"2024-03-11T19:57:50.797863+0000","last_fullsized":"2024-03-11T19:57:50.797863+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:56:36.831506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":97,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684333+0000","last_change":"2024-03-11T19:55:45.700029+0000","last_active":"2024-03-11T19:57:46.684333+0000","last_peered":"2024-03-11T19:57:46.684333+0000","last_clean":"2024-03-11T19:57:46.684333+0000","last_became_active":"2024-03-11T19:55:45.697296+0000","last_became_peered":"2024-03-11T19:55:45.697296+0000","last_unstale":"2024-03-11T19:57:46.684333+0000","last_undegraded":"2024-03-11T19:57:46.684333+0000","last_fullsized":"2024-03-11T19:57:46.684333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:59:35.025845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.577523+0000","last_change":"2024-03-11T19:57:47.577523+0000","last_active":"2024-03-11T19:57:47.577523+0000","last_peered":"2024-03-11T19:57:47.577523+0000","last_clean":"2024-03-11T19:57:47.577523+0000","last_became_active":"2024-03-11T19:57:47.575852+0000","last_became_peered":"2024-03-11T19:57:47.575852+0000","last_unstale":"2024-03-11T19:57:47.577523+0000","last_undegraded":"2024-03-11T19:57:47.577523+0000","last_fullsized":"2024-03-11T19:57:47.577523+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:00:25.108706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685341+0000","last_change":"2024-03-11T19:55:44.677491+0000","last_active":"2024-03-11T19:57:46.685341+0000","last_peered":"2024-03-11T19:57:46.685341+0000","last_clean":"2024-03-11T19:57:46.685341+0000","last_became_active":"2024-03-11T19:55:44.677176+0000","last_became_peered":"2024-03-11T19:55:44.677176+0000","last_unstale":"2024-03-11T19:57:46.685341+0000","last_undegraded":"2024-03-11T19:57:46.685341+0000","last_fullsized":"2024-03-11T19:57:46.685341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:51.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680318+0000","last_change":"2024-03-11T19:55:45.703310+0000","last_active":"2024-03-11T19:57:46.680318+0000","last_peered":"2024-03-11T19:57:46.680318+0000","last_clean":"2024-03-11T19:57:46.680318+0000","last_became_active":"2024-03-11T19:55:45.702933+0000","last_became_peered":"2024-03-11T19:55:45.702933+0000","last_unstale":"2024-03-11T19:57:46.680318+0000","last_undegraded":"2024-03-11T19:57:46.680318+0000","last_fullsized":"2024-03-11T19:57:46.680318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:07:58.787949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680420+0000","last_change":"2024-03-11T19:55:50.103849+0000","last_active":"2024-03-11T19:57:46.680420+0000","last_peered":"2024-03-11T19:57:46.680420+0000","last_clean":"2024-03-11T19:57:46.680420+0000","last_became_active":"2024-03-11T19:55:50.103360+0000","last_became_peered":"2024-03-11T19:55:50.103360+0000","last_unstale":"2024-03-11T19:57:46.680420+0000","last_undegraded":"2024-03-11T19:57:46.680420+0000","last_fullsized":"2024-03-11T19:57:46.680420+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:38:05.151041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685858+0000","last_change":"2024-03-11T19:55:44.669839+0000","last_active":"2024-03-11T19:57:46.685858+0000","last_peered":"2024-03-11T19:57:46.685858+0000","last_clean":"2024-03-11T19:57:46.685858+0000","last_became_active":"2024-03-11T19:55:44.669494+0000","last_became_peered":"2024-03-11T19:55:44.669494+0000","last_unstale":"2024-03-11T19:57:46.685858+0000","last_undegraded":"2024-03-11T19:57:46.685858+0000","last_fullsized":"2024-03-11T19:57:46.685858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:00.313847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687005+0000","last_change":"2024-03-11T19:56:45.693864+0000","last_active":"2024-03-11T19:57:46.687005+0000","last_peered":"2024-03-11T19:57:46.687005+0000","last_clean":"2024-03-11T19:57:46.687005+0000","last_became_active":"2024-03-11T19:56:45.693091+0000","last_became_peered":"2024-03-11T19:56:45.693091+0000","last_unstale":"2024-03-11T19:57:46.687005+0000","last_undegraded":"2024-03-11T19:57:46.687005+0000","last_fullsized":"2024-03-11T19:57:46.687005+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:26:37.233485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685431+0000","last_change":"2024-03-11T19:55:50.143916+0000","last_active":"2024-03-11T19:57:46.685431+0000","last_peered":"2024-03-11T19:57:46.685431+0000","last_clean":"2024-03-11T19:57:46.685431+0000","last_became_active":"2024-03-11T19:55:50.143341+0000","last_became_peered":"2024-03-11T19:55:50.143341+0000","last_unstale":"2024-03-11T19:57:46.685431+0000","last_undegraded":"2024-03-11T19:57:46.685431+0000","last_fullsized":"2024-03-11T19:57:46.685431+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:40.637529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.691467+0000","last_change":"2024-03-11T19:55:44.688468+0000","last_active":"2024-03-11T19:57:46.691467+0000","last_peered":"2024-03-11T19:57:46.691467+0000","last_clean":"2024-03-11T19:57:46.691467+0000","last_became_active":"2024-03-11T19:55:44.688065+0000","last_became_peered":"2024-03-11T19:55:44.688065+0000","last_unstale":"2024-03-11T19:57:46.691467+0000","last_undegraded":"2024-03-11T19:57:46.691467+0000","last_fullsized":"2024-03-11T19:57:46.691467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:29:23.392749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683940+0000","last_change":"2024-03-11T19:55:45.728970+0000","last_active":"2024-03-11T19:57:46.683940+0000","last_peered":"2024-03-11T19:57:46.683940+0000","last_clean":"2024-03-11T19:57:46.683940+0000","last_became_active":"2024-03-11T19:55:45.728516+0000","last_became_peered":"2024-03-11T19:55:45.728516+0000","last_unstale":"2024-03-11T19:57:46.683940+0000","last_undegraded":"2024-03-11T19:57:46.683940+0000","last_fullsized":"2024-03-11T19:57:46.683940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:19.242610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684094+0000","last_change":"2024-03-11T19:55:50.111884+0000","last_active":"2024-03-11T19:57:46.684094+0000","last_peered":"2024-03-11T19:57:46.684094+0000","last_clean":"2024-03-11T19:57:46.684094+0000","last_became_active":"2024-03-11T19:55:50.110802+0000","last_became_peered":"2024-03-11T19:55:50.110802+0000","last_unstale":"2024-03-11T19:57:46.684094+0000","last_undegraded":"2024-03-11T19:57:46.684094+0000","last_fullsized":"2024-03-11T19:57:46.684094+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:35.866368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":131,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:51.848738+0000","last_change":"2024-03-11T19:55:44.677156+0000","last_active":"2024-03-11T19:57:51.848738+0000","last_peered":"2024-03-11T19:57:51.848738+0000","last_clean":"2024-03-11T19:57:51.848738+0000","last_became_active":"2024-03-11T19:55:44.676656+0000","last_became_peered":"2024-03-11T19:55:44.676656+0000","last_unstale":"2024-03-11T19:57:51.848738+0000","last_undegraded":"2024-03-11T19:57:51.848738+0000","last_fullsized":"2024-03-11T19:57:51.848738+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:40:27.521669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683235+0000","last_change":"2024-03-11T19:55:45.715545+0000","last_active":"2024-03-11T19:57:46.683235+0000","last_peered":"2024-03-11T19:57:46.683235+0000","last_clean":"2024-03-11T19:57:46.683235+0000","last_became_active":"2024-03-11T19:55:45.715042+0000","last_became_peered":"2024-03-11T19:55:45.715042+0000","last_unstale":"2024-03-11T19:57:46.683235+0000","last_undegraded":"2024-03-11T19:57:46.683235+0000","last_fullsized":"2024-03-11T19:57:46.683235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:06:33.362235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685037+0000","last_change":"2024-03-11T19:55:50.145603+0000","last_active":"2024-03-11T19:57:46.685037+0000","last_peered":"2024-03-11T19:57:46.685037+0000","last_clean":"2024-03-11T19:57:46.685037+0000","last_became_active":"2024-03-11T19:55:50.143689+0000","last_became_peered":"2024-03-11T19:55:50.143689+0000","last_unstale":"2024-03-11T19:57:46.685037+0000","last_undegraded":"2024-03-11T19:57:46.685037+0000","last_fullsized":"2024-03-11T19:57:46.685037+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:57.289983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680431+0000","last_change":"2024-03-11T19:55:45.702011+0000","last_active":"2024-03-11T19:57:46.680431+0000","last_peered":"2024-03-11T19:57:46.680431+0000","last_clean":"2024-03-11T19:57:46.680431+0000","last_became_active":"2024-03-11T19:55:45.701673+0000","last_became_peered":"2024-03-11T19:55:45.701673+0000","last_unstale":"2024-03-11T19:57:46.680431+0000","last_undegraded":"2024-03-11T19:57:46.680431+0000","last_fullsized":"2024-03-11T19:57:46.680431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:17.028276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":131,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:50.797169+0000","last_change":"2024-03-11T19:55:44.689121+0000","last_active":"2024-03-11T19:57:50.797169+0000","last_peered":"2024-03-11T19:57:50.797169+0000","last_clean":"2024-03-11T19:57:50.797169+0000","last_became_active":"2024-03-11T19:55:44.688453+0000","last_became_peered":"2024-03-11T19:55:44.688453+0000","last_unstale":"2024-03-11T19:57:50.797169+0000","last_undegraded":"2024-03-11T19:57:50.797169+0000","last_fullsized":"2024-03-11T19:57:50.797169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:31.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":94,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.698213+0000","last_change":"2024-03-11T19:57:46.698213+0000","last_active":"2024-03-11T19:57:46.698213+0000","last_peered":"2024-03-11T19:57:46.698213+0000","last_clean":"2024-03-11T19:57:46.698213+0000","last_became_active":"2024-03-11T19:57:46.693710+0000","last_became_peered":"2024-03-11T19:57:46.693710+0000","last_unstale":"2024-03-11T19:57:46.698213+0000","last_undegraded":"2024-03-11T19:57:46.698213+0000","last_fullsized":"2024-03-11T19:57:46.698213+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:06:00.214092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.691362+0000","last_change":"2024-03-11T19:56:45.703746+0000","last_active":"2024-03-11T19:57:46.691362+0000","last_peered":"2024-03-11T19:57:46.691362+0000","last_clean":"2024-03-11T19:57:46.691362+0000","last_became_active":"2024-03-11T19:56:45.701252+0000","last_became_peered":"2024-03-11T19:56:45.701252+0000","last_unstale":"2024-03-11T19:57:46.691362+0000","last_undegraded":"2024-03-11T19:57:46.691362+0000","last_fullsized":"2024-03-11T19:57:46.691362+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:16:50.219593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684249+0000","last_change":"2024-03-11T19:55:44.672644+0000","last_active":"2024-03-11T19:57:46.684249+0000","last_peered":"2024-03-11T19:57:46.684249+0000","last_clean":"2024-03-11T19:57:46.684249+0000","last_became_active":"2024-03-11T19:55:44.671645+0000","last_became_peered":"2024-03-11T19:55:44.671645+0000","last_unstale":"2024-03-11T19:57:46.684249+0000","last_undegraded":"2024-03-11T19:57:46.684249+0000","last_fullsized":"2024-03-11T19:57:46.684249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:25:46.637768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":367,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684193+0000","last_change":"2024-03-11T19:54:20.915891+0000","last_active":"2024-03-11T19:57:46.684193+0000","last_peered":"2024-03-11T19:57:46.684193+0000","last_clean":"2024-03-11T19:57:46.684193+0000","last_became_active":"2024-03-11T19:54:20.915450+0000","last_became_peered":"2024-03-11T19:54:20.915450+0000","last_unstale":"2024-03-11T19:57:46.684193+0000","last_undegraded":"2024-03-11T19:57:46.684193+0000","last_fullsized":"2024-03-11T19:57:46.684193+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_clean_scrub_stamp":"2024-03-11T19:50:29.052858+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:47.966076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680850+0000","last_change":"2024-03-11T19:55:50.144053+0000","last_active":"2024-03-11T19:57:46.680850+0000","last_peered":"2024-03-11T19:57:46.680850+0000","last_clean":"2024-03-11T19:57:46.680850+0000","last_became_active":"2024-03-11T19:55:50.143509+0000","last_became_peered":"2024-03-11T19:55:50.143509+0000","last_unstale":"2024-03-11T19:57:46.680850+0000","last_undegraded":"2024-03-11T19:57:46.680850+0000","last_fullsized":"2024-03-11T19:57:46.680850+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:57:15.448812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683348+0000","last_change":"2024-03-11T19:55:45.712621+0000","last_active":"2024-03-11T19:57:46.683348+0000","last_peered":"2024-03-11T19:57:46.683348+0000","last_clean":"2024-03-11T19:57:46.683348+0000","last_became_active":"2024-03-11T19:55:45.712161+0000","last_became_peered":"2024-03-11T19:55:45.712161+0000","last_unstale":"2024-03-11T19:57:46.683348+0000","last_undegraded":"2024-03-11T19:57:46.683348+0000","last_fullsized":"2024-03-11T19:57:46.683348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:34:52.911944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684355+0000","last_change":"2024-03-11T19:55:44.674606+0000","last_active":"2024-03-11T19:57:46.684355+0000","last_peered":"2024-03-11T19:57:46.684355+0000","last_clean":"2024-03-11T19:57:46.684355+0000","last_became_active":"2024-03-11T19:55:44.674355+0000","last_became_peered":"2024-03-11T19:55:44.674355+0000","last_unstale":"2024-03-11T19:57:46.684355+0000","last_undegraded":"2024-03-11T19:57:46.684355+0000","last_fullsized":"2024-03-11T19:57:46.684355+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:31:01.662582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686690+0000","last_change":"2024-03-11T19:55:50.101468+0000","last_active":"2024-03-11T19:57:46.686690+0000","last_peered":"2024-03-11T19:57:46.686690+0000","last_clean":"2024-03-11T19:57:46.686690+0000","last_became_active":"2024-03-11T19:55:50.101040+0000","last_became_peered":"2024-03-11T19:55:50.101040+0000","last_unstale":"2024-03-11T19:57:46.686690+0000","last_undegraded":"2024-03-11T19:57:46.686690+0000","last_fullsized":"2024-03-11T19:57:46.686690+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:55.354176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686046+0000","last_change":"2024-03-11T19:55:44.681467+0000","last_active":"2024-03-11T19:57:46.686046+0000","last_peered":"2024-03-11T19:57:46.686046+0000","last_clean":"2024-03-11T19:57:46.686046+0000","last_became_active":"2024-03-11T19:55:44.681176+0000","last_became_peered":"2024-03-11T19:55:44.681176+0000","last_unstale":"2024-03-11T19:57:46.686046+0000","last_undegraded":"2024-03-11T19:57:46.686046+0000","last_fullsized":"2024-03-11T19:57:46.686046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:27:38.019116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685401+0000","last_change":"2024-03-11T19:55:45.709798+0000","last_active":"2024-03-11T19:57:46.685401+0000","last_peered":"2024-03-11T19:57:46.685401+0000","last_clean":"2024-03-11T19:57:46.685401+0000","last_became_active":"2024-03-11T19:55:45.709277+0000","last_became_peered":"2024-03-11T19:55:45.709277+0000","last_unstale":"2024-03-11T19:57:46.685401+0000","last_undegraded":"2024-03-11T19:57:46.685401+0000","last_fullsized":"2024-03-11T19:57:46.685401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:58.852316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572588+0000","last_change":"2024-03-11T19:55:50.146601+0000","last_active":"2024-03-11T19:57:47.572588+0000","last_peered":"2024-03-11T19:57:47.572588+0000","last_clean":"2024-03-11T19:57:47.572588+0000","last_became_active":"2024-03-11T19:55:50.144584+0000","last_became_peered":"2024-03-11T19:55:50.144584+0000","last_unstale":"2024-03-11T19:57:47.572588+0000","last_undegraded":"2024-03-11T19:57:47.572588+0000","last_fullsized":"2024-03-11T19:57:47.572588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:00.365713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'14","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.690534+0000","last_change":"2024-03-11T19:55:44.689109+0000","last_active":"2024-03-11T19:57:46.690534+0000","last_peered":"2024-03-11T19:57:46.690534+0000","last_clean":"2024-03-11T19:57:46.690534+0000","last_became_active":"2024-03-11T19:55:44.687323+0000","last_became_peered":"2024-03-11T19:55:44.687323+0000","last_unstale":"2024-03-11T19:57:46.690534+0000","last_undegraded":"2024-03-11T19:57:46.690534+0000","last_fullsized":"2024-03-11T19:57:46.690534+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:24.613901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573714+0000","last_change":"2024-03-11T19:55:45.726033+0000","last_active":"2024-03-11T19:57:47.573714+0000","last_peered":"2024-03-11T19:57:47.573714+0000","last_clean":"2024-03-11T19:57:47.573714+0000","last_became_active":"2024-03-11T19:55:45.725353+0000","last_became_peered":"2024-03-11T19:55:45.725353+0000","last_unstale":"2024-03-11T19:57:47.573714+0000","last_undegraded":"2024-03-11T19:57:47.573714+0000","last_fullsized":"2024-03-11T19:57:47.573714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:36:24.341870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802087+0000","last_change":"2024-03-11T19:55:50.089496+0000","last_active":"2024-03-11T19:57:46.802087+0000","last_peered":"2024-03-11T19:57:46.802087+0000","last_clean":"2024-03-11T19:57:46.802087+0000","last_became_active":"2024-03-11T19:55:50.089072+0000","last_became_peered":"2024-03-11T19:55:50.089072+0000","last_unstale":"2024-03-11T19:57:46.802087+0000","last_undegraded":"2024-03-11T19:57:46.802087+0000","last_fullsized":"2024-03-11T19:57:46.802087+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:27:32.539333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684876+0000","last_change":"2024-03-11T19:55:44.673147+0000","last_active":"2024-03-11T19:57:46.684876+0000","last_peered":"2024-03-11T19:57:46.684876+0000","last_clean":"2024-03-11T19:57:46.684876+0000","last_became_active":"2024-03-11T19:55:44.672787+0000","last_became_peered":"2024-03-11T19:55:44.672787+0000","last_unstale":"2024-03-11T19:57:46.684876+0000","last_undegraded":"2024-03-11T19:57:46.684876+0000","last_fullsized":"2024-03-11T19:57:46.684876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:11:23.864419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881937+0000","last_change":"2024-03-11T19:55:45.732376+0000","last_active":"2024-03-11T19:57:46.881937+0000","last_peered":"2024-03-11T19:57:46.881937+0000","last_clean":"2024-03-11T19:57:46.881937+0000","last_became_active":"2024-03-11T19:55:45.731956+0000","last_became_peered":"2024-03-11T19:55:45.731956+0000","last_unstale":"2024-03-11T19:57:46.881937+0000","last_undegraded":"2024-03-11T19:57:46.881937+0000","last_fullsized":"2024-03-11T19:57:46.881937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:43:48.423678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684605+0000","last_change":"2024-03-11T19:55:50.144080+0000","last_active":"2024-03-11T19:57:46.684605+0000","last_peered":"2024-03-11T19:57:46.684605+0000","last_clean":"2024-03-11T19:57:46.684605+0000","last_became_active":"2024-03-11T19:55:50.143585+0000","last_became_peered":"2024-03-11T19:55:50.143585+0000","last_unstale":"2024-03-11T19:57:46.684605+0000","last_undegraded":"2024-03-11T19:57:46.684605+0000","last_fullsized":"2024-03-11T19:57:46.684605+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:47:51.388823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023178+0000","last_change":"2024-03-11T19:55:44.677354+0000","last_active":"2024-03-11T19:57:47.023178+0000","last_peered":"2024-03-11T19:57:47.023178+0000","last_clean":"2024-03-11T19:57:47.023178+0000","last_became_active":"2024-03-11T19:55:44.677013+0000","last_became_peered":"2024-03-11T19:55:44.677013+0000","last_unstale":"2024-03-11T19:57:47.023178+0000","last_undegraded":"2024-03-11T19:57:47.023178+0000","last_fullsized":"2024-03-11T19:57:47.023178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:50:48.855320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685353+0000","last_change":"2024-03-11T19:55:45.702278+0000","last_active":"2024-03-11T19:57:46.685353+0000","last_peered":"2024-03-11T19:57:46.685353+0000","last_clean":"2024-03-11T19:57:46.685353+0000","last_became_active":"2024-03-11T19:55:45.701843+0000","last_became_peered":"2024-03-11T19:55:45.701843+0000","last_unstale":"2024-03-11T19:57:46.685353+0000","last_undegraded":"2024-03-11T19:57:46.685353+0000","last_fullsized":"2024-03-11T19:57:46.685353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:55:28.351093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.690731+0000","last_change":"2024-03-11T19:55:50.106083+0000","last_active":"2024-03-11T19:57:46.690731+0000","last_peered":"2024-03-11T19:57:46.690731+0000","last_clean":"2024-03-11T19:57:46.690731+0000","last_became_active":"2024-03-11T19:55:50.105591+0000","last_became_peered":"2024-03-11T19:55:50.105591+0000","last_unstale":"2024-03-11T19:57:46.690731+0000","last_undegraded":"2024-03-11T19:57:46.690731+0000","last_fullsized":"2024-03-11T19:57:46.690731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:34:58.713672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573135+0000","last_change":"2024-03-11T19:55:44.688128+0000","last_active":"2024-03-11T19:57:47.573135+0000","last_peered":"2024-03-11T19:57:47.573135+0000","last_clean":"2024-03-11T19:57:47.573135+0000","last_became_active":"2024-03-11T19:55:44.684165+0000","last_became_peered":"2024-03-11T19:55:44.684165+0000","last_unstale":"2024-03-11T19:57:47.573135+0000","last_undegraded":"2024-03-11T19:57:47.573135+0000","last_fullsized":"2024-03-11T19:57:47.573135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:37:19.415960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686707+0000","last_change":"2024-03-11T19:55:45.718968+0000","last_active":"2024-03-11T19:57:46.686707+0000","last_peered":"2024-03-11T19:57:46.686707+0000","last_clean":"2024-03-11T19:57:46.686707+0000","last_became_active":"2024-03-11T19:55:45.718625+0000","last_became_peered":"2024-03-11T19:55:45.718625+0000","last_unstale":"2024-03-11T19:57:46.686707+0000","last_undegraded":"2024-03-11T19:57:46.686707+0000","last_fullsized":"2024-03-11T19:57:46.686707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:27.197227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684464+0000","last_change":"2024-03-11T19:55:50.097617+0000","last_active":"2024-03-11T19:57:46.684464+0000","last_peered":"2024-03-11T19:57:46.684464+0000","last_clean":"2024-03-11T19:57:46.684464+0000","last_became_active":"2024-03-11T19:55:50.097137+0000","last_became_peered":"2024-03-11T19:55:50.097137+0000","last_unstale":"2024-03-11T19:57:46.684464+0000","last_undegraded":"2024-03-11T19:57:46.684464+0000","last_fullsized":"2024-03-11T19:57:46.684464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:19:20.995137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802186+0000","last_change":"2024-03-11T19:55:44.669944+0000","last_active":"2024-03-11T19:57:46.802186+0000","last_peered":"2024-03-11T19:57:46.802186+0000","last_clean":"2024-03-11T19:57:46.802186+0000","last_became_active":"2024-03-11T19:55:44.669309+0000","last_became_peered":"2024-03-11T19:55:44.669309+0000","last_unstale":"2024-03-11T19:57:46.802186+0000","last_undegraded":"2024-03-11T19:57:46.802186+0000","last_fullsized":"2024-03-11T19:57:46.802186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:06:41.489579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.574440+0000","last_change":"2024-03-11T19:55:45.731348+0000","last_active":"2024-03-11T19:57:47.574440+0000","last_peered":"2024-03-11T19:57:47.574440+0000","last_clean":"2024-03-11T19:57:47.574440+0000","last_became_active":"2024-03-11T19:55:45.730957+0000","last_became_peered":"2024-03-11T19:55:45.730957+0000","last_unstale":"2024-03-11T19:57:47.574440+0000","last_undegraded":"2024-03-11T19:57:47.574440+0000","last_fullsized":"2024-03-11T19:57:47.574440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:06:50.422511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684428+0000","last_change":"2024-03-11T19:55:50.144050+0000","last_active":"2024-03-11T19:57:46.684428+0000","last_peered":"2024-03-11T19:57:46.684428+0000","last_clean":"2024-03-11T19:57:46.684428+0000","last_became_active":"2024-03-11T19:55:50.143591+0000","last_became_peered":"2024-03-11T19:55:50.143591+0000","last_unstale":"2024-03-11T19:57:46.684428+0000","last_undegraded":"2024-03-11T19:57:46.684428+0000","last_fullsized":"2024-03-11T19:57:46.684428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:02.006665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685506+0000","last_change":"2024-03-11T19:55:44.671973+0000","last_active":"2024-03-11T19:57:46.685506+0000","last_peered":"2024-03-11T19:57:46.685506+0000","last_clean":"2024-03-11T19:57:46.685506+0000","last_became_active":"2024-03-11T19:55:44.671661+0000","last_became_peered":"2024-03-11T19:55:44.671661+0000","last_unstale":"2024-03-11T19:57:46.685506+0000","last_undegraded":"2024-03-11T19:57:46.685506+0000","last_fullsized":"2024-03-11T19:57:46.685506+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:35.747386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684764+0000","last_change":"2024-03-11T19:55:45.705545+0000","last_active":"2024-03-11T19:57:46.684764+0000","last_peered":"2024-03-11T19:57:46.684764+0000","last_clean":"2024-03-11T19:57:46.684764+0000","last_became_active":"2024-03-11T19:55:45.705083+0000","last_became_peered":"2024-03-11T19:55:45.705083+0000","last_unstale":"2024-03-11T19:57:46.684764+0000","last_undegraded":"2024-03-11T19:57:46.684764+0000","last_fullsized":"2024-03-11T19:57:46.684764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:02:58.219026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.022984+0000","last_change":"2024-03-11T19:55:50.097951+0000","last_active":"2024-03-11T19:57:47.022984+0000","last_peered":"2024-03-11T19:57:47.022984+0000","last_clean":"2024-03-11T19:57:47.022984+0000","last_became_active":"2024-03-11T19:55:50.097446+0000","last_became_peered":"2024-03-11T19:55:50.097446+0000","last_unstale":"2024-03-11T19:57:47.022984+0000","last_undegraded":"2024-03-11T19:57:47.022984+0000","last_fullsized":"2024-03-11T19:57:47.022984+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:12.105801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'7","reported_seq":136,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:50.811575+0000","last_change":"2024-03-11T19:55:44.673136+0000","last_active":"2024-03-11T19:57:50.811575+0000","last_peered":"2024-03-11T19:57:50.811575+0000","last_clean":"2024-03-11T19:57:50.811575+0000","last_became_active":"2024-03-11T19:55:44.672467+0000","last_became_peered":"2024-03-11T19:55:44.672467+0000","last_unstale":"2024-03-11T19:57:50.811575+0000","last_undegraded":"2024-03-11T19:57:50.811575+0000","last_fullsized":"2024-03-11T19:57:50.811575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:46:05.931253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":98,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685595+0000","last_change":"2024-03-11T19:55:45.710561+0000","last_active":"2024-03-11T19:57:46.685595+0000","last_peered":"2024-03-11T19:57:46.685595+0000","last_clean":"2024-03-11T19:57:46.685595+0000","last_became_active":"2024-03-11T19:55:45.710164+0000","last_became_peered":"2024-03-11T19:55:45.710164+0000","last_unstale":"2024-03-11T19:57:46.685595+0000","last_undegraded":"2024-03-11T19:57:46.685595+0000","last_fullsized":"2024-03-11T19:57:46.685595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:00:33.977595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.880197+0000","last_change":"2024-03-11T19:55:50.113390+0000","last_active":"2024-03-11T19:57:46.880197+0000","last_peered":"2024-03-11T19:57:46.880197+0000","last_clean":"2024-03-11T19:57:46.880197+0000","last_became_active":"2024-03-11T19:55:50.112656+0000","last_became_peered":"2024-03-11T19:55:50.112656+0000","last_unstale":"2024-03-11T19:57:46.880197+0000","last_undegraded":"2024-03-11T19:57:46.880197+0000","last_fullsized":"2024-03-11T19:57:46.880197+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:11.952585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680977+0000","last_change":"2024-03-11T19:55:44.684907+0000","last_active":"2024-03-11T19:57:46.680977+0000","last_peered":"2024-03-11T19:57:46.680977+0000","last_clean":"2024-03-11T19:57:46.680977+0000","last_became_active":"2024-03-11T19:55:44.684600+0000","last_became_peered":"2024-03-11T19:55:44.684600+0000","last_unstale":"2024-03-11T19:57:46.680977+0000","last_undegraded":"2024-03-11T19:57:46.680977+0000","last_fullsized":"2024-03-11T19:57:46.680977+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:12:33.666756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685169+0000","last_change":"2024-03-11T19:55:45.723431+0000","last_active":"2024-03-11T19:57:46.685169+0000","last_peered":"2024-03-11T19:57:46.685169+0000","last_clean":"2024-03-11T19:57:46.685169+0000","last_became_active":"2024-03-11T19:55:45.723135+0000","last_became_peered":"2024-03-11T19:55:45.723135+0000","last_unstale":"2024-03-11T19:57:46.685169+0000","last_undegraded":"2024-03-11T19:57:46.685169+0000","last_fullsized":"2024-03-11T19:57:46.685169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:12:35.654094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680102+0000","last_change":"2024-03-11T19:55:50.092876+0000","last_active":"2024-03-11T19:57:46.680102+0000","last_peered":"2024-03-11T19:57:46.680102+0000","last_clean":"2024-03-11T19:57:46.680102+0000","last_became_active":"2024-03-11T19:55:50.092478+0000","last_became_peered":"2024-03-11T19:55:50.092478+0000","last_unstale":"2024-03-11T19:57:46.680102+0000","last_undegraded":"2024-03-11T19:57:46.680102+0000","last_fullsized":"2024-03-11T19:57:46.680102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:39.248325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684886+0000","last_change":"2024-03-11T19:56:45.711269+0000","last_active":"2024-03-11T19:57:46.684886+0000","last_peered":"2024-03-11T19:57:46.684886+0000","last_clean":"2024-03-11T19:57:46.684886+0000","last_became_active":"2024-03-11T19:56:45.704657+0000","last_became_peered":"2024-03-11T19:56:45.704657+0000","last_unstale":"2024-03-11T19:57:46.684886+0000","last_undegraded":"2024-03-11T19:57:46.684886+0000","last_fullsized":"2024-03-11T19:57:46.684886+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:51.549965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685910+0000","last_change":"2024-03-11T19:55:45.702924+0000","last_active":"2024-03-11T19:57:46.685910+0000","last_peered":"2024-03-11T19:57:46.685910+0000","last_clean":"2024-03-11T19:57:46.685910+0000","last_became_active":"2024-03-11T19:55:45.702530+0000","last_became_peered":"2024-03-11T19:55:45.702530+0000","last_unstale":"2024-03-11T19:57:46.685910+0000","last_undegraded":"2024-03-11T19:57:46.685910+0000","last_fullsized":"2024-03-11T19:57:46.685910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:14.679964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686802+0000","last_change":"2024-03-11T19:55:45.706123+0000","last_active":"2024-03-11T19:57:46.686802+0000","last_peered":"2024-03-11T19:57:46.686802+0000","last_clean":"2024-03-11T19:57:46.686802+0000","last_became_active":"2024-03-11T19:55:45.705706+0000","last_became_peered":"2024-03-11T19:55:45.705706+0000","last_unstale":"2024-03-11T19:57:46.686802+0000","last_undegraded":"2024-03-11T19:57:46.686802+0000","last_fullsized":"2024-03-11T19:57:46.686802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:42:58.989823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802706+0000","last_change":"2024-03-11T19:55:45.703188+0000","last_active":"2024-03-11T19:57:46.802706+0000","last_peered":"2024-03-11T19:57:46.802706+0000","last_clean":"2024-03-11T19:57:46.802706+0000","last_became_active":"2024-03-11T19:55:45.702803+0000","last_became_peered":"2024-03-11T19:55:45.702803+0000","last_unstale":"2024-03-11T19:57:46.802706+0000","last_undegraded":"2024-03-11T19:57:46.802706+0000","last_fullsized":"2024-03-11T19:57:46.802706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:20.922963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802270+0000","last_change":"2024-03-11T19:55:45.728856+0000","last_active":"2024-03-11T19:57:46.802270+0000","last_peered":"2024-03-11T19:57:46.802270+0000","last_clean":"2024-03-11T19:57:46.802270+0000","last_became_active":"2024-03-11T19:55:45.728526+0000","last_became_peered":"2024-03-11T19:55:45.728526+0000","last_unstale":"2024-03-11T19:57:46.802270+0000","last_undegraded":"2024-03-11T19:57:46.802270+0000","last_fullsized":"2024-03-11T19:57:46.802270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:25:30.836073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881301+0000","last_change":"2024-03-11T19:55:45.725075+0000","last_active":"2024-03-11T19:57:46.881301+0000","last_peered":"2024-03-11T19:57:46.881301+0000","last_clean":"2024-03-11T19:57:46.881301+0000","last_became_active":"2024-03-11T19:55:45.724627+0000","last_became_peered":"2024-03-11T19:55:45.724627+0000","last_unstale":"2024-03-11T19:57:46.881301+0000","last_undegraded":"2024-03-11T19:57:46.881301+0000","last_fullsized":"2024-03-11T19:57:46.881301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:01.039660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684561+0000","last_change":"2024-03-11T19:55:45.718879+0000","last_active":"2024-03-11T19:57:46.684561+0000","last_peered":"2024-03-11T19:57:46.684561+0000","last_clean":"2024-03-11T19:57:46.684561+0000","last_became_active":"2024-03-11T19:55:45.718535+0000","last_became_peered":"2024-03-11T19:55:45.718535+0000","last_unstale":"2024-03-11T19:57:46.684561+0000","last_undegraded":"2024-03-11T19:57:46.684561+0000","last_fullsized":"2024-03-11T19:57:46.684561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:48.236368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687310+0000","last_change":"2024-03-11T19:55:45.703432+0000","last_active":"2024-03-11T19:57:46.687310+0000","last_peered":"2024-03-11T19:57:46.687310+0000","last_clean":"2024-03-11T19:57:46.687310+0000","last_became_active":"2024-03-11T19:55:45.702669+0000","last_became_peered":"2024-03-11T19:55:45.702669+0000","last_unstale":"2024-03-11T19:57:46.687310+0000","last_undegraded":"2024-03-11T19:57:46.687310+0000","last_fullsized":"2024-03-11T19:57:46.687310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:02.496981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802680+0000","last_change":"2024-03-11T19:55:45.708733+0000","last_active":"2024-03-11T19:57:46.802680+0000","last_peered":"2024-03-11T19:57:46.802680+0000","last_clean":"2024-03-11T19:57:46.802680+0000","last_became_active":"2024-03-11T19:55:45.708369+0000","last_became_peered":"2024-03-11T19:55:45.708369+0000","last_unstale":"2024-03-11T19:57:46.802680+0000","last_undegraded":"2024-03-11T19:57:46.802680+0000","last_fullsized":"2024-03-11T19:57:46.802680+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:36.619802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573994+0000","last_change":"2024-03-11T19:55:45.725698+0000","last_active":"2024-03-11T19:57:47.573994+0000","last_peered":"2024-03-11T19:57:47.573994+0000","last_clean":"2024-03-11T19:57:47.573994+0000","last_became_active":"2024-03-11T19:55:45.724872+0000","last_became_peered":"2024-03-11T19:55:45.724872+0000","last_unstale":"2024-03-11T19:57:47.573994+0000","last_undegraded":"2024-03-11T19:57:47.573994+0000","last_fullsized":"2024-03-11T19:57:47.573994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.982969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680237+0000","last_change":"2024-03-11T19:55:45.726923+0000","last_active":"2024-03-11T19:57:46.680237+0000","last_peered":"2024-03-11T19:57:46.680237+0000","last_clean":"2024-03-11T19:57:46.680237+0000","last_became_active":"2024-03-11T19:55:45.726589+0000","last_became_peered":"2024-03-11T19:55:45.726589+0000","last_unstale":"2024-03-11T19:57:46.680237+0000","last_undegraded":"2024-03-11T19:57:46.680237+0000","last_fullsized":"2024-03-11T19:57:46.680237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:29:15.197359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.690624+0000","last_change":"2024-03-11T19:56:45.698623+0000","last_active":"2024-03-11T19:57:46.690624+0000","last_peered":"2024-03-11T19:57:46.690624+0000","last_clean":"2024-03-11T19:57:46.690624+0000","last_became_active":"2024-03-11T19:56:45.697805+0000","last_became_peered":"2024-03-11T19:56:45.697805+0000","last_unstale":"2024-03-11T19:57:46.690624+0000","last_undegraded":"2024-03-11T19:57:46.690624+0000","last_fullsized":"2024-03-11T19:57:46.690624+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:16:31.624856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023161+0000","last_change":"2024-03-11T19:55:45.729141+0000","last_active":"2024-03-11T19:57:47.023161+0000","last_peered":"2024-03-11T19:57:47.023161+0000","last_clean":"2024-03-11T19:57:47.023161+0000","last_became_active":"2024-03-11T19:55:45.728781+0000","last_became_peered":"2024-03-11T19:55:45.728781+0000","last_unstale":"2024-03-11T19:57:47.023161+0000","last_undegraded":"2024-03-11T19:57:47.023161+0000","last_fullsized":"2024-03-11T19:57:47.023161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:02.939181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683974+0000","last_change":"2024-03-11T19:55:45.723892+0000","last_active":"2024-03-11T19:57:46.683974+0000","last_peered":"2024-03-11T19:57:46.683974+0000","last_clean":"2024-03-11T19:57:46.683974+0000","last_became_active":"2024-03-11T19:55:45.723544+0000","last_became_peered":"2024-03-11T19:55:45.723544+0000","last_unstale":"2024-03-11T19:57:46.683974+0000","last_undegraded":"2024-03-11T19:57:46.683974+0000","last_fullsized":"2024-03-11T19:57:46.683974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:09:08.720854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687440+0000","last_change":"2024-03-11T19:56:45.699717+0000","last_active":"2024-03-11T19:57:46.687440+0000","last_peered":"2024-03-11T19:57:46.687440+0000","last_clean":"2024-03-11T19:57:46.687440+0000","last_became_active":"2024-03-11T19:56:45.699346+0000","last_became_peered":"2024-03-11T19:56:45.699346+0000","last_unstale":"2024-03-11T19:57:46.687440+0000","last_undegraded":"2024-03-11T19:57:46.687440+0000","last_fullsized":"2024-03-11T19:57:46.687440+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:52.920371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68957,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":974,"num_read_kb":563,"num_write":250,"num_write_kb":145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":206871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":217,"ondisk_log_size":217,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383721,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1069056,"data_stored":521390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645357,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32904,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712248,"statfs":{"total":95995035648,"available":95961341952,"internally_reserved":0,"allocated":1761280,"data_stored":1172659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1093632,"data_stored":524302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070523,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":520002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36984,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708168,"statfs":{"total":95995035648,"available":95957164032,"internally_reserved":0,"allocated":1683456,"data_stored":1117222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":515710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1130496,"data_stored":578211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":517698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36464,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708688,"statfs":{"total":95995035648,"available":95957696512,"internally_reserved":0,"allocated":1150976,"data_stored":579495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182486,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":515418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":519784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36992,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708160,"statfs":{"total":95995035648,"available":95957155840,"internally_reserved":0,"allocated":1687552,"data_stored":1108708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:57:56.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph status --format=json 2024-03-11T19:57:57.438 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:57:57.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:57 smithi097 ceph-mon[31898]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:57:57.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:57 smithi097 ceph-mon[31898]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 614 B/s wr, 10 op/s; 11 B/s, 0 objects/s recovering 2024-03-11T19:57:58.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:57 smithi161 ceph-mon[36332]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:57:58.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:57 smithi161 ceph-mon[36332]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 614 B/s wr, 10 op/s; 11 B/s, 0 objects/s recovering 2024-03-11T19:57:58.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:57 smithi114 ceph-mon[36376]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:57:58.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:57 smithi114 ceph-mon[36376]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 614 B/s wr, 10 op/s; 11 B/s, 0 objects/s recovering 2024-03-11T19:57:59.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.677+0000 7fa7f5400700 1 -- 172.21.15.97:0/3661667305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7f006a410 msgr2=0x7fa7f006a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:59.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.677+0000 7fa7f5400700 1 --2- 172.21.15.97:0/3661667305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7f006a410 0x7fa7f006a7f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009990 tx=0x7fa7e0009ca0 comp rx=0 tx=0).stop 2024-03-11T19:57:59.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.677+0000 7fa7f5400700 1 -- 172.21.15.97:0/3661667305 shutdown_connections 2024-03-11T19:57:59.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.677+0000 7fa7f5400700 1 --2- 172.21.15.97:0/3661667305 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7f006a410 0x7fa7f006a7f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:59.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.677+0000 7fa7f5400700 1 --2- 172.21.15.97:0/3661667305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7f0106ff0 0x7fa7f010eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:59.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.677+0000 7fa7f5400700 1 --2- 172.21.15.97:0/3661667305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7f006adc0 0x7fa7f01068c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:59.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.677+0000 7fa7f5400700 1 -- 172.21.15.97:0/3661667305 >> 172.21.15.97:0/3661667305 conn(0x7fa7f0076bc0 msgr2=0x7fa7f0076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:57:59.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.677+0000 7fa7f5400700 1 -- 172.21.15.97:0/3661667305 shutdown_connections 2024-03-11T19:57:59.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.677+0000 7fa7f5400700 1 -- 172.21.15.97:0/3661667305 wait complete. 2024-03-11T19:57:59.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.678+0000 7fa7f5400700 1 Processor -- start 2024-03-11T19:57:59.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.678+0000 7fa7f5400700 1 -- start start 2024-03-11T19:57:59.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.678+0000 7fa7f5400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7f006a410 0x7fa7f019a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:59.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.679+0000 7fa7f5400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7f006adc0 0x7fa7f019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:59.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.679+0000 7fa7f5400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7f0106ff0 0x7fa7f019ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:59.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.679+0000 7fa7f5400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f019f300 con 0x7fa7f006adc0 2024-03-11T19:57:59.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.679+0000 7fa7f5400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f019f470 con 0x7fa7f0106ff0 2024-03-11T19:57:59.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.679+0000 7fa7f5400700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f019f630 con 0x7fa7f006a410 2024-03-11T19:57:59.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.679+0000 7fa7ede00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7f006adc0 0x7fa7f019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:59.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.679+0000 7fa7ee800700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7f006a410 0x7fa7f019a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:59.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.679+0000 7fa7ef200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7f0106ff0 0x7fa7f019ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.679+0000 7fa7ee800700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7f006a410 0x7fa7f019a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:44158/0 (socket says 172.21.15.97:44158) 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.679+0000 7fa7ef200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7f0106ff0 0x7fa7f019ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:56956/0 (socket says 172.21.15.97:56956) 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.679+0000 7fa7ee800700 1 -- 172.21.15.97:0/4043791278 learned_addr learned my addr 172.21.15.97:0/4043791278 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.680+0000 7fa7ef200700 1 -- 172.21.15.97:0/4043791278 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7f006a410 msgr2=0x7fa7f019a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.680+0000 7fa7ef200700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7f006a410 0x7fa7f019a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.680+0000 7fa7ef200700 1 -- 172.21.15.97:0/4043791278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7f006adc0 msgr2=0x7fa7f019a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.680+0000 7fa7ef200700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7f006adc0 0x7fa7f019a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.680+0000 7fa7ef200700 1 -- 172.21.15.97:0/4043791278 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e00095f0 con 0x7fa7f0106ff0 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.680+0000 7fa7ee800700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7f006a410 0x7fa7f019a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.680+0000 7fa7ede00700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7f006adc0 0x7fa7f019a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.680+0000 7fa7ef200700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7f0106ff0 0x7fa7f019ecf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa7e800efc0 tx=0x7fa7e800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.680+0000 7fa7e7400700 1 -- 172.21.15.97:0/4043791278 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e8018070 con 0x7fa7f0106ff0 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.680+0000 7fa7e7400700 1 -- 172.21.15.97:0/4043791278 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7e800f040 con 0x7fa7f0106ff0 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.680+0000 7fa7f5400700 1 -- 172.21.15.97:0/4043791278 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f019f8b0 con 0x7fa7f0106ff0 2024-03-11T19:57:59.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.680+0000 7fa7e7400700 1 -- 172.21.15.97:0/4043791278 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e800cd10 con 0x7fa7f0106ff0 2024-03-11T19:57:59.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.680+0000 7fa7f5400700 1 -- 172.21.15.97:0/4043791278 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f01a7170 con 0x7fa7f0106ff0 2024-03-11T19:57:59.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.682+0000 7fa7f5400700 1 -- 172.21.15.97:0/4043791278 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f0068930 con 0x7fa7f0106ff0 2024-03-11T19:57:59.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.683+0000 7fa7e7400700 1 -- 172.21.15.97:0/4043791278 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa7e8004ad0 con 0x7fa7f0106ff0 2024-03-11T19:57:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.685+0000 7fa7e7400700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa7d40a1ac0 0x7fa7d40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:57:59.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.685+0000 7fa7e7400700 1 -- 172.21.15.97:0/4043791278 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fa7e80cd2c0 con 0x7fa7f0106ff0 2024-03-11T19:57:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.685+0000 7fa7ee800700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa7d40a1ac0 0x7fa7d40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:57:59.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.686+0000 7fa7ee800700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa7d40a1ac0 0x7fa7d40a3f80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00098e0 tx=0x7fa7e000b2a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:57:59.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.689+0000 7fa7e7400700 1 -- 172.21.15.97:0/4043791278 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa7e8096140 con 0x7fa7f0106ff0 2024-03-11T19:57:59.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.998+0000 7fa7f5400700 1 -- 172.21.15.97:0/4043791278 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa7f0111d50 con 0x7fa7f0106ff0 2024-03-11T19:57:59.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:57:59.999+0000 7fa7e7400700 1 -- 172.21.15.97:0/4043791278 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1968 (secure 0 0 0) 0x7fa7e8099800 con 0x7fa7f0106ff0 2024-03-11T19:57:59.998 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:57:59.998 INFO:teuthology.orchestra.run.smithi097.stdout:{"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":544,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":93,"num_osds":12,"num_up_osds":12,"osd_up_since":1710186883,"num_in_osds":12,"osd_in_since":1710186862,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":129}],"num_pgs":129,"num_pools":4,"num_objects":102,"data_bytes":659559,"bytes_used":393859072,"bytes_avail":1151546568704,"bytes_total":1151940427776,"recovering_objects_per_sec":0,"recovering_bytes_per_sec":9,"recovering_keys_per_sec":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"read_bytes_sec":5777,"write_bytes_sec":517,"read_op_per_sec":11,"write_op_per_sec":0},"fsmap":{"epoch":28,"id":1,"up":5,"in":5,"max":5,"by_rank":[{"filesystem_id":1,"rank":4,"name":"l","status":"up:standby-replay","gid":24454},{"filesystem_id":1,"rank":4,"name":"k","status":"up:active","gid":24464},{"filesystem_id":1,"rank":3,"name":"d","status":"up:standby-replay","gid":24499},{"filesystem_id":1,"rank":2,"name":"e","status":"up:standby-replay","gid":24512},{"filesystem_id":1,"rank":1,"name":"f","status":"up:standby-replay","gid":24517},{"filesystem_id":1,"rank":0,"name":"c","status":"up:standby-replay","gid":24529},{"filesystem_id":1,"rank":3,"name":"h","status":"up:active","gid":24545},{"filesystem_id":1,"rank":2,"name":"i","status":"up:active","gid":24547},{"filesystem_id":1,"rank":1,"name":"a","status":"up:active","gid":24583},{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":24599}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":2,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.97:8443/"}},"servicemap":{"epoch":30,"modified":"2024-03-11T19:57:26.539873+0000","services":{}},"progress_events":{}} 2024-03-11T19:58:00.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:00.001+0000 7fa7f5400700 1 -- 172.21.15.97:0/4043791278 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa7d40a1ac0 msgr2=0x7fa7d40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:00.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:00.001+0000 7fa7f5400700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa7d40a1ac0 0x7fa7d40a3f80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00098e0 tx=0x7fa7e000b2a0 comp rx=0 tx=0).stop 2024-03-11T19:58:00.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:00.001+0000 7fa7f5400700 1 -- 172.21.15.97:0/4043791278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7f0106ff0 msgr2=0x7fa7f019ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:00.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:00.001+0000 7fa7f5400700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7f0106ff0 0x7fa7f019ecf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa7e800efc0 tx=0x7fa7e800c3a0 comp rx=0 tx=0).stop 2024-03-11T19:58:00.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:00.002+0000 7fa7f5400700 1 -- 172.21.15.97:0/4043791278 shutdown_connections 2024-03-11T19:58:00.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:00.002+0000 7fa7f5400700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fa7d40a1ac0 0x7fa7d40a3f80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:00.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:00.002+0000 7fa7f5400700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa7f0106ff0 0x7fa7f019ecf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:00.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:00.002+0000 7fa7f5400700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa7f006a410 0x7fa7f019a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:00.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:00.002+0000 7fa7f5400700 1 --2- 172.21.15.97:0/4043791278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa7f006adc0 0x7fa7f019a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:00.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:00.002+0000 7fa7f5400700 1 -- 172.21.15.97:0/4043791278 >> 172.21.15.97:0/4043791278 conn(0x7fa7f0076bc0 msgr2=0x7fa7f00fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:00.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:00.002+0000 7fa7f5400700 1 -- 172.21.15.97:0/4043791278 shutdown_connections 2024-03-11T19:58:00.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:00.002+0000 7fa7f5400700 1 -- 172.21.15.97:0/4043791278 wait complete. 2024-03-11T19:58:00.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:57:59 smithi161 ceph-mon[36332]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 517 B/s wr, 12 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T19:58:00.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:57:59 smithi097 ceph-mon[31898]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 517 B/s wr, 12 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T19:58:00.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:57:59 smithi114 ceph-mon[36376]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 517 B/s wr, 12 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T19:58:00.858 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-03-11T19:58:00.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T19:58:01.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:00 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/4043791278' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T19:58:01.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:00 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/4043791278' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T19:58:01.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:00 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/4043791278' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T19:58:01.307 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:58:02.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:01 smithi161 ceph-mon[36332]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:02.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:01 smithi097 ceph-mon[31898]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:02.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:01 smithi114 ceph-mon[36376]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:02.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:02 smithi097 ceph-mon[31898]: pgmap v392: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:03.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:02 smithi114 ceph-mon[36376]: pgmap v392: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:03.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:02 smithi161 ceph-mon[36332]: pgmap v392: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:03.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.584+0000 7fd069e00700 1 -- 172.21.15.97:0/3010524906 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd064073f90 msgr2=0x7fd064074370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:03.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.584+0000 7fd069e00700 1 --2- 172.21.15.97:0/3010524906 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd064073f90 0x7fd064074370 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd054009990 tx=0x7fd054009ca0 comp rx=0 tx=0).stop 2024-03-11T19:58:03.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.584+0000 7fd069e00700 1 -- 172.21.15.97:0/3010524906 shutdown_connections 2024-03-11T19:58:03.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.584+0000 7fd069e00700 1 --2- 172.21.15.97:0/3010524906 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd064073f90 0x7fd064074370 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:03.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.584+0000 7fd069e00700 1 --2- 172.21.15.97:0/3010524906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06410e550 0x7fd064110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:03.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.584+0000 7fd069e00700 1 --2- 172.21.15.97:0/3010524906 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0640748b0 0x7fd06410e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:03.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.584+0000 7fd069e00700 1 -- 172.21.15.97:0/3010524906 >> 172.21.15.97:0/3010524906 conn(0x7fd0640fd740 msgr2=0x7fd0640ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:03.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.585+0000 7fd069e00700 1 -- 172.21.15.97:0/3010524906 shutdown_connections 2024-03-11T19:58:03.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.585+0000 7fd069e00700 1 -- 172.21.15.97:0/3010524906 wait complete. 2024-03-11T19:58:03.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.586+0000 7fd069e00700 1 Processor -- start 2024-03-11T19:58:03.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.586+0000 7fd069e00700 1 -- start start 2024-03-11T19:58:03.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.586+0000 7fd069e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd064073f90 0x7fd06419a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:03.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.586+0000 7fd069e00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0640748b0 0x7fd06419a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:03.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.586+0000 7fd069e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06410e550 0x7fd06419ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:03.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.586+0000 7fd069e00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06419f290 con 0x7fd064073f90 2024-03-11T19:58:03.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.586+0000 7fd069e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06419f400 con 0x7fd06410e550 2024-03-11T19:58:03.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.586+0000 7fd069e00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06419f5c0 con 0x7fd0640748b0 2024-03-11T19:58:03.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.586+0000 7fd062a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd064073f90 0x7fd06419a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:03.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.586+0000 7fd062a00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd064073f90 0x7fd06419a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39394/0 (socket says 172.21.15.97:39394) 2024-03-11T19:58:03.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.587+0000 7fd062a00700 1 -- 172.21.15.97:0/722576191 learned_addr learned my addr 172.21.15.97:0/722576191 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:03.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.587+0000 7fd062000700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0640748b0 0x7fd06419a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:03.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.587+0000 7fd063400700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06410e550 0x7fd06419ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:03.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.587+0000 7fd062000700 1 -- 172.21.15.97:0/722576191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06410e550 msgr2=0x7fd06419ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:03.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.587+0000 7fd062000700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06410e550 0x7fd06419ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:03.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.587+0000 7fd062000700 1 -- 172.21.15.97:0/722576191 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd064073f90 msgr2=0x7fd06419a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:03.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.587+0000 7fd062000700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd064073f90 0x7fd06419a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:03.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.587+0000 7fd062000700 1 -- 172.21.15.97:0/722576191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0540095f0 con 0x7fd0640748b0 2024-03-11T19:58:03.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.587+0000 7fd062a00700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd064073f90 0x7fd06419a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:03.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.587+0000 7fd062000700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0640748b0 0x7fd06419a8f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd05000e9f0 tx=0x7fd05000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:03.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.588+0000 7fd068a00700 1 -- 172.21.15.97:0/722576191 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05000c9c0 con 0x7fd0640748b0 2024-03-11T19:58:03.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.588+0000 7fd069e00700 1 -- 172.21.15.97:0/722576191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06419f8a0 con 0x7fd0640748b0 2024-03-11T19:58:03.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.588+0000 7fd068a00700 1 -- 172.21.15.97:0/722576191 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd050010430 con 0x7fd0640748b0 2024-03-11T19:58:03.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.588+0000 7fd068a00700 1 -- 172.21.15.97:0/722576191 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd050018680 con 0x7fd0640748b0 2024-03-11T19:58:03.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.588+0000 7fd069e00700 1 -- 172.21.15.97:0/722576191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0641a7100 con 0x7fd0640748b0 2024-03-11T19:58:03.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.589+0000 7fd069e00700 1 -- 172.21.15.97:0/722576191 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd064109760 con 0x7fd0640748b0 2024-03-11T19:58:03.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.591+0000 7fd068a00700 1 -- 172.21.15.97:0/722576191 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd0500105a0 con 0x7fd0640748b0 2024-03-11T19:58:03.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.592+0000 7fd068a00700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd0480a1b40 0x7fd0480a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:03.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.592+0000 7fd068a00700 1 -- 172.21.15.97:0/722576191 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fd050014070 con 0x7fd0640748b0 2024-03-11T19:58:03.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.593+0000 7fd062a00700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd0480a1b40 0x7fd0480a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:03.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.593+0000 7fd062a00700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd0480a1b40 0x7fd0480a4000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd054006010 tx=0x7fd05400b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:03.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.597+0000 7fd068a00700 1 -- 172.21.15.97:0/722576191 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd050052d20 con 0x7fd0640748b0 2024-03-11T19:58:03.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.811+0000 7fd069e00700 1 -- 172.21.15.97:0/722576191 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd064113f60 con 0x7fd0480a1b40 2024-03-11T19:58:03.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.823+0000 7fd068a00700 1 -- 172.21.15.97:0/722576191 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340539 (secure 0 0 0) 0x7fd064113f60 con 0x7fd0480a1b40 2024-03-11T19:58:03.829 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:03.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.832+0000 7fd069e00700 1 -- 172.21.15.97:0/722576191 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd0480a1b40 msgr2=0x7fd0480a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:03.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.832+0000 7fd069e00700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd0480a1b40 0x7fd0480a4000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd054006010 tx=0x7fd05400b280 comp rx=0 tx=0).stop 2024-03-11T19:58:03.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.832+0000 7fd069e00700 1 -- 172.21.15.97:0/722576191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0640748b0 msgr2=0x7fd06419a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:03.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.832+0000 7fd069e00700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0640748b0 0x7fd06419a8f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd05000e9f0 tx=0x7fd05000ed00 comp rx=0 tx=0).stop 2024-03-11T19:58:03.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.832+0000 7fd069e00700 1 -- 172.21.15.97:0/722576191 shutdown_connections 2024-03-11T19:58:03.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.832+0000 7fd069e00700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd0480a1b40 0x7fd0480a4000 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:03.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.832+0000 7fd069e00700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd064073f90 0x7fd06419a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:03.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.832+0000 7fd069e00700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd06410e550 0x7fd06419ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:03.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.832+0000 7fd069e00700 1 --2- 172.21.15.97:0/722576191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0640748b0 0x7fd06419a8f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:03.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.832+0000 7fd069e00700 1 -- 172.21.15.97:0/722576191 >> 172.21.15.97:0/722576191 conn(0x7fd0640fd740 msgr2=0x7fd0640fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:03.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.832+0000 7fd069e00700 1 -- 172.21.15.97:0/722576191 shutdown_connections 2024-03-11T19:58:03.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:03.832+0000 7fd069e00700 1 -- 172.21.15.97:0/722576191 wait complete. 2024-03-11T19:58:03.833 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-11T19:58:04.661 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-03-11T19:58:02.556791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659559,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1200,"num_read_kb":732,"num_write":433,"num_write_kb":3085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1181435,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":856,"ondisk_log_size":856,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384628,"kb_used_data":14612,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557196,"statfs":{"total":1151940427776,"available":1151546568704,"internally_reserved":0,"allocated":14962688,"data_stored":8190599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":75,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006575"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687052+0000","last_change":"2024-03-11T19:55:45.725019+0000","last_active":"2024-03-11T19:57:46.687052+0000","last_peered":"2024-03-11T19:57:46.687052+0000","last_clean":"2024-03-11T19:57:46.687052+0000","last_became_active":"2024-03-11T19:55:45.724602+0000","last_became_peered":"2024-03-11T19:55:45.724602+0000","last_unstale":"2024-03-11T19:57:46.687052+0000","last_undegraded":"2024-03-11T19:57:46.687052+0000","last_fullsized":"2024-03-11T19:57:46.687052+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:16:18.218381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881861+0000","last_change":"2024-03-11T19:55:45.724589+0000","last_active":"2024-03-11T19:57:46.881861+0000","last_peered":"2024-03-11T19:57:46.881861+0000","last_clean":"2024-03-11T19:57:46.881861+0000","last_became_active":"2024-03-11T19:55:45.724109+0000","last_became_peered":"2024-03-11T19:55:45.724109+0000","last_unstale":"2024-03-11T19:57:46.881861+0000","last_undegraded":"2024-03-11T19:57:46.881861+0000","last_fullsized":"2024-03-11T19:57:46.881861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:58:12.510437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881733+0000","last_change":"2024-03-11T19:55:45.718831+0000","last_active":"2024-03-11T19:57:46.881733+0000","last_peered":"2024-03-11T19:57:46.881733+0000","last_clean":"2024-03-11T19:57:46.881733+0000","last_became_active":"2024-03-11T19:55:45.717652+0000","last_became_peered":"2024-03-11T19:55:45.717652+0000","last_unstale":"2024-03-11T19:57:46.881733+0000","last_undegraded":"2024-03-11T19:57:46.881733+0000","last_fullsized":"2024-03-11T19:57:46.881733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:16.471754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685116+0000","last_change":"2024-03-11T19:55:45.701640+0000","last_active":"2024-03-11T19:57:46.685116+0000","last_peered":"2024-03-11T19:57:46.685116+0000","last_clean":"2024-03-11T19:57:46.685116+0000","last_became_active":"2024-03-11T19:55:45.699207+0000","last_became_peered":"2024-03-11T19:55:45.699207+0000","last_unstale":"2024-03-11T19:57:46.685116+0000","last_undegraded":"2024-03-11T19:57:46.685116+0000","last_fullsized":"2024-03-11T19:57:46.685116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:01.900429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685627+0000","last_change":"2024-03-11T19:55:45.732040+0000","last_active":"2024-03-11T19:57:46.685627+0000","last_peered":"2024-03-11T19:57:46.685627+0000","last_clean":"2024-03-11T19:57:46.685627+0000","last_became_active":"2024-03-11T19:55:45.731647+0000","last_became_peered":"2024-03-11T19:55:45.731647+0000","last_unstale":"2024-03-11T19:57:46.685627+0000","last_undegraded":"2024-03-11T19:57:46.685627+0000","last_fullsized":"2024-03-11T19:57:46.685627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:51:57.345329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683200+0000","last_change":"2024-03-11T19:55:45.707986+0000","last_active":"2024-03-11T19:57:46.683200+0000","last_peered":"2024-03-11T19:57:46.683200+0000","last_clean":"2024-03-11T19:57:46.683200+0000","last_became_active":"2024-03-11T19:55:45.707641+0000","last_became_peered":"2024-03-11T19:55:45.707641+0000","last_unstale":"2024-03-11T19:57:46.683200+0000","last_undegraded":"2024-03-11T19:57:46.683200+0000","last_fullsized":"2024-03-11T19:57:46.683200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:15:50.635234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683168+0000","last_change":"2024-03-11T19:56:45.705737+0000","last_active":"2024-03-11T19:57:46.683168+0000","last_peered":"2024-03-11T19:57:46.683168+0000","last_clean":"2024-03-11T19:57:46.683168+0000","last_became_active":"2024-03-11T19:56:45.705477+0000","last_became_peered":"2024-03-11T19:56:45.705477+0000","last_unstale":"2024-03-11T19:57:46.683168+0000","last_undegraded":"2024-03-11T19:57:46.683168+0000","last_fullsized":"2024-03-11T19:57:46.683168+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:01:14.068476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572051+0000","last_change":"2024-03-11T19:55:45.729587+0000","last_active":"2024-03-11T19:57:47.572051+0000","last_peered":"2024-03-11T19:57:47.572051+0000","last_clean":"2024-03-11T19:57:47.572051+0000","last_became_active":"2024-03-11T19:55:45.729037+0000","last_became_peered":"2024-03-11T19:55:45.729037+0000","last_unstale":"2024-03-11T19:57:47.572051+0000","last_undegraded":"2024-03-11T19:57:47.572051+0000","last_fullsized":"2024-03-11T19:57:47.572051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:55:43.445192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023626+0000","last_change":"2024-03-11T19:55:45.709793+0000","last_active":"2024-03-11T19:57:47.023626+0000","last_peered":"2024-03-11T19:57:47.023626+0000","last_clean":"2024-03-11T19:57:47.023626+0000","last_became_active":"2024-03-11T19:55:45.708631+0000","last_became_peered":"2024-03-11T19:55:45.708631+0000","last_unstale":"2024-03-11T19:57:47.023626+0000","last_undegraded":"2024-03-11T19:57:47.023626+0000","last_fullsized":"2024-03-11T19:57:47.023626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:02.144066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684105+0000","last_change":"2024-03-11T19:55:45.704227+0000","last_active":"2024-03-11T19:57:46.684105+0000","last_peered":"2024-03-11T19:57:46.684105+0000","last_clean":"2024-03-11T19:57:46.684105+0000","last_became_active":"2024-03-11T19:55:45.703885+0000","last_became_peered":"2024-03-11T19:55:45.703885+0000","last_unstale":"2024-03-11T19:57:46.684105+0000","last_undegraded":"2024-03-11T19:57:46.684105+0000","last_fullsized":"2024-03-11T19:57:46.684105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:36:21.401988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685372+0000","last_change":"2024-03-11T19:55:45.708950+0000","last_active":"2024-03-11T19:57:46.685372+0000","last_peered":"2024-03-11T19:57:46.685372+0000","last_clean":"2024-03-11T19:57:46.685372+0000","last_became_active":"2024-03-11T19:55:45.708544+0000","last_became_peered":"2024-03-11T19:55:45.708544+0000","last_unstale":"2024-03-11T19:57:46.685372+0000","last_undegraded":"2024-03-11T19:57:46.685372+0000","last_fullsized":"2024-03-11T19:57:46.685372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:06:34.244449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685013+0000","last_change":"2024-03-11T19:55:45.703923+0000","last_active":"2024-03-11T19:57:46.685013+0000","last_peered":"2024-03-11T19:57:46.685013+0000","last_clean":"2024-03-11T19:57:46.685013+0000","last_became_active":"2024-03-11T19:55:45.703595+0000","last_became_peered":"2024-03-11T19:55:45.703595+0000","last_unstale":"2024-03-11T19:57:46.685013+0000","last_undegraded":"2024-03-11T19:57:46.685013+0000","last_fullsized":"2024-03-11T19:57:46.685013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:00:19.683907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683892+0000","last_change":"2024-03-11T19:55:45.707499+0000","last_active":"2024-03-11T19:57:46.683892+0000","last_peered":"2024-03-11T19:57:46.683892+0000","last_clean":"2024-03-11T19:57:46.683892+0000","last_became_active":"2024-03-11T19:55:45.706344+0000","last_became_peered":"2024-03-11T19:55:45.706344+0000","last_unstale":"2024-03-11T19:57:46.683892+0000","last_undegraded":"2024-03-11T19:57:46.683892+0000","last_fullsized":"2024-03-11T19:57:46.683892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:38:24.082332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.880311+0000","last_change":"2024-03-11T19:55:45.722182+0000","last_active":"2024-03-11T19:57:46.880311+0000","last_peered":"2024-03-11T19:57:46.880311+0000","last_clean":"2024-03-11T19:57:46.880311+0000","last_became_active":"2024-03-11T19:55:45.721783+0000","last_became_peered":"2024-03-11T19:55:45.721783+0000","last_unstale":"2024-03-11T19:57:46.880311+0000","last_undegraded":"2024-03-11T19:57:46.880311+0000","last_fullsized":"2024-03-11T19:57:46.880311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:41:08.559002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.880277+0000","last_change":"2024-03-11T19:55:45.718881+0000","last_active":"2024-03-11T19:57:46.880277+0000","last_peered":"2024-03-11T19:57:46.880277+0000","last_clean":"2024-03-11T19:57:46.880277+0000","last_became_active":"2024-03-11T19:55:45.718458+0000","last_became_peered":"2024-03-11T19:55:45.718458+0000","last_unstale":"2024-03-11T19:57:46.880277+0000","last_undegraded":"2024-03-11T19:57:46.880277+0000","last_fullsized":"2024-03-11T19:57:46.880277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:22.075892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023120+0000","last_change":"2024-03-11T19:55:45.726163+0000","last_active":"2024-03-11T19:57:47.023120+0000","last_peered":"2024-03-11T19:57:47.023120+0000","last_clean":"2024-03-11T19:57:47.023120+0000","last_became_active":"2024-03-11T19:55:45.725772+0000","last_became_peered":"2024-03-11T19:55:45.725772+0000","last_unstale":"2024-03-11T19:57:47.023120+0000","last_undegraded":"2024-03-11T19:57:47.023120+0000","last_fullsized":"2024-03-11T19:57:47.023120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:36:19.057920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686772+0000","last_change":"2024-03-11T19:55:45.724787+0000","last_active":"2024-03-11T19:57:46.686772+0000","last_peered":"2024-03-11T19:57:46.686772+0000","last_clean":"2024-03-11T19:57:46.686772+0000","last_became_active":"2024-03-11T19:55:45.724437+0000","last_became_peered":"2024-03-11T19:55:45.724437+0000","last_unstale":"2024-03-11T19:57:46.686772+0000","last_undegraded":"2024-03-11T19:57:46.686772+0000","last_fullsized":"2024-03-11T19:57:46.686772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:55:53.811679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.691041+0000","last_change":"2024-03-11T19:55:45.714934+0000","last_active":"2024-03-11T19:57:46.691041+0000","last_peered":"2024-03-11T19:57:46.691041+0000","last_clean":"2024-03-11T19:57:46.691041+0000","last_became_active":"2024-03-11T19:55:45.709595+0000","last_became_peered":"2024-03-11T19:55:45.709595+0000","last_unstale":"2024-03-11T19:57:46.691041+0000","last_undegraded":"2024-03-11T19:57:46.691041+0000","last_fullsized":"2024-03-11T19:57:46.691041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:48.319073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802786+0000","last_change":"2024-03-11T19:55:45.725747+0000","last_active":"2024-03-11T19:57:46.802786+0000","last_peered":"2024-03-11T19:57:46.802786+0000","last_clean":"2024-03-11T19:57:46.802786+0000","last_became_active":"2024-03-11T19:55:45.725239+0000","last_became_peered":"2024-03-11T19:55:45.725239+0000","last_unstale":"2024-03-11T19:57:46.802786+0000","last_undegraded":"2024-03-11T19:57:46.802786+0000","last_fullsized":"2024-03-11T19:57:46.802786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:51:24.046681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802300+0000","last_change":"2024-03-11T19:55:50.109913+0000","last_active":"2024-03-11T19:57:46.802300+0000","last_peered":"2024-03-11T19:57:46.802300+0000","last_clean":"2024-03-11T19:57:46.802300+0000","last_became_active":"2024-03-11T19:55:50.109471+0000","last_became_peered":"2024-03-11T19:55:50.109471+0000","last_unstale":"2024-03-11T19:57:46.802300+0000","last_undegraded":"2024-03-11T19:57:46.802300+0000","last_fullsized":"2024-03-11T19:57:46.802300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:40.258022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":147,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:59.865723+0000","last_change":"2024-03-11T19:55:44.680181+0000","last_active":"2024-03-11T19:57:59.865723+0000","last_peered":"2024-03-11T19:57:59.865723+0000","last_clean":"2024-03-11T19:57:59.865723+0000","last_became_active":"2024-03-11T19:55:44.679807+0000","last_became_peered":"2024-03-11T19:55:44.679807+0000","last_unstale":"2024-03-11T19:57:59.865723+0000","last_undegraded":"2024-03-11T19:57:59.865723+0000","last_fullsized":"2024-03-11T19:57:59.865723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:23:04.532836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":103,"num_read_kb":103,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573255+0000","last_change":"2024-03-11T19:55:45.723547+0000","last_active":"2024-03-11T19:57:47.573255+0000","last_peered":"2024-03-11T19:57:47.573255+0000","last_clean":"2024-03-11T19:57:47.573255+0000","last_became_active":"2024-03-11T19:55:45.722516+0000","last_became_peered":"2024-03-11T19:55:45.722516+0000","last_unstale":"2024-03-11T19:57:47.573255+0000","last_undegraded":"2024-03-11T19:57:47.573255+0000","last_fullsized":"2024-03-11T19:57:47.573255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:40.271814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683243+0000","last_change":"2024-03-11T19:55:50.102141+0000","last_active":"2024-03-11T19:57:46.683243+0000","last_peered":"2024-03-11T19:57:46.683243+0000","last_clean":"2024-03-11T19:57:46.683243+0000","last_became_active":"2024-03-11T19:55:50.101667+0000","last_became_peered":"2024-03-11T19:55:50.101667+0000","last_unstale":"2024-03-11T19:57:46.683243+0000","last_undegraded":"2024-03-11T19:57:46.683243+0000","last_fullsized":"2024-03-11T19:57:46.683243+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:54:11.453059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573605+0000","last_change":"2024-03-11T19:55:44.684623+0000","last_active":"2024-03-11T19:57:47.573605+0000","last_peered":"2024-03-11T19:57:47.573605+0000","last_clean":"2024-03-11T19:57:47.573605+0000","last_became_active":"2024-03-11T19:55:44.683611+0000","last_became_peered":"2024-03-11T19:55:44.683611+0000","last_unstale":"2024-03-11T19:57:47.573605+0000","last_undegraded":"2024-03-11T19:57:47.573605+0000","last_fullsized":"2024-03-11T19:57:47.573605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:22:50.954364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881778+0000","last_change":"2024-03-11T19:55:45.723619+0000","last_active":"2024-03-11T19:57:46.881778+0000","last_peered":"2024-03-11T19:57:46.881778+0000","last_clean":"2024-03-11T19:57:46.881778+0000","last_became_active":"2024-03-11T19:55:45.722685+0000","last_became_peered":"2024-03-11T19:55:45.722685+0000","last_unstale":"2024-03-11T19:57:46.881778+0000","last_undegraded":"2024-03-11T19:57:46.881778+0000","last_fullsized":"2024-03-11T19:57:46.881778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:47:13.747776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.576246+0000","last_change":"2024-03-11T19:57:47.576246+0000","last_active":"2024-03-11T19:57:47.576246+0000","last_peered":"2024-03-11T19:57:47.576246+0000","last_clean":"2024-03-11T19:57:47.576246+0000","last_became_active":"2024-03-11T19:57:47.575837+0000","last_became_peered":"2024-03-11T19:57:47.575837+0000","last_unstale":"2024-03-11T19:57:47.576246+0000","last_undegraded":"2024-03-11T19:57:47.576246+0000","last_fullsized":"2024-03-11T19:57:47.576246+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:00.914062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687185+0000","last_change":"2024-03-11T19:56:45.714194+0000","last_active":"2024-03-11T19:57:46.687185+0000","last_peered":"2024-03-11T19:57:46.687185+0000","last_clean":"2024-03-11T19:57:46.687185+0000","last_became_active":"2024-03-11T19:56:45.694559+0000","last_became_peered":"2024-03-11T19:56:45.694559+0000","last_unstale":"2024-03-11T19:57:46.687185+0000","last_undegraded":"2024-03-11T19:57:46.687185+0000","last_fullsized":"2024-03-11T19:57:46.687185+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:01:20.307915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023254+0000","last_change":"2024-03-11T19:55:45.715157+0000","last_active":"2024-03-11T19:57:47.023254+0000","last_peered":"2024-03-11T19:57:47.023254+0000","last_clean":"2024-03-11T19:57:47.023254+0000","last_became_active":"2024-03-11T19:55:45.714536+0000","last_became_peered":"2024-03-11T19:55:45.714536+0000","last_unstale":"2024-03-11T19:57:47.023254+0000","last_undegraded":"2024-03-11T19:57:47.023254+0000","last_fullsized":"2024-03-11T19:57:47.023254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:02.416736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.682631+0000","last_change":"2024-03-11T19:55:50.102157+0000","last_active":"2024-03-11T19:57:46.682631+0000","last_peered":"2024-03-11T19:57:46.682631+0000","last_clean":"2024-03-11T19:57:46.682631+0000","last_became_active":"2024-03-11T19:55:50.101715+0000","last_became_peered":"2024-03-11T19:55:50.101715+0000","last_unstale":"2024-03-11T19:57:46.682631+0000","last_undegraded":"2024-03-11T19:57:46.682631+0000","last_fullsized":"2024-03-11T19:57:46.682631+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:12:18.780708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687126+0000","last_change":"2024-03-11T19:56:45.698585+0000","last_active":"2024-03-11T19:57:46.687126+0000","last_peered":"2024-03-11T19:57:46.687126+0000","last_clean":"2024-03-11T19:57:46.687126+0000","last_became_active":"2024-03-11T19:56:45.692929+0000","last_became_peered":"2024-03-11T19:56:45.692929+0000","last_unstale":"2024-03-11T19:57:46.687126+0000","last_undegraded":"2024-03-11T19:57:46.687126+0000","last_fullsized":"2024-03-11T19:57:46.687126+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:43:11.158934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680889+0000","last_change":"2024-03-11T19:55:45.709973+0000","last_active":"2024-03-11T19:57:46.680889+0000","last_peered":"2024-03-11T19:57:46.680889+0000","last_clean":"2024-03-11T19:57:46.680889+0000","last_became_active":"2024-03-11T19:55:45.709645+0000","last_became_peered":"2024-03-11T19:55:45.709645+0000","last_unstale":"2024-03-11T19:57:46.680889+0000","last_undegraded":"2024-03-11T19:57:46.680889+0000","last_fullsized":"2024-03-11T19:57:46.680889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:16.516982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802538+0000","last_change":"2024-03-11T19:55:50.094093+0000","last_active":"2024-03-11T19:57:46.802538+0000","last_peered":"2024-03-11T19:57:46.802538+0000","last_clean":"2024-03-11T19:57:46.802538+0000","last_became_active":"2024-03-11T19:55:50.093653+0000","last_became_peered":"2024-03-11T19:55:50.093653+0000","last_unstale":"2024-03-11T19:57:46.802538+0000","last_undegraded":"2024-03-11T19:57:46.802538+0000","last_fullsized":"2024-03-11T19:57:46.802538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:28:41.668001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":144,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:59.789225+0000","last_change":"2024-03-11T19:55:44.671547+0000","last_active":"2024-03-11T19:57:59.789225+0000","last_peered":"2024-03-11T19:57:59.789225+0000","last_clean":"2024-03-11T19:57:59.789225+0000","last_became_active":"2024-03-11T19:55:44.671211+0000","last_became_peered":"2024-03-11T19:55:44.671211+0000","last_unstale":"2024-03-11T19:57:59.789225+0000","last_undegraded":"2024-03-11T19:57:59.789225+0000","last_fullsized":"2024-03-11T19:57:59.789225+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:18:46.261250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685626+0000","last_change":"2024-03-11T19:55:45.707968+0000","last_active":"2024-03-11T19:57:46.685626+0000","last_peered":"2024-03-11T19:57:46.685626+0000","last_clean":"2024-03-11T19:57:46.685626+0000","last_became_active":"2024-03-11T19:55:45.706998+0000","last_became_peered":"2024-03-11T19:55:45.706998+0000","last_unstale":"2024-03-11T19:57:46.685626+0000","last_undegraded":"2024-03-11T19:57:46.685626+0000","last_fullsized":"2024-03-11T19:57:46.685626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:49:35.711123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802348+0000","last_change":"2024-03-11T19:55:50.104310+0000","last_active":"2024-03-11T19:57:46.802348+0000","last_peered":"2024-03-11T19:57:46.802348+0000","last_clean":"2024-03-11T19:57:46.802348+0000","last_became_active":"2024-03-11T19:55:50.103831+0000","last_became_peered":"2024-03-11T19:55:50.103831+0000","last_unstale":"2024-03-11T19:57:46.802348+0000","last_undegraded":"2024-03-11T19:57:46.802348+0000","last_fullsized":"2024-03-11T19:57:46.802348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:33.213987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683083+0000","last_change":"2024-03-11T19:55:44.677615+0000","last_active":"2024-03-11T19:57:46.683083+0000","last_peered":"2024-03-11T19:57:46.683083+0000","last_clean":"2024-03-11T19:57:46.683083+0000","last_became_active":"2024-03-11T19:55:44.677153+0000","last_became_peered":"2024-03-11T19:55:44.677153+0000","last_unstale":"2024-03-11T19:57:46.683083+0000","last_undegraded":"2024-03-11T19:57:46.683083+0000","last_fullsized":"2024-03-11T19:57:46.683083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:52:59.153222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680692+0000","last_change":"2024-03-11T19:55:45.731827+0000","last_active":"2024-03-11T19:57:46.680692+0000","last_peered":"2024-03-11T19:57:46.680692+0000","last_clean":"2024-03-11T19:57:46.680692+0000","last_became_active":"2024-03-11T19:55:45.731460+0000","last_became_peered":"2024-03-11T19:55:45.731460+0000","last_unstale":"2024-03-11T19:57:46.680692+0000","last_undegraded":"2024-03-11T19:57:46.680692+0000","last_fullsized":"2024-03-11T19:57:46.680692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:12.277198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686677+0000","last_change":"2024-03-11T19:55:50.109812+0000","last_active":"2024-03-11T19:57:46.686677+0000","last_peered":"2024-03-11T19:57:46.686677+0000","last_clean":"2024-03-11T19:57:46.686677+0000","last_became_active":"2024-03-11T19:55:50.109420+0000","last_became_peered":"2024-03-11T19:55:50.109420+0000","last_unstale":"2024-03-11T19:57:46.686677+0000","last_undegraded":"2024-03-11T19:57:46.686677+0000","last_fullsized":"2024-03-11T19:57:46.686677+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:08:30.105941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683574+0000","last_change":"2024-03-11T19:55:44.683173+0000","last_active":"2024-03-11T19:57:46.683574+0000","last_peered":"2024-03-11T19:57:46.683574+0000","last_clean":"2024-03-11T19:57:46.683574+0000","last_became_active":"2024-03-11T19:55:44.682778+0000","last_became_peered":"2024-03-11T19:55:44.682778+0000","last_unstale":"2024-03-11T19:57:46.683574+0000","last_undegraded":"2024-03-11T19:57:46.683574+0000","last_fullsized":"2024-03-11T19:57:46.683574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:15.119864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572517+0000","last_change":"2024-03-11T19:55:45.721833+0000","last_active":"2024-03-11T19:57:47.572517+0000","last_peered":"2024-03-11T19:57:47.572517+0000","last_clean":"2024-03-11T19:57:47.572517+0000","last_became_active":"2024-03-11T19:55:45.721490+0000","last_became_peered":"2024-03-11T19:55:45.721490+0000","last_unstale":"2024-03-11T19:57:47.572517+0000","last_undegraded":"2024-03-11T19:57:47.572517+0000","last_fullsized":"2024-03-11T19:57:47.572517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:09.824912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881479+0000","last_change":"2024-03-11T19:55:50.108424+0000","last_active":"2024-03-11T19:57:46.881479+0000","last_peered":"2024-03-11T19:57:46.881479+0000","last_clean":"2024-03-11T19:57:46.881479+0000","last_became_active":"2024-03-11T19:55:50.107072+0000","last_became_peered":"2024-03-11T19:55:50.107072+0000","last_unstale":"2024-03-11T19:57:46.881479+0000","last_undegraded":"2024-03-11T19:57:46.881479+0000","last_fullsized":"2024-03-11T19:57:46.881479+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:31.751597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680227+0000","last_change":"2024-03-11T19:55:44.671570+0000","last_active":"2024-03-11T19:57:46.680227+0000","last_peered":"2024-03-11T19:57:46.680227+0000","last_clean":"2024-03-11T19:57:46.680227+0000","last_became_active":"2024-03-11T19:55:44.671007+0000","last_became_peered":"2024-03-11T19:55:44.671007+0000","last_unstale":"2024-03-11T19:57:46.680227+0000","last_undegraded":"2024-03-11T19:57:46.680227+0000","last_fullsized":"2024-03-11T19:57:46.680227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:23.865011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.691569+0000","last_change":"2024-03-11T19:55:45.733048+0000","last_active":"2024-03-11T19:57:46.691569+0000","last_peered":"2024-03-11T19:57:46.691569+0000","last_clean":"2024-03-11T19:57:46.691569+0000","last_became_active":"2024-03-11T19:55:45.731656+0000","last_became_peered":"2024-03-11T19:55:45.731656+0000","last_unstale":"2024-03-11T19:57:46.691569+0000","last_undegraded":"2024-03-11T19:57:46.691569+0000","last_fullsized":"2024-03-11T19:57:46.691569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:28:56.433327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572169+0000","last_change":"2024-03-11T19:55:50.105532+0000","last_active":"2024-03-11T19:57:47.572169+0000","last_peered":"2024-03-11T19:57:47.572169+0000","last_clean":"2024-03-11T19:57:47.572169+0000","last_became_active":"2024-03-11T19:55:50.102200+0000","last_became_peered":"2024-03-11T19:55:50.102200+0000","last_unstale":"2024-03-11T19:57:47.572169+0000","last_undegraded":"2024-03-11T19:57:47.572169+0000","last_fullsized":"2024-03-11T19:57:47.572169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:37.060472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":244,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:58:01.791717+0000","last_change":"2024-03-11T19:55:44.678224+0000","last_active":"2024-03-11T19:58:01.791717+0000","last_peered":"2024-03-11T19:58:01.791717+0000","last_clean":"2024-03-11T19:58:01.791717+0000","last_became_active":"2024-03-11T19:55:44.677936+0000","last_became_peered":"2024-03-11T19:55:44.677936+0000","last_unstale":"2024-03-11T19:58:01.791717+0000","last_undegraded":"2024-03-11T19:58:01.791717+0000","last_fullsized":"2024-03-11T19:58:01.791717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:20:42.486501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":208,"num_read_kb":208,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023625+0000","last_change":"2024-03-11T19:55:45.709917+0000","last_active":"2024-03-11T19:57:47.023625+0000","last_peered":"2024-03-11T19:57:47.023625+0000","last_clean":"2024-03-11T19:57:47.023625+0000","last_became_active":"2024-03-11T19:55:45.709318+0000","last_became_peered":"2024-03-11T19:55:45.709318+0000","last_unstale":"2024-03-11T19:57:47.023625+0000","last_undegraded":"2024-03-11T19:57:47.023625+0000","last_fullsized":"2024-03-11T19:57:47.023625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:08.298158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.883428+0000","last_change":"2024-03-11T19:57:46.882937+0000","last_active":"2024-03-11T19:57:46.883428+0000","last_peered":"2024-03-11T19:57:46.883428+0000","last_clean":"2024-03-11T19:57:46.883428+0000","last_became_active":"2024-03-11T19:57:46.882408+0000","last_became_peered":"2024-03-11T19:57:46.882408+0000","last_unstale":"2024-03-11T19:57:46.883428+0000","last_undegraded":"2024-03-11T19:57:46.883428+0000","last_fullsized":"2024-03-11T19:57:46.883428+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:58:27.842775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685774+0000","last_change":"2024-03-11T19:55:44.676004+0000","last_active":"2024-03-11T19:57:46.685774+0000","last_peered":"2024-03-11T19:57:46.685774+0000","last_clean":"2024-03-11T19:57:46.685774+0000","last_became_active":"2024-03-11T19:55:44.675640+0000","last_became_peered":"2024-03-11T19:55:44.675640+0000","last_unstale":"2024-03-11T19:57:46.685774+0000","last_undegraded":"2024-03-11T19:57:46.685774+0000","last_fullsized":"2024-03-11T19:57:46.685774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:15.096780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881928+0000","last_change":"2024-03-11T19:56:45.702569+0000","last_active":"2024-03-11T19:57:46.881928+0000","last_peered":"2024-03-11T19:57:46.881928+0000","last_clean":"2024-03-11T19:57:46.881928+0000","last_became_active":"2024-03-11T19:56:45.701890+0000","last_became_peered":"2024-03-11T19:56:45.701890+0000","last_unstale":"2024-03-11T19:57:46.881928+0000","last_undegraded":"2024-03-11T19:57:46.881928+0000","last_fullsized":"2024-03-11T19:57:46.881928+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:40.434712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684317+0000","last_change":"2024-03-11T19:55:50.096136+0000","last_active":"2024-03-11T19:57:46.684317+0000","last_peered":"2024-03-11T19:57:46.684317+0000","last_clean":"2024-03-11T19:57:46.684317+0000","last_became_active":"2024-03-11T19:55:50.095670+0000","last_became_peered":"2024-03-11T19:55:50.095670+0000","last_unstale":"2024-03-11T19:57:46.684317+0000","last_undegraded":"2024-03-11T19:57:46.684317+0000","last_fullsized":"2024-03-11T19:57:46.684317+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:40.943273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680839+0000","last_change":"2024-03-11T19:55:44.681547+0000","last_active":"2024-03-11T19:57:46.680839+0000","last_peered":"2024-03-11T19:57:46.680839+0000","last_clean":"2024-03-11T19:57:46.680839+0000","last_became_active":"2024-03-11T19:55:44.681237+0000","last_became_peered":"2024-03-11T19:55:44.681237+0000","last_unstale":"2024-03-11T19:57:46.680839+0000","last_undegraded":"2024-03-11T19:57:46.680839+0000","last_fullsized":"2024-03-11T19:57:46.680839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:08:09.509915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023380+0000","last_change":"2024-03-11T19:55:45.705025+0000","last_active":"2024-03-11T19:57:47.023380+0000","last_peered":"2024-03-11T19:57:47.023380+0000","last_clean":"2024-03-11T19:57:47.023380+0000","last_became_active":"2024-03-11T19:55:45.702680+0000","last_became_peered":"2024-03-11T19:55:45.702680+0000","last_unstale":"2024-03-11T19:57:47.023380+0000","last_undegraded":"2024-03-11T19:57:47.023380+0000","last_fullsized":"2024-03-11T19:57:47.023380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:28.245002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.689568+0000","last_change":"2024-03-11T19:55:50.099648+0000","last_active":"2024-03-11T19:57:46.689568+0000","last_peered":"2024-03-11T19:57:46.689568+0000","last_clean":"2024-03-11T19:57:46.689568+0000","last_became_active":"2024-03-11T19:55:50.098516+0000","last_became_peered":"2024-03-11T19:55:50.098516+0000","last_unstale":"2024-03-11T19:57:46.689568+0000","last_undegraded":"2024-03-11T19:57:46.689568+0000","last_fullsized":"2024-03-11T19:57:46.689568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:19.644046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":280,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:58:01.791661+0000","last_change":"2024-03-11T19:55:44.672460+0000","last_active":"2024-03-11T19:58:01.791661+0000","last_peered":"2024-03-11T19:58:01.791661+0000","last_clean":"2024-03-11T19:58:01.791661+0000","last_became_active":"2024-03-11T19:55:44.672123+0000","last_became_peered":"2024-03-11T19:55:44.672123+0000","last_unstale":"2024-03-11T19:58:01.791661+0000","last_undegraded":"2024-03-11T19:58:01.791661+0000","last_fullsized":"2024-03-11T19:58:01.791661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:43.940638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61471,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":215,"num_read_kb":168,"num_write":37,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685900+0000","last_change":"2024-03-11T19:55:45.702933+0000","last_active":"2024-03-11T19:57:46.685900+0000","last_peered":"2024-03-11T19:57:46.685900+0000","last_clean":"2024-03-11T19:57:46.685900+0000","last_became_active":"2024-03-11T19:55:45.702585+0000","last_became_peered":"2024-03-11T19:55:45.702585+0000","last_unstale":"2024-03-11T19:57:46.685900+0000","last_undegraded":"2024-03-11T19:57:46.685900+0000","last_fullsized":"2024-03-11T19:57:46.685900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:54:38.762156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.681155+0000","last_change":"2024-03-11T19:55:50.102571+0000","last_active":"2024-03-11T19:57:46.681155+0000","last_peered":"2024-03-11T19:57:46.681155+0000","last_clean":"2024-03-11T19:57:46.681155+0000","last_became_active":"2024-03-11T19:55:50.102099+0000","last_became_peered":"2024-03-11T19:55:50.102099+0000","last_unstale":"2024-03-11T19:57:46.681155+0000","last_undegraded":"2024-03-11T19:57:46.681155+0000","last_fullsized":"2024-03-11T19:57:46.681155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:17.631378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802401+0000","last_change":"2024-03-11T19:55:44.671027+0000","last_active":"2024-03-11T19:57:46.802401+0000","last_peered":"2024-03-11T19:57:46.802401+0000","last_clean":"2024-03-11T19:57:46.802401+0000","last_became_active":"2024-03-11T19:55:44.669696+0000","last_became_peered":"2024-03-11T19:55:44.669696+0000","last_unstale":"2024-03-11T19:57:46.802401+0000","last_undegraded":"2024-03-11T19:57:46.802401+0000","last_fullsized":"2024-03-11T19:57:46.802401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:48:20.169638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683919+0000","last_change":"2024-03-11T19:55:45.729119+0000","last_active":"2024-03-11T19:57:46.683919+0000","last_peered":"2024-03-11T19:57:46.683919+0000","last_clean":"2024-03-11T19:57:46.683919+0000","last_became_active":"2024-03-11T19:55:45.727442+0000","last_became_peered":"2024-03-11T19:55:45.727442+0000","last_unstale":"2024-03-11T19:57:46.683919+0000","last_undegraded":"2024-03-11T19:57:46.683919+0000","last_fullsized":"2024-03-11T19:57:46.683919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:14:51.483199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686945+0000","last_change":"2024-03-11T19:55:50.104326+0000","last_active":"2024-03-11T19:57:46.686945+0000","last_peered":"2024-03-11T19:57:46.686945+0000","last_clean":"2024-03-11T19:57:46.686945+0000","last_became_active":"2024-03-11T19:55:50.103881+0000","last_became_peered":"2024-03-11T19:55:50.103881+0000","last_unstale":"2024-03-11T19:57:46.686945+0000","last_undegraded":"2024-03-11T19:57:46.686945+0000","last_fullsized":"2024-03-11T19:57:46.686945+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:12:38.819591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680603+0000","last_change":"2024-03-11T19:55:44.684900+0000","last_active":"2024-03-11T19:57:46.680603+0000","last_peered":"2024-03-11T19:57:46.680603+0000","last_clean":"2024-03-11T19:57:46.680603+0000","last_became_active":"2024-03-11T19:55:44.684590+0000","last_became_peered":"2024-03-11T19:55:44.684590+0000","last_unstale":"2024-03-11T19:57:46.680603+0000","last_undegraded":"2024-03-11T19:57:46.680603+0000","last_fullsized":"2024-03-11T19:57:46.680603+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:07.744366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684928+0000","last_change":"2024-03-11T19:55:45.704787+0000","last_active":"2024-03-11T19:57:46.684928+0000","last_peered":"2024-03-11T19:57:46.684928+0000","last_clean":"2024-03-11T19:57:46.684928+0000","last_became_active":"2024-03-11T19:55:45.704405+0000","last_became_peered":"2024-03-11T19:55:45.704405+0000","last_unstale":"2024-03-11T19:57:46.684928+0000","last_undegraded":"2024-03-11T19:57:46.684928+0000","last_fullsized":"2024-03-11T19:57:46.684928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:58:05.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.690120+0000","last_change":"2024-03-11T19:55:50.104793+0000","last_active":"2024-03-11T19:57:46.690120+0000","last_peered":"2024-03-11T19:57:46.690120+0000","last_clean":"2024-03-11T19:57:46.690120+0000","last_became_active":"2024-03-11T19:55:50.104415+0000","last_became_peered":"2024-03-11T19:55:50.104415+0000","last_unstale":"2024-03-11T19:57:46.690120+0000","last_undegraded":"2024-03-11T19:57:46.690120+0000","last_fullsized":"2024-03-11T19:57:46.690120+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:53.307313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":141,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:58:00.816719+0000","last_change":"2024-03-11T19:55:44.669403+0000","last_active":"2024-03-11T19:58:00.816719+0000","last_peered":"2024-03-11T19:58:00.816719+0000","last_clean":"2024-03-11T19:58:00.816719+0000","last_became_active":"2024-03-11T19:55:44.668908+0000","last_became_peered":"2024-03-11T19:55:44.668908+0000","last_unstale":"2024-03-11T19:58:00.816719+0000","last_undegraded":"2024-03-11T19:58:00.816719+0000","last_fullsized":"2024-03-11T19:58:00.816719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:56:36.831506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":107,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684333+0000","last_change":"2024-03-11T19:55:45.700029+0000","last_active":"2024-03-11T19:57:46.684333+0000","last_peered":"2024-03-11T19:57:46.684333+0000","last_clean":"2024-03-11T19:57:46.684333+0000","last_became_active":"2024-03-11T19:55:45.697296+0000","last_became_peered":"2024-03-11T19:55:45.697296+0000","last_unstale":"2024-03-11T19:57:46.684333+0000","last_undegraded":"2024-03-11T19:57:46.684333+0000","last_fullsized":"2024-03-11T19:57:46.684333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:59:35.025845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.577523+0000","last_change":"2024-03-11T19:57:47.577523+0000","last_active":"2024-03-11T19:57:47.577523+0000","last_peered":"2024-03-11T19:57:47.577523+0000","last_clean":"2024-03-11T19:57:47.577523+0000","last_became_active":"2024-03-11T19:57:47.575852+0000","last_became_peered":"2024-03-11T19:57:47.575852+0000","last_unstale":"2024-03-11T19:57:47.577523+0000","last_undegraded":"2024-03-11T19:57:47.577523+0000","last_fullsized":"2024-03-11T19:57:47.577523+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:00:25.108706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685341+0000","last_change":"2024-03-11T19:55:44.677491+0000","last_active":"2024-03-11T19:57:46.685341+0000","last_peered":"2024-03-11T19:57:46.685341+0000","last_clean":"2024-03-11T19:57:46.685341+0000","last_became_active":"2024-03-11T19:55:44.677176+0000","last_became_peered":"2024-03-11T19:55:44.677176+0000","last_unstale":"2024-03-11T19:57:46.685341+0000","last_undegraded":"2024-03-11T19:57:46.685341+0000","last_fullsized":"2024-03-11T19:57:46.685341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:51.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680318+0000","last_change":"2024-03-11T19:55:45.703310+0000","last_active":"2024-03-11T19:57:46.680318+0000","last_peered":"2024-03-11T19:57:46.680318+0000","last_clean":"2024-03-11T19:57:46.680318+0000","last_became_active":"2024-03-11T19:55:45.702933+0000","last_became_peered":"2024-03-11T19:55:45.702933+0000","last_unstale":"2024-03-11T19:57:46.680318+0000","last_undegraded":"2024-03-11T19:57:46.680318+0000","last_fullsized":"2024-03-11T19:57:46.680318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:07:58.787949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680521+0000","last_change":"2024-03-11T19:55:50.103849+0000","last_active":"2024-03-11T19:57:46.680521+0000","last_peered":"2024-03-11T19:57:46.680521+0000","last_clean":"2024-03-11T19:57:46.680521+0000","last_became_active":"2024-03-11T19:55:50.103360+0000","last_became_peered":"2024-03-11T19:55:50.103360+0000","last_unstale":"2024-03-11T19:57:46.680521+0000","last_undegraded":"2024-03-11T19:57:46.680521+0000","last_fullsized":"2024-03-11T19:57:46.680521+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:38:05.151041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685858+0000","last_change":"2024-03-11T19:55:44.669839+0000","last_active":"2024-03-11T19:57:46.685858+0000","last_peered":"2024-03-11T19:57:46.685858+0000","last_clean":"2024-03-11T19:57:46.685858+0000","last_became_active":"2024-03-11T19:55:44.669494+0000","last_became_peered":"2024-03-11T19:55:44.669494+0000","last_unstale":"2024-03-11T19:57:46.685858+0000","last_undegraded":"2024-03-11T19:57:46.685858+0000","last_fullsized":"2024-03-11T19:57:46.685858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:00.313847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687005+0000","last_change":"2024-03-11T19:56:45.693864+0000","last_active":"2024-03-11T19:57:46.687005+0000","last_peered":"2024-03-11T19:57:46.687005+0000","last_clean":"2024-03-11T19:57:46.687005+0000","last_became_active":"2024-03-11T19:56:45.693091+0000","last_became_peered":"2024-03-11T19:56:45.693091+0000","last_unstale":"2024-03-11T19:57:46.687005+0000","last_undegraded":"2024-03-11T19:57:46.687005+0000","last_fullsized":"2024-03-11T19:57:46.687005+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:26:37.233485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685560+0000","last_change":"2024-03-11T19:55:50.143916+0000","last_active":"2024-03-11T19:57:46.685560+0000","last_peered":"2024-03-11T19:57:46.685560+0000","last_clean":"2024-03-11T19:57:46.685560+0000","last_became_active":"2024-03-11T19:55:50.143341+0000","last_became_peered":"2024-03-11T19:55:50.143341+0000","last_unstale":"2024-03-11T19:57:46.685560+0000","last_undegraded":"2024-03-11T19:57:46.685560+0000","last_fullsized":"2024-03-11T19:57:46.685560+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:40.637529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.691467+0000","last_change":"2024-03-11T19:55:44.688468+0000","last_active":"2024-03-11T19:57:46.691467+0000","last_peered":"2024-03-11T19:57:46.691467+0000","last_clean":"2024-03-11T19:57:46.691467+0000","last_became_active":"2024-03-11T19:55:44.688065+0000","last_became_peered":"2024-03-11T19:55:44.688065+0000","last_unstale":"2024-03-11T19:57:46.691467+0000","last_undegraded":"2024-03-11T19:57:46.691467+0000","last_fullsized":"2024-03-11T19:57:46.691467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:29:23.392749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683940+0000","last_change":"2024-03-11T19:55:45.728970+0000","last_active":"2024-03-11T19:57:46.683940+0000","last_peered":"2024-03-11T19:57:46.683940+0000","last_clean":"2024-03-11T19:57:46.683940+0000","last_became_active":"2024-03-11T19:55:45.728516+0000","last_became_peered":"2024-03-11T19:55:45.728516+0000","last_unstale":"2024-03-11T19:57:46.683940+0000","last_undegraded":"2024-03-11T19:57:46.683940+0000","last_fullsized":"2024-03-11T19:57:46.683940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:19.242610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684235+0000","last_change":"2024-03-11T19:55:50.111884+0000","last_active":"2024-03-11T19:57:46.684235+0000","last_peered":"2024-03-11T19:57:46.684235+0000","last_clean":"2024-03-11T19:57:46.684235+0000","last_became_active":"2024-03-11T19:55:50.110802+0000","last_became_peered":"2024-03-11T19:55:50.110802+0000","last_unstale":"2024-03-11T19:57:46.684235+0000","last_undegraded":"2024-03-11T19:57:46.684235+0000","last_fullsized":"2024-03-11T19:57:46.684235+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:35.866368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":136,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:55.857328+0000","last_change":"2024-03-11T19:55:44.677156+0000","last_active":"2024-03-11T19:57:55.857328+0000","last_peered":"2024-03-11T19:57:55.857328+0000","last_clean":"2024-03-11T19:57:55.857328+0000","last_became_active":"2024-03-11T19:55:44.676656+0000","last_became_peered":"2024-03-11T19:55:44.676656+0000","last_unstale":"2024-03-11T19:57:55.857328+0000","last_undegraded":"2024-03-11T19:57:55.857328+0000","last_fullsized":"2024-03-11T19:57:55.857328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:40:27.521669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":99,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683235+0000","last_change":"2024-03-11T19:55:45.715545+0000","last_active":"2024-03-11T19:57:46.683235+0000","last_peered":"2024-03-11T19:57:46.683235+0000","last_clean":"2024-03-11T19:57:46.683235+0000","last_became_active":"2024-03-11T19:55:45.715042+0000","last_became_peered":"2024-03-11T19:55:45.715042+0000","last_unstale":"2024-03-11T19:57:46.683235+0000","last_undegraded":"2024-03-11T19:57:46.683235+0000","last_fullsized":"2024-03-11T19:57:46.683235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:06:33.362235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685165+0000","last_change":"2024-03-11T19:55:50.145603+0000","last_active":"2024-03-11T19:57:46.685165+0000","last_peered":"2024-03-11T19:57:46.685165+0000","last_clean":"2024-03-11T19:57:46.685165+0000","last_became_active":"2024-03-11T19:55:50.143689+0000","last_became_peered":"2024-03-11T19:55:50.143689+0000","last_unstale":"2024-03-11T19:57:46.685165+0000","last_undegraded":"2024-03-11T19:57:46.685165+0000","last_fullsized":"2024-03-11T19:57:46.685165+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:57.289983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680431+0000","last_change":"2024-03-11T19:55:45.702011+0000","last_active":"2024-03-11T19:57:46.680431+0000","last_peered":"2024-03-11T19:57:46.680431+0000","last_clean":"2024-03-11T19:57:46.680431+0000","last_became_active":"2024-03-11T19:55:45.701673+0000","last_became_peered":"2024-03-11T19:55:45.701673+0000","last_unstale":"2024-03-11T19:57:46.680431+0000","last_undegraded":"2024-03-11T19:57:46.680431+0000","last_fullsized":"2024-03-11T19:57:46.680431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:17.028276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":141,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:59.814331+0000","last_change":"2024-03-11T19:55:44.689121+0000","last_active":"2024-03-11T19:57:59.814331+0000","last_peered":"2024-03-11T19:57:59.814331+0000","last_clean":"2024-03-11T19:57:59.814331+0000","last_became_active":"2024-03-11T19:55:44.688453+0000","last_became_peered":"2024-03-11T19:55:44.688453+0000","last_unstale":"2024-03-11T19:57:59.814331+0000","last_undegraded":"2024-03-11T19:57:59.814331+0000","last_fullsized":"2024-03-11T19:57:59.814331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:31.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.698656+0000","last_change":"2024-03-11T19:57:46.698213+0000","last_active":"2024-03-11T19:57:46.698656+0000","last_peered":"2024-03-11T19:57:46.698656+0000","last_clean":"2024-03-11T19:57:46.698656+0000","last_became_active":"2024-03-11T19:57:46.693710+0000","last_became_peered":"2024-03-11T19:57:46.693710+0000","last_unstale":"2024-03-11T19:57:46.698656+0000","last_undegraded":"2024-03-11T19:57:46.698656+0000","last_fullsized":"2024-03-11T19:57:46.698656+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:06:00.214092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.691362+0000","last_change":"2024-03-11T19:56:45.703746+0000","last_active":"2024-03-11T19:57:46.691362+0000","last_peered":"2024-03-11T19:57:46.691362+0000","last_clean":"2024-03-11T19:57:46.691362+0000","last_became_active":"2024-03-11T19:56:45.701252+0000","last_became_peered":"2024-03-11T19:56:45.701252+0000","last_unstale":"2024-03-11T19:57:46.691362+0000","last_undegraded":"2024-03-11T19:57:46.691362+0000","last_fullsized":"2024-03-11T19:57:46.691362+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:16:50.219593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684249+0000","last_change":"2024-03-11T19:55:44.672644+0000","last_active":"2024-03-11T19:57:46.684249+0000","last_peered":"2024-03-11T19:57:46.684249+0000","last_clean":"2024-03-11T19:57:46.684249+0000","last_became_active":"2024-03-11T19:55:44.671645+0000","last_became_peered":"2024-03-11T19:55:44.671645+0000","last_unstale":"2024-03-11T19:57:46.684249+0000","last_undegraded":"2024-03-11T19:57:46.684249+0000","last_fullsized":"2024-03-11T19:57:46.684249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:25:46.637768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":367,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684193+0000","last_change":"2024-03-11T19:54:20.915891+0000","last_active":"2024-03-11T19:57:46.684193+0000","last_peered":"2024-03-11T19:57:46.684193+0000","last_clean":"2024-03-11T19:57:46.684193+0000","last_became_active":"2024-03-11T19:54:20.915450+0000","last_became_peered":"2024-03-11T19:54:20.915450+0000","last_unstale":"2024-03-11T19:57:46.684193+0000","last_undegraded":"2024-03-11T19:57:46.684193+0000","last_fullsized":"2024-03-11T19:57:46.684193+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_clean_scrub_stamp":"2024-03-11T19:50:29.052858+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:47.966076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680928+0000","last_change":"2024-03-11T19:55:50.144053+0000","last_active":"2024-03-11T19:57:46.680928+0000","last_peered":"2024-03-11T19:57:46.680928+0000","last_clean":"2024-03-11T19:57:46.680928+0000","last_became_active":"2024-03-11T19:55:50.143509+0000","last_became_peered":"2024-03-11T19:55:50.143509+0000","last_unstale":"2024-03-11T19:57:46.680928+0000","last_undegraded":"2024-03-11T19:57:46.680928+0000","last_fullsized":"2024-03-11T19:57:46.680928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:57:15.448812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683348+0000","last_change":"2024-03-11T19:55:45.712621+0000","last_active":"2024-03-11T19:57:46.683348+0000","last_peered":"2024-03-11T19:57:46.683348+0000","last_clean":"2024-03-11T19:57:46.683348+0000","last_became_active":"2024-03-11T19:55:45.712161+0000","last_became_peered":"2024-03-11T19:55:45.712161+0000","last_unstale":"2024-03-11T19:57:46.683348+0000","last_undegraded":"2024-03-11T19:57:46.683348+0000","last_fullsized":"2024-03-11T19:57:46.683348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:34:52.911944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684355+0000","last_change":"2024-03-11T19:55:44.674606+0000","last_active":"2024-03-11T19:57:46.684355+0000","last_peered":"2024-03-11T19:57:46.684355+0000","last_clean":"2024-03-11T19:57:46.684355+0000","last_became_active":"2024-03-11T19:55:44.674355+0000","last_became_peered":"2024-03-11T19:55:44.674355+0000","last_unstale":"2024-03-11T19:57:46.684355+0000","last_undegraded":"2024-03-11T19:57:46.684355+0000","last_fullsized":"2024-03-11T19:57:46.684355+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:31:01.662582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686781+0000","last_change":"2024-03-11T19:55:50.101468+0000","last_active":"2024-03-11T19:57:46.686781+0000","last_peered":"2024-03-11T19:57:46.686781+0000","last_clean":"2024-03-11T19:57:46.686781+0000","last_became_active":"2024-03-11T19:55:50.101040+0000","last_became_peered":"2024-03-11T19:55:50.101040+0000","last_unstale":"2024-03-11T19:57:46.686781+0000","last_undegraded":"2024-03-11T19:57:46.686781+0000","last_fullsized":"2024-03-11T19:57:46.686781+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:55.354176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686046+0000","last_change":"2024-03-11T19:55:44.681467+0000","last_active":"2024-03-11T19:57:46.686046+0000","last_peered":"2024-03-11T19:57:46.686046+0000","last_clean":"2024-03-11T19:57:46.686046+0000","last_became_active":"2024-03-11T19:55:44.681176+0000","last_became_peered":"2024-03-11T19:55:44.681176+0000","last_unstale":"2024-03-11T19:57:46.686046+0000","last_undegraded":"2024-03-11T19:57:46.686046+0000","last_fullsized":"2024-03-11T19:57:46.686046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:27:38.019116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685401+0000","last_change":"2024-03-11T19:55:45.709798+0000","last_active":"2024-03-11T19:57:46.685401+0000","last_peered":"2024-03-11T19:57:46.685401+0000","last_clean":"2024-03-11T19:57:46.685401+0000","last_became_active":"2024-03-11T19:55:45.709277+0000","last_became_peered":"2024-03-11T19:55:45.709277+0000","last_unstale":"2024-03-11T19:57:46.685401+0000","last_undegraded":"2024-03-11T19:57:46.685401+0000","last_fullsized":"2024-03-11T19:57:46.685401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:58.852316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572588+0000","last_change":"2024-03-11T19:55:50.146601+0000","last_active":"2024-03-11T19:57:47.572588+0000","last_peered":"2024-03-11T19:57:47.572588+0000","last_clean":"2024-03-11T19:57:47.572588+0000","last_became_active":"2024-03-11T19:55:50.144584+0000","last_became_peered":"2024-03-11T19:55:50.144584+0000","last_unstale":"2024-03-11T19:57:47.572588+0000","last_undegraded":"2024-03-11T19:57:47.572588+0000","last_fullsized":"2024-03-11T19:57:47.572588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:00.365713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'14","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.690894+0000","last_change":"2024-03-11T19:55:44.689109+0000","last_active":"2024-03-11T19:57:46.690894+0000","last_peered":"2024-03-11T19:57:46.690894+0000","last_clean":"2024-03-11T19:57:46.690894+0000","last_became_active":"2024-03-11T19:55:44.687323+0000","last_became_peered":"2024-03-11T19:55:44.687323+0000","last_unstale":"2024-03-11T19:57:46.690894+0000","last_undegraded":"2024-03-11T19:57:46.690894+0000","last_fullsized":"2024-03-11T19:57:46.690894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:24.613901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573872+0000","last_change":"2024-03-11T19:55:45.726033+0000","last_active":"2024-03-11T19:57:47.573872+0000","last_peered":"2024-03-11T19:57:47.573872+0000","last_clean":"2024-03-11T19:57:47.573872+0000","last_became_active":"2024-03-11T19:55:45.725353+0000","last_became_peered":"2024-03-11T19:55:45.725353+0000","last_unstale":"2024-03-11T19:57:47.573872+0000","last_undegraded":"2024-03-11T19:57:47.573872+0000","last_fullsized":"2024-03-11T19:57:47.573872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:36:24.341870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802159+0000","last_change":"2024-03-11T19:55:50.089496+0000","last_active":"2024-03-11T19:57:46.802159+0000","last_peered":"2024-03-11T19:57:46.802159+0000","last_clean":"2024-03-11T19:57:46.802159+0000","last_became_active":"2024-03-11T19:55:50.089072+0000","last_became_peered":"2024-03-11T19:55:50.089072+0000","last_unstale":"2024-03-11T19:57:46.802159+0000","last_undegraded":"2024-03-11T19:57:46.802159+0000","last_fullsized":"2024-03-11T19:57:46.802159+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:27:32.539333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685271+0000","last_change":"2024-03-11T19:55:44.673147+0000","last_active":"2024-03-11T19:57:46.685271+0000","last_peered":"2024-03-11T19:57:46.685271+0000","last_clean":"2024-03-11T19:57:46.685271+0000","last_became_active":"2024-03-11T19:55:44.672787+0000","last_became_peered":"2024-03-11T19:55:44.672787+0000","last_unstale":"2024-03-11T19:57:46.685271+0000","last_undegraded":"2024-03-11T19:57:46.685271+0000","last_fullsized":"2024-03-11T19:57:46.685271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:11:23.864419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.882032+0000","last_change":"2024-03-11T19:55:45.732376+0000","last_active":"2024-03-11T19:57:46.882032+0000","last_peered":"2024-03-11T19:57:46.882032+0000","last_clean":"2024-03-11T19:57:46.882032+0000","last_became_active":"2024-03-11T19:55:45.731956+0000","last_became_peered":"2024-03-11T19:55:45.731956+0000","last_unstale":"2024-03-11T19:57:46.882032+0000","last_undegraded":"2024-03-11T19:57:46.882032+0000","last_fullsized":"2024-03-11T19:57:46.882032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:43:48.423678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684605+0000","last_change":"2024-03-11T19:55:50.144080+0000","last_active":"2024-03-11T19:57:46.684605+0000","last_peered":"2024-03-11T19:57:46.684605+0000","last_clean":"2024-03-11T19:57:46.684605+0000","last_became_active":"2024-03-11T19:55:50.143585+0000","last_became_peered":"2024-03-11T19:55:50.143585+0000","last_unstale":"2024-03-11T19:57:46.684605+0000","last_undegraded":"2024-03-11T19:57:46.684605+0000","last_fullsized":"2024-03-11T19:57:46.684605+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:47:51.388823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023178+0000","last_change":"2024-03-11T19:55:44.677354+0000","last_active":"2024-03-11T19:57:47.023178+0000","last_peered":"2024-03-11T19:57:47.023178+0000","last_clean":"2024-03-11T19:57:47.023178+0000","last_became_active":"2024-03-11T19:55:44.677013+0000","last_became_peered":"2024-03-11T19:55:44.677013+0000","last_unstale":"2024-03-11T19:57:47.023178+0000","last_undegraded":"2024-03-11T19:57:47.023178+0000","last_fullsized":"2024-03-11T19:57:47.023178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:50:48.855320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685452+0000","last_change":"2024-03-11T19:55:45.702278+0000","last_active":"2024-03-11T19:57:46.685452+0000","last_peered":"2024-03-11T19:57:46.685452+0000","last_clean":"2024-03-11T19:57:46.685452+0000","last_became_active":"2024-03-11T19:55:45.701843+0000","last_became_peered":"2024-03-11T19:55:45.701843+0000","last_unstale":"2024-03-11T19:57:46.685452+0000","last_undegraded":"2024-03-11T19:57:46.685452+0000","last_fullsized":"2024-03-11T19:57:46.685452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:55:28.351093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.690731+0000","last_change":"2024-03-11T19:55:50.106083+0000","last_active":"2024-03-11T19:57:46.690731+0000","last_peered":"2024-03-11T19:57:46.690731+0000","last_clean":"2024-03-11T19:57:46.690731+0000","last_became_active":"2024-03-11T19:55:50.105591+0000","last_became_peered":"2024-03-11T19:55:50.105591+0000","last_unstale":"2024-03-11T19:57:46.690731+0000","last_undegraded":"2024-03-11T19:57:46.690731+0000","last_fullsized":"2024-03-11T19:57:46.690731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:34:58.713672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573135+0000","last_change":"2024-03-11T19:55:44.688128+0000","last_active":"2024-03-11T19:57:47.573135+0000","last_peered":"2024-03-11T19:57:47.573135+0000","last_clean":"2024-03-11T19:57:47.573135+0000","last_became_active":"2024-03-11T19:55:44.684165+0000","last_became_peered":"2024-03-11T19:55:44.684165+0000","last_unstale":"2024-03-11T19:57:47.573135+0000","last_undegraded":"2024-03-11T19:57:47.573135+0000","last_fullsized":"2024-03-11T19:57:47.573135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:37:19.415960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686783+0000","last_change":"2024-03-11T19:55:45.718968+0000","last_active":"2024-03-11T19:57:46.686783+0000","last_peered":"2024-03-11T19:57:46.686783+0000","last_clean":"2024-03-11T19:57:46.686783+0000","last_became_active":"2024-03-11T19:55:45.718625+0000","last_became_peered":"2024-03-11T19:55:45.718625+0000","last_unstale":"2024-03-11T19:57:46.686783+0000","last_undegraded":"2024-03-11T19:57:46.686783+0000","last_fullsized":"2024-03-11T19:57:46.686783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:27.197227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684464+0000","last_change":"2024-03-11T19:55:50.097617+0000","last_active":"2024-03-11T19:57:46.684464+0000","last_peered":"2024-03-11T19:57:46.684464+0000","last_clean":"2024-03-11T19:57:46.684464+0000","last_became_active":"2024-03-11T19:55:50.097137+0000","last_became_peered":"2024-03-11T19:55:50.097137+0000","last_unstale":"2024-03-11T19:57:46.684464+0000","last_undegraded":"2024-03-11T19:57:46.684464+0000","last_fullsized":"2024-03-11T19:57:46.684464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:19:20.995137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802304+0000","last_change":"2024-03-11T19:55:44.669944+0000","last_active":"2024-03-11T19:57:46.802304+0000","last_peered":"2024-03-11T19:57:46.802304+0000","last_clean":"2024-03-11T19:57:46.802304+0000","last_became_active":"2024-03-11T19:55:44.669309+0000","last_became_peered":"2024-03-11T19:55:44.669309+0000","last_unstale":"2024-03-11T19:57:46.802304+0000","last_undegraded":"2024-03-11T19:57:46.802304+0000","last_fullsized":"2024-03-11T19:57:46.802304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:06:41.489579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.574440+0000","last_change":"2024-03-11T19:55:45.731348+0000","last_active":"2024-03-11T19:57:47.574440+0000","last_peered":"2024-03-11T19:57:47.574440+0000","last_clean":"2024-03-11T19:57:47.574440+0000","last_became_active":"2024-03-11T19:55:45.730957+0000","last_became_peered":"2024-03-11T19:55:45.730957+0000","last_unstale":"2024-03-11T19:57:47.574440+0000","last_undegraded":"2024-03-11T19:57:47.574440+0000","last_fullsized":"2024-03-11T19:57:47.574440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:06:50.422511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684428+0000","last_change":"2024-03-11T19:55:50.144050+0000","last_active":"2024-03-11T19:57:46.684428+0000","last_peered":"2024-03-11T19:57:46.684428+0000","last_clean":"2024-03-11T19:57:46.684428+0000","last_became_active":"2024-03-11T19:55:50.143591+0000","last_became_peered":"2024-03-11T19:55:50.143591+0000","last_unstale":"2024-03-11T19:57:46.684428+0000","last_undegraded":"2024-03-11T19:57:46.684428+0000","last_fullsized":"2024-03-11T19:57:46.684428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:02.006665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685506+0000","last_change":"2024-03-11T19:55:44.671973+0000","last_active":"2024-03-11T19:57:46.685506+0000","last_peered":"2024-03-11T19:57:46.685506+0000","last_clean":"2024-03-11T19:57:46.685506+0000","last_became_active":"2024-03-11T19:55:44.671661+0000","last_became_peered":"2024-03-11T19:55:44.671661+0000","last_unstale":"2024-03-11T19:57:46.685506+0000","last_undegraded":"2024-03-11T19:57:46.685506+0000","last_fullsized":"2024-03-11T19:57:46.685506+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:35.747386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684815+0000","last_change":"2024-03-11T19:55:45.705545+0000","last_active":"2024-03-11T19:57:46.684815+0000","last_peered":"2024-03-11T19:57:46.684815+0000","last_clean":"2024-03-11T19:57:46.684815+0000","last_became_active":"2024-03-11T19:55:45.705083+0000","last_became_peered":"2024-03-11T19:55:45.705083+0000","last_unstale":"2024-03-11T19:57:46.684815+0000","last_undegraded":"2024-03-11T19:57:46.684815+0000","last_fullsized":"2024-03-11T19:57:46.684815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:02:58.219026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.022984+0000","last_change":"2024-03-11T19:55:50.097951+0000","last_active":"2024-03-11T19:57:47.022984+0000","last_peered":"2024-03-11T19:57:47.022984+0000","last_clean":"2024-03-11T19:57:47.022984+0000","last_became_active":"2024-03-11T19:55:50.097446+0000","last_became_peered":"2024-03-11T19:55:50.097446+0000","last_unstale":"2024-03-11T19:57:47.022984+0000","last_undegraded":"2024-03-11T19:57:47.022984+0000","last_fullsized":"2024-03-11T19:57:47.022984+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:12.105801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'7","reported_seq":141,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:55.822266+0000","last_change":"2024-03-11T19:55:44.673136+0000","last_active":"2024-03-11T19:57:55.822266+0000","last_peered":"2024-03-11T19:57:55.822266+0000","last_clean":"2024-03-11T19:57:55.822266+0000","last_became_active":"2024-03-11T19:55:44.672467+0000","last_became_peered":"2024-03-11T19:55:44.672467+0000","last_unstale":"2024-03-11T19:57:55.822266+0000","last_undegraded":"2024-03-11T19:57:55.822266+0000","last_fullsized":"2024-03-11T19:57:55.822266+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:46:05.931253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685595+0000","last_change":"2024-03-11T19:55:45.710561+0000","last_active":"2024-03-11T19:57:46.685595+0000","last_peered":"2024-03-11T19:57:46.685595+0000","last_clean":"2024-03-11T19:57:46.685595+0000","last_became_active":"2024-03-11T19:55:45.710164+0000","last_became_peered":"2024-03-11T19:55:45.710164+0000","last_unstale":"2024-03-11T19:57:46.685595+0000","last_undegraded":"2024-03-11T19:57:46.685595+0000","last_fullsized":"2024-03-11T19:57:46.685595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:00:33.977595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.880197+0000","last_change":"2024-03-11T19:55:50.113390+0000","last_active":"2024-03-11T19:57:46.880197+0000","last_peered":"2024-03-11T19:57:46.880197+0000","last_clean":"2024-03-11T19:57:46.880197+0000","last_became_active":"2024-03-11T19:55:50.112656+0000","last_became_peered":"2024-03-11T19:55:50.112656+0000","last_unstale":"2024-03-11T19:57:46.880197+0000","last_undegraded":"2024-03-11T19:57:46.880197+0000","last_fullsized":"2024-03-11T19:57:46.880197+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:11.952585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.681043+0000","last_change":"2024-03-11T19:55:44.684907+0000","last_active":"2024-03-11T19:57:46.681043+0000","last_peered":"2024-03-11T19:57:46.681043+0000","last_clean":"2024-03-11T19:57:46.681043+0000","last_became_active":"2024-03-11T19:55:44.684600+0000","last_became_peered":"2024-03-11T19:55:44.684600+0000","last_unstale":"2024-03-11T19:57:46.681043+0000","last_undegraded":"2024-03-11T19:57:46.681043+0000","last_fullsized":"2024-03-11T19:57:46.681043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:12:33.666756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685169+0000","last_change":"2024-03-11T19:55:45.723431+0000","last_active":"2024-03-11T19:57:46.685169+0000","last_peered":"2024-03-11T19:57:46.685169+0000","last_clean":"2024-03-11T19:57:46.685169+0000","last_became_active":"2024-03-11T19:55:45.723135+0000","last_became_peered":"2024-03-11T19:55:45.723135+0000","last_unstale":"2024-03-11T19:57:46.685169+0000","last_undegraded":"2024-03-11T19:57:46.685169+0000","last_fullsized":"2024-03-11T19:57:46.685169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:12:35.654094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680159+0000","last_change":"2024-03-11T19:55:50.092876+0000","last_active":"2024-03-11T19:57:46.680159+0000","last_peered":"2024-03-11T19:57:46.680159+0000","last_clean":"2024-03-11T19:57:46.680159+0000","last_became_active":"2024-03-11T19:55:50.092478+0000","last_became_peered":"2024-03-11T19:55:50.092478+0000","last_unstale":"2024-03-11T19:57:46.680159+0000","last_undegraded":"2024-03-11T19:57:46.680159+0000","last_fullsized":"2024-03-11T19:57:46.680159+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:39.248325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685058+0000","last_change":"2024-03-11T19:56:45.711269+0000","last_active":"2024-03-11T19:57:46.685058+0000","last_peered":"2024-03-11T19:57:46.685058+0000","last_clean":"2024-03-11T19:57:46.685058+0000","last_became_active":"2024-03-11T19:56:45.704657+0000","last_became_peered":"2024-03-11T19:56:45.704657+0000","last_unstale":"2024-03-11T19:57:46.685058+0000","last_undegraded":"2024-03-11T19:57:46.685058+0000","last_fullsized":"2024-03-11T19:57:46.685058+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:51.549965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685910+0000","last_change":"2024-03-11T19:55:45.702924+0000","last_active":"2024-03-11T19:57:46.685910+0000","last_peered":"2024-03-11T19:57:46.685910+0000","last_clean":"2024-03-11T19:57:46.685910+0000","last_became_active":"2024-03-11T19:55:45.702530+0000","last_became_peered":"2024-03-11T19:55:45.702530+0000","last_unstale":"2024-03-11T19:57:46.685910+0000","last_undegraded":"2024-03-11T19:57:46.685910+0000","last_fullsized":"2024-03-11T19:57:46.685910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:14.679964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686802+0000","last_change":"2024-03-11T19:55:45.706123+0000","last_active":"2024-03-11T19:57:46.686802+0000","last_peered":"2024-03-11T19:57:46.686802+0000","last_clean":"2024-03-11T19:57:46.686802+0000","last_became_active":"2024-03-11T19:55:45.705706+0000","last_became_peered":"2024-03-11T19:55:45.705706+0000","last_unstale":"2024-03-11T19:57:46.686802+0000","last_undegraded":"2024-03-11T19:57:46.686802+0000","last_fullsized":"2024-03-11T19:57:46.686802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:42:58.989823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802706+0000","last_change":"2024-03-11T19:55:45.703188+0000","last_active":"2024-03-11T19:57:46.802706+0000","last_peered":"2024-03-11T19:57:46.802706+0000","last_clean":"2024-03-11T19:57:46.802706+0000","last_became_active":"2024-03-11T19:55:45.702803+0000","last_became_peered":"2024-03-11T19:55:45.702803+0000","last_unstale":"2024-03-11T19:57:46.802706+0000","last_undegraded":"2024-03-11T19:57:46.802706+0000","last_fullsized":"2024-03-11T19:57:46.802706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:20.922963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802270+0000","last_change":"2024-03-11T19:55:45.728856+0000","last_active":"2024-03-11T19:57:46.802270+0000","last_peered":"2024-03-11T19:57:46.802270+0000","last_clean":"2024-03-11T19:57:46.802270+0000","last_became_active":"2024-03-11T19:55:45.728526+0000","last_became_peered":"2024-03-11T19:55:45.728526+0000","last_unstale":"2024-03-11T19:57:46.802270+0000","last_undegraded":"2024-03-11T19:57:46.802270+0000","last_fullsized":"2024-03-11T19:57:46.802270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:25:30.836073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881301+0000","last_change":"2024-03-11T19:55:45.725075+0000","last_active":"2024-03-11T19:57:46.881301+0000","last_peered":"2024-03-11T19:57:46.881301+0000","last_clean":"2024-03-11T19:57:46.881301+0000","last_became_active":"2024-03-11T19:55:45.724627+0000","last_became_peered":"2024-03-11T19:55:45.724627+0000","last_unstale":"2024-03-11T19:57:46.881301+0000","last_undegraded":"2024-03-11T19:57:46.881301+0000","last_fullsized":"2024-03-11T19:57:46.881301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:01.039660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684719+0000","last_change":"2024-03-11T19:55:45.718879+0000","last_active":"2024-03-11T19:57:46.684719+0000","last_peered":"2024-03-11T19:57:46.684719+0000","last_clean":"2024-03-11T19:57:46.684719+0000","last_became_active":"2024-03-11T19:55:45.718535+0000","last_became_peered":"2024-03-11T19:55:45.718535+0000","last_unstale":"2024-03-11T19:57:46.684719+0000","last_undegraded":"2024-03-11T19:57:46.684719+0000","last_fullsized":"2024-03-11T19:57:46.684719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:48.236368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687368+0000","last_change":"2024-03-11T19:55:45.703432+0000","last_active":"2024-03-11T19:57:46.687368+0000","last_peered":"2024-03-11T19:57:46.687368+0000","last_clean":"2024-03-11T19:57:46.687368+0000","last_became_active":"2024-03-11T19:55:45.702669+0000","last_became_peered":"2024-03-11T19:55:45.702669+0000","last_unstale":"2024-03-11T19:57:46.687368+0000","last_undegraded":"2024-03-11T19:57:46.687368+0000","last_fullsized":"2024-03-11T19:57:46.687368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:02.496981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802765+0000","last_change":"2024-03-11T19:55:45.708733+0000","last_active":"2024-03-11T19:57:46.802765+0000","last_peered":"2024-03-11T19:57:46.802765+0000","last_clean":"2024-03-11T19:57:46.802765+0000","last_became_active":"2024-03-11T19:55:45.708369+0000","last_became_peered":"2024-03-11T19:55:45.708369+0000","last_unstale":"2024-03-11T19:57:46.802765+0000","last_undegraded":"2024-03-11T19:57:46.802765+0000","last_fullsized":"2024-03-11T19:57:46.802765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:36.619802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.574156+0000","last_change":"2024-03-11T19:55:45.725698+0000","last_active":"2024-03-11T19:57:47.574156+0000","last_peered":"2024-03-11T19:57:47.574156+0000","last_clean":"2024-03-11T19:57:47.574156+0000","last_became_active":"2024-03-11T19:55:45.724872+0000","last_became_peered":"2024-03-11T19:55:45.724872+0000","last_unstale":"2024-03-11T19:57:47.574156+0000","last_undegraded":"2024-03-11T19:57:47.574156+0000","last_fullsized":"2024-03-11T19:57:47.574156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.982969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680314+0000","last_change":"2024-03-11T19:55:45.726923+0000","last_active":"2024-03-11T19:57:46.680314+0000","last_peered":"2024-03-11T19:57:46.680314+0000","last_clean":"2024-03-11T19:57:46.680314+0000","last_became_active":"2024-03-11T19:55:45.726589+0000","last_became_peered":"2024-03-11T19:55:45.726589+0000","last_unstale":"2024-03-11T19:57:46.680314+0000","last_undegraded":"2024-03-11T19:57:46.680314+0000","last_fullsized":"2024-03-11T19:57:46.680314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:29:15.197359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.690624+0000","last_change":"2024-03-11T19:56:45.698623+0000","last_active":"2024-03-11T19:57:46.690624+0000","last_peered":"2024-03-11T19:57:46.690624+0000","last_clean":"2024-03-11T19:57:46.690624+0000","last_became_active":"2024-03-11T19:56:45.697805+0000","last_became_peered":"2024-03-11T19:56:45.697805+0000","last_unstale":"2024-03-11T19:57:46.690624+0000","last_undegraded":"2024-03-11T19:57:46.690624+0000","last_fullsized":"2024-03-11T19:57:46.690624+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:16:31.624856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023161+0000","last_change":"2024-03-11T19:55:45.729141+0000","last_active":"2024-03-11T19:57:47.023161+0000","last_peered":"2024-03-11T19:57:47.023161+0000","last_clean":"2024-03-11T19:57:47.023161+0000","last_became_active":"2024-03-11T19:55:45.728781+0000","last_became_peered":"2024-03-11T19:55:45.728781+0000","last_unstale":"2024-03-11T19:57:47.023161+0000","last_undegraded":"2024-03-11T19:57:47.023161+0000","last_fullsized":"2024-03-11T19:57:47.023161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:02.939181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684123+0000","last_change":"2024-03-11T19:55:45.723892+0000","last_active":"2024-03-11T19:57:46.684123+0000","last_peered":"2024-03-11T19:57:46.684123+0000","last_clean":"2024-03-11T19:57:46.684123+0000","last_became_active":"2024-03-11T19:55:45.723544+0000","last_became_peered":"2024-03-11T19:55:45.723544+0000","last_unstale":"2024-03-11T19:57:46.684123+0000","last_undegraded":"2024-03-11T19:57:46.684123+0000","last_fullsized":"2024-03-11T19:57:46.684123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:09:08.720854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687440+0000","last_change":"2024-03-11T19:56:45.699717+0000","last_active":"2024-03-11T19:57:46.687440+0000","last_peered":"2024-03-11T19:57:46.687440+0000","last_clean":"2024-03-11T19:57:46.687440+0000","last_became_active":"2024-03-11T19:56:45.699346+0000","last_became_peered":"2024-03-11T19:56:45.699346+0000","last_unstale":"2024-03-11T19:57:46.687440+0000","last_undegraded":"2024-03-11T19:57:46.687440+0000","last_fullsized":"2024-03-11T19:57:46.687440+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:52.920371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68957,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1064,"num_read_kb":613,"num_write":250,"num_write_kb":145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":206871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":217,"ondisk_log_size":217,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1069056,"data_stored":521390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32904,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712248,"statfs":{"total":95995035648,"available":95961341952,"internally_reserved":0,"allocated":1761280,"data_stored":1172659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1093632,"data_stored":524302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070524,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":520002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332161,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36984,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708168,"statfs":{"total":95995035648,"available":95957164032,"internally_reserved":0,"allocated":1683456,"data_stored":1117222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":515710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1130496,"data_stored":578211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738472,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":517698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36464,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708688,"statfs":{"total":95995035648,"available":95957696512,"internally_reserved":0,"allocated":1150976,"data_stored":579495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182487,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":515418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920850,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":519784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757325,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36992,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708160,"statfs":{"total":95995035648,"available":95957155840,"internally_reserved":0,"allocated":1687552,"data_stored":1108708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:58:05.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:05 smithi097 ceph-mon[31898]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:58:05.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:05 smithi097 ceph-mon[31898]: pgmap v393: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:05.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:05 smithi114 ceph-mon[36376]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:58:05.990 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:05 smithi114 ceph-mon[36376]: pgmap v393: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:06.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:05 smithi161 ceph-mon[36332]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:58:06.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:05 smithi161 ceph-mon[36332]: pgmap v393: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:07.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T19:58:07.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:07 smithi097 ceph-mon[31898]: pgmap v394: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:07.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:07 smithi114 ceph-mon[36376]: pgmap v394: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:08.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:07 smithi161 ceph-mon[36332]: pgmap v394: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:08.116 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:58:09.854 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:09 smithi097 ceph-mon[31898]: pgmap v395: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:09.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:09 smithi114 ceph-mon[36376]: pgmap v395: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:10.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:09 smithi161 ceph-mon[36332]: pgmap v395: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.364+0000 7f894e400700 1 -- 172.21.15.97:0/3871020254 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894810e490 msgr2=0x7f8948110900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.364+0000 7f894e400700 1 --2- 172.21.15.97:0/3871020254 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894810e490 0x7f8948110900 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f894000b0a0 tx=0x7f894000b3b0 comp rx=0 tx=0).stop 2024-03-11T19:58:10.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.364+0000 7f894e400700 1 -- 172.21.15.97:0/3871020254 shutdown_connections 2024-03-11T19:58:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.365+0000 7f894e400700 1 --2- 172.21.15.97:0/3871020254 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894810e490 0x7f8948110900 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.365+0000 7f894e400700 1 --2- 172.21.15.97:0/3871020254 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89480740a0 0x7f8948074480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.365+0000 7f894e400700 1 --2- 172.21.15.97:0/3871020254 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89480749c0 0x7f894810df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.365+0000 7f894e400700 1 -- 172.21.15.97:0/3871020254 >> 172.21.15.97:0/3871020254 conn(0x7f89480fd720 msgr2=0x7f89480ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.365+0000 7f894e400700 1 -- 172.21.15.97:0/3871020254 shutdown_connections 2024-03-11T19:58:10.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.365+0000 7f894e400700 1 -- 172.21.15.97:0/3871020254 wait complete. 2024-03-11T19:58:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.366+0000 7f894e400700 1 Processor -- start 2024-03-11T19:58:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.366+0000 7f894e400700 1 -- start start 2024-03-11T19:58:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.366+0000 7f894e400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89480740a0 0x7f894819a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.366+0000 7f894e400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89480749c0 0x7f894819aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.366+0000 7f894e400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894810e490 0x7f894819ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.366+0000 7f894e400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894819f410 con 0x7f894810e490 2024-03-11T19:58:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.366+0000 7f894e400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894819f580 con 0x7f89480749c0 2024-03-11T19:58:10.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.366+0000 7f894e400700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894819f740 con 0x7f89480740a0 2024-03-11T19:58:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.367+0000 7f8946a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89480749c0 0x7f894819aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.367+0000 7f8947e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894810e490 0x7f894819ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.367+0000 7f8946a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89480749c0 0x7f894819aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:35626/0 (socket says 172.21.15.97:35626) 2024-03-11T19:58:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.367+0000 7f8946a00700 1 -- 172.21.15.97:0/2581977112 learned_addr learned my addr 172.21.15.97:0/2581977112 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.367+0000 7f8947400700 1 --2- 172.21.15.97:0/2581977112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89480740a0 0x7f894819a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.367+0000 7f8946a00700 1 -- 172.21.15.97:0/2581977112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89480740a0 msgr2=0x7f894819a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:10.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.367+0000 7f8946a00700 1 --2- 172.21.15.97:0/2581977112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89480740a0 0x7f894819a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.367+0000 7f8946a00700 1 -- 172.21.15.97:0/2581977112 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894810e490 msgr2=0x7f894819ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.367+0000 7f8946a00700 1 --2- 172.21.15.97:0/2581977112 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894810e490 0x7f894819ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.367+0000 7f8946a00700 1 -- 172.21.15.97:0/2581977112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8940009d00 con 0x7f89480749c0 2024-03-11T19:58:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.367+0000 7f8946a00700 1 --2- 172.21.15.97:0/2581977112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89480749c0 0x7f894819aac0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f893400e9a0 tx=0x7f893400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.368+0000 7f893fe00700 1 -- 172.21.15.97:0/2581977112 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934009790 con 0x7f89480749c0 2024-03-11T19:58:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.368+0000 7f894e400700 1 -- 172.21.15.97:0/2581977112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894819fa20 con 0x7f89480749c0 2024-03-11T19:58:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.368+0000 7f893fe00700 1 -- 172.21.15.97:0/2581977112 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8934010430 con 0x7f89480749c0 2024-03-11T19:58:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.368+0000 7f893fe00700 1 -- 172.21.15.97:0/2581977112 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934018680 con 0x7f89480749c0 2024-03-11T19:58:10.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.368+0000 7f894e400700 1 -- 172.21.15.97:0/2581977112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89481a7280 con 0x7f89480749c0 2024-03-11T19:58:10.369 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.369+0000 7f894e400700 1 -- 172.21.15.97:0/2581977112 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8928005330 con 0x7f89480749c0 2024-03-11T19:58:10.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.371+0000 7f893fe00700 1 -- 172.21.15.97:0/2581977112 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f893400ca10 con 0x7f89480749c0 2024-03-11T19:58:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.372+0000 7f893fe00700 1 --2- 172.21.15.97:0/2581977112 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f892c0a1b40 0x7f892c0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.372+0000 7f893fe00700 1 -- 172.21.15.97:0/2581977112 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f8934014070 con 0x7f89480749c0 2024-03-11T19:58:10.371 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.373+0000 7f8947400700 1 --2- 172.21.15.97:0/2581977112 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f892c0a1b40 0x7f892c0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:10.372 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.373+0000 7f8947400700 1 --2- 172.21.15.97:0/2581977112 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f892c0a1b40 0x7f892c0a4000 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8938009c60 tx=0x7f8938009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:10.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.376+0000 7f893fe00700 1 -- 172.21.15.97:0/2581977112 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8934052d20 con 0x7f89480749c0 2024-03-11T19:58:10.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.591+0000 7f894e400700 1 -- 172.21.15.97:0/2581977112 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8928000bc0 con 0x7f892c0a1b40 2024-03-11T19:58:10.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.607+0000 7f893fe00700 1 -- 172.21.15.97:0/2581977112 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340540 (secure 0 0 0) 0x7f8928000bc0 con 0x7f892c0a1b40 2024-03-11T19:58:10.611 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:10.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.613+0000 7f894e400700 1 -- 172.21.15.97:0/2581977112 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f892c0a1b40 msgr2=0x7f892c0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:10.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.613+0000 7f894e400700 1 --2- 172.21.15.97:0/2581977112 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f892c0a1b40 0x7f892c0a4000 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8938009c60 tx=0x7f8938009260 comp rx=0 tx=0).stop 2024-03-11T19:58:10.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.613+0000 7f894e400700 1 -- 172.21.15.97:0/2581977112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89480749c0 msgr2=0x7f894819aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:10.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.613+0000 7f894e400700 1 --2- 172.21.15.97:0/2581977112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89480749c0 0x7f894819aac0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f893400e9a0 tx=0x7f893400ecb0 comp rx=0 tx=0).stop 2024-03-11T19:58:10.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.613+0000 7f894e400700 1 -- 172.21.15.97:0/2581977112 shutdown_connections 2024-03-11T19:58:10.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.613+0000 7f894e400700 1 --2- 172.21.15.97:0/2581977112 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f892c0a1b40 0x7f892c0a4000 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:10.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.613+0000 7f894e400700 1 --2- 172.21.15.97:0/2581977112 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f894810e490 0x7f894819ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:10.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.613+0000 7f894e400700 1 --2- 172.21.15.97:0/2581977112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89480740a0 0x7f894819a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:10.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.613+0000 7f894e400700 1 --2- 172.21.15.97:0/2581977112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f89480749c0 0x7f894819aac0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:10.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.613+0000 7f894e400700 1 -- 172.21.15.97:0/2581977112 >> 172.21.15.97:0/2581977112 conn(0x7f89480fd720 msgr2=0x7f89480fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:10.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.613+0000 7f894e400700 1 -- 172.21.15.97:0/2581977112 shutdown_connections 2024-03-11T19:58:10.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:10.613+0000 7f894e400700 1 -- 172.21.15.97:0/2581977112 wait complete. 2024-03-11T19:58:10.614 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-03-11T19:58:11.545 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-03-11T19:58:10.561160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659559,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1260,"num_read_kb":757,"num_write":433,"num_write_kb":3085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1181435,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":856,"ondisk_log_size":856,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384628,"kb_used_data":14612,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557196,"statfs":{"total":1151940427776,"available":1151546568704,"internally_reserved":0,"allocated":14962688,"data_stored":8190599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":100,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006662"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687052+0000","last_change":"2024-03-11T19:55:45.725019+0000","last_active":"2024-03-11T19:57:46.687052+0000","last_peered":"2024-03-11T19:57:46.687052+0000","last_clean":"2024-03-11T19:57:46.687052+0000","last_became_active":"2024-03-11T19:55:45.724602+0000","last_became_peered":"2024-03-11T19:55:45.724602+0000","last_unstale":"2024-03-11T19:57:46.687052+0000","last_undegraded":"2024-03-11T19:57:46.687052+0000","last_fullsized":"2024-03-11T19:57:46.687052+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:16:18.218381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881861+0000","last_change":"2024-03-11T19:55:45.724589+0000","last_active":"2024-03-11T19:57:46.881861+0000","last_peered":"2024-03-11T19:57:46.881861+0000","last_clean":"2024-03-11T19:57:46.881861+0000","last_became_active":"2024-03-11T19:55:45.724109+0000","last_became_peered":"2024-03-11T19:55:45.724109+0000","last_unstale":"2024-03-11T19:57:46.881861+0000","last_undegraded":"2024-03-11T19:57:46.881861+0000","last_fullsized":"2024-03-11T19:57:46.881861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:58:12.510437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881733+0000","last_change":"2024-03-11T19:55:45.718831+0000","last_active":"2024-03-11T19:57:46.881733+0000","last_peered":"2024-03-11T19:57:46.881733+0000","last_clean":"2024-03-11T19:57:46.881733+0000","last_became_active":"2024-03-11T19:55:45.717652+0000","last_became_peered":"2024-03-11T19:55:45.717652+0000","last_unstale":"2024-03-11T19:57:46.881733+0000","last_undegraded":"2024-03-11T19:57:46.881733+0000","last_fullsized":"2024-03-11T19:57:46.881733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:16.471754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685116+0000","last_change":"2024-03-11T19:55:45.701640+0000","last_active":"2024-03-11T19:57:46.685116+0000","last_peered":"2024-03-11T19:57:46.685116+0000","last_clean":"2024-03-11T19:57:46.685116+0000","last_became_active":"2024-03-11T19:55:45.699207+0000","last_became_peered":"2024-03-11T19:55:45.699207+0000","last_unstale":"2024-03-11T19:57:46.685116+0000","last_undegraded":"2024-03-11T19:57:46.685116+0000","last_fullsized":"2024-03-11T19:57:46.685116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:01.900429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685627+0000","last_change":"2024-03-11T19:55:45.732040+0000","last_active":"2024-03-11T19:57:46.685627+0000","last_peered":"2024-03-11T19:57:46.685627+0000","last_clean":"2024-03-11T19:57:46.685627+0000","last_became_active":"2024-03-11T19:55:45.731647+0000","last_became_peered":"2024-03-11T19:55:45.731647+0000","last_unstale":"2024-03-11T19:57:46.685627+0000","last_undegraded":"2024-03-11T19:57:46.685627+0000","last_fullsized":"2024-03-11T19:57:46.685627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:51:57.345329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683200+0000","last_change":"2024-03-11T19:55:45.707986+0000","last_active":"2024-03-11T19:57:46.683200+0000","last_peered":"2024-03-11T19:57:46.683200+0000","last_clean":"2024-03-11T19:57:46.683200+0000","last_became_active":"2024-03-11T19:55:45.707641+0000","last_became_peered":"2024-03-11T19:55:45.707641+0000","last_unstale":"2024-03-11T19:57:46.683200+0000","last_undegraded":"2024-03-11T19:57:46.683200+0000","last_fullsized":"2024-03-11T19:57:46.683200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:15:50.635234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683168+0000","last_change":"2024-03-11T19:56:45.705737+0000","last_active":"2024-03-11T19:57:46.683168+0000","last_peered":"2024-03-11T19:57:46.683168+0000","last_clean":"2024-03-11T19:57:46.683168+0000","last_became_active":"2024-03-11T19:56:45.705477+0000","last_became_peered":"2024-03-11T19:56:45.705477+0000","last_unstale":"2024-03-11T19:57:46.683168+0000","last_undegraded":"2024-03-11T19:57:46.683168+0000","last_fullsized":"2024-03-11T19:57:46.683168+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:01:14.068476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572051+0000","last_change":"2024-03-11T19:55:45.729587+0000","last_active":"2024-03-11T19:57:47.572051+0000","last_peered":"2024-03-11T19:57:47.572051+0000","last_clean":"2024-03-11T19:57:47.572051+0000","last_became_active":"2024-03-11T19:55:45.729037+0000","last_became_peered":"2024-03-11T19:55:45.729037+0000","last_unstale":"2024-03-11T19:57:47.572051+0000","last_undegraded":"2024-03-11T19:57:47.572051+0000","last_fullsized":"2024-03-11T19:57:47.572051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:55:43.445192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023626+0000","last_change":"2024-03-11T19:55:45.709793+0000","last_active":"2024-03-11T19:57:47.023626+0000","last_peered":"2024-03-11T19:57:47.023626+0000","last_clean":"2024-03-11T19:57:47.023626+0000","last_became_active":"2024-03-11T19:55:45.708631+0000","last_became_peered":"2024-03-11T19:55:45.708631+0000","last_unstale":"2024-03-11T19:57:47.023626+0000","last_undegraded":"2024-03-11T19:57:47.023626+0000","last_fullsized":"2024-03-11T19:57:47.023626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:45:02.144066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684105+0000","last_change":"2024-03-11T19:55:45.704227+0000","last_active":"2024-03-11T19:57:46.684105+0000","last_peered":"2024-03-11T19:57:46.684105+0000","last_clean":"2024-03-11T19:57:46.684105+0000","last_became_active":"2024-03-11T19:55:45.703885+0000","last_became_peered":"2024-03-11T19:55:45.703885+0000","last_unstale":"2024-03-11T19:57:46.684105+0000","last_undegraded":"2024-03-11T19:57:46.684105+0000","last_fullsized":"2024-03-11T19:57:46.684105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:36:21.401988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685372+0000","last_change":"2024-03-11T19:55:45.708950+0000","last_active":"2024-03-11T19:57:46.685372+0000","last_peered":"2024-03-11T19:57:46.685372+0000","last_clean":"2024-03-11T19:57:46.685372+0000","last_became_active":"2024-03-11T19:55:45.708544+0000","last_became_peered":"2024-03-11T19:55:45.708544+0000","last_unstale":"2024-03-11T19:57:46.685372+0000","last_undegraded":"2024-03-11T19:57:46.685372+0000","last_fullsized":"2024-03-11T19:57:46.685372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:06:34.244449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685013+0000","last_change":"2024-03-11T19:55:45.703923+0000","last_active":"2024-03-11T19:57:46.685013+0000","last_peered":"2024-03-11T19:57:46.685013+0000","last_clean":"2024-03-11T19:57:46.685013+0000","last_became_active":"2024-03-11T19:55:45.703595+0000","last_became_peered":"2024-03-11T19:55:45.703595+0000","last_unstale":"2024-03-11T19:57:46.685013+0000","last_undegraded":"2024-03-11T19:57:46.685013+0000","last_fullsized":"2024-03-11T19:57:46.685013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:00:19.683907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683892+0000","last_change":"2024-03-11T19:55:45.707499+0000","last_active":"2024-03-11T19:57:46.683892+0000","last_peered":"2024-03-11T19:57:46.683892+0000","last_clean":"2024-03-11T19:57:46.683892+0000","last_became_active":"2024-03-11T19:55:45.706344+0000","last_became_peered":"2024-03-11T19:55:45.706344+0000","last_unstale":"2024-03-11T19:57:46.683892+0000","last_undegraded":"2024-03-11T19:57:46.683892+0000","last_fullsized":"2024-03-11T19:57:46.683892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:38:24.082332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.880533+0000","last_change":"2024-03-11T19:55:45.722182+0000","last_active":"2024-03-11T19:57:46.880533+0000","last_peered":"2024-03-11T19:57:46.880533+0000","last_clean":"2024-03-11T19:57:46.880533+0000","last_became_active":"2024-03-11T19:55:45.721783+0000","last_became_peered":"2024-03-11T19:55:45.721783+0000","last_unstale":"2024-03-11T19:57:46.880533+0000","last_undegraded":"2024-03-11T19:57:46.880533+0000","last_fullsized":"2024-03-11T19:57:46.880533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:41:08.559002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.880423+0000","last_change":"2024-03-11T19:55:45.718881+0000","last_active":"2024-03-11T19:57:46.880423+0000","last_peered":"2024-03-11T19:57:46.880423+0000","last_clean":"2024-03-11T19:57:46.880423+0000","last_became_active":"2024-03-11T19:55:45.718458+0000","last_became_peered":"2024-03-11T19:55:45.718458+0000","last_unstale":"2024-03-11T19:57:46.880423+0000","last_undegraded":"2024-03-11T19:57:46.880423+0000","last_fullsized":"2024-03-11T19:57:46.880423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:22.075892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023200+0000","last_change":"2024-03-11T19:55:45.726163+0000","last_active":"2024-03-11T19:57:47.023200+0000","last_peered":"2024-03-11T19:57:47.023200+0000","last_clean":"2024-03-11T19:57:47.023200+0000","last_became_active":"2024-03-11T19:55:45.725772+0000","last_became_peered":"2024-03-11T19:55:45.725772+0000","last_unstale":"2024-03-11T19:57:47.023200+0000","last_undegraded":"2024-03-11T19:57:47.023200+0000","last_fullsized":"2024-03-11T19:57:47.023200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:36:19.057920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686834+0000","last_change":"2024-03-11T19:55:45.724787+0000","last_active":"2024-03-11T19:57:46.686834+0000","last_peered":"2024-03-11T19:57:46.686834+0000","last_clean":"2024-03-11T19:57:46.686834+0000","last_became_active":"2024-03-11T19:55:45.724437+0000","last_became_peered":"2024-03-11T19:55:45.724437+0000","last_unstale":"2024-03-11T19:57:46.686834+0000","last_undegraded":"2024-03-11T19:57:46.686834+0000","last_fullsized":"2024-03-11T19:57:46.686834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:55:53.811679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.691205+0000","last_change":"2024-03-11T19:55:45.714934+0000","last_active":"2024-03-11T19:57:46.691205+0000","last_peered":"2024-03-11T19:57:46.691205+0000","last_clean":"2024-03-11T19:57:46.691205+0000","last_became_active":"2024-03-11T19:55:45.709595+0000","last_became_peered":"2024-03-11T19:55:45.709595+0000","last_unstale":"2024-03-11T19:57:46.691205+0000","last_undegraded":"2024-03-11T19:57:46.691205+0000","last_fullsized":"2024-03-11T19:57:46.691205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:48.319073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802841+0000","last_change":"2024-03-11T19:55:45.725747+0000","last_active":"2024-03-11T19:57:46.802841+0000","last_peered":"2024-03-11T19:57:46.802841+0000","last_clean":"2024-03-11T19:57:46.802841+0000","last_became_active":"2024-03-11T19:55:45.725239+0000","last_became_peered":"2024-03-11T19:55:45.725239+0000","last_unstale":"2024-03-11T19:57:46.802841+0000","last_undegraded":"2024-03-11T19:57:46.802841+0000","last_fullsized":"2024-03-11T19:57:46.802841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:51:24.046681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802300+0000","last_change":"2024-03-11T19:55:50.109913+0000","last_active":"2024-03-11T19:57:46.802300+0000","last_peered":"2024-03-11T19:57:46.802300+0000","last_clean":"2024-03-11T19:57:46.802300+0000","last_became_active":"2024-03-11T19:55:50.109471+0000","last_became_peered":"2024-03-11T19:55:50.109471+0000","last_unstale":"2024-03-11T19:57:46.802300+0000","last_undegraded":"2024-03-11T19:57:46.802300+0000","last_fullsized":"2024-03-11T19:57:46.802300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:40.258022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":152,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:58:04.875740+0000","last_change":"2024-03-11T19:55:44.680181+0000","last_active":"2024-03-11T19:58:04.875740+0000","last_peered":"2024-03-11T19:58:04.875740+0000","last_clean":"2024-03-11T19:58:04.875740+0000","last_became_active":"2024-03-11T19:55:44.679807+0000","last_became_peered":"2024-03-11T19:55:44.679807+0000","last_unstale":"2024-03-11T19:58:04.875740+0000","last_undegraded":"2024-03-11T19:58:04.875740+0000","last_fullsized":"2024-03-11T19:58:04.875740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:23:04.532836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573255+0000","last_change":"2024-03-11T19:55:45.723547+0000","last_active":"2024-03-11T19:57:47.573255+0000","last_peered":"2024-03-11T19:57:47.573255+0000","last_clean":"2024-03-11T19:57:47.573255+0000","last_became_active":"2024-03-11T19:55:45.722516+0000","last_became_peered":"2024-03-11T19:55:45.722516+0000","last_unstale":"2024-03-11T19:57:47.573255+0000","last_undegraded":"2024-03-11T19:57:47.573255+0000","last_fullsized":"2024-03-11T19:57:47.573255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:40.271814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683243+0000","last_change":"2024-03-11T19:55:50.102141+0000","last_active":"2024-03-11T19:57:46.683243+0000","last_peered":"2024-03-11T19:57:46.683243+0000","last_clean":"2024-03-11T19:57:46.683243+0000","last_became_active":"2024-03-11T19:55:50.101667+0000","last_became_peered":"2024-03-11T19:55:50.101667+0000","last_unstale":"2024-03-11T19:57:46.683243+0000","last_undegraded":"2024-03-11T19:57:46.683243+0000","last_fullsized":"2024-03-11T19:57:46.683243+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:54:11.453059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573605+0000","last_change":"2024-03-11T19:55:44.684623+0000","last_active":"2024-03-11T19:57:47.573605+0000","last_peered":"2024-03-11T19:57:47.573605+0000","last_clean":"2024-03-11T19:57:47.573605+0000","last_became_active":"2024-03-11T19:55:44.683611+0000","last_became_peered":"2024-03-11T19:55:44.683611+0000","last_unstale":"2024-03-11T19:57:47.573605+0000","last_undegraded":"2024-03-11T19:57:47.573605+0000","last_fullsized":"2024-03-11T19:57:47.573605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:22:50.954364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881778+0000","last_change":"2024-03-11T19:55:45.723619+0000","last_active":"2024-03-11T19:57:46.881778+0000","last_peered":"2024-03-11T19:57:46.881778+0000","last_clean":"2024-03-11T19:57:46.881778+0000","last_became_active":"2024-03-11T19:55:45.722685+0000","last_became_peered":"2024-03-11T19:55:45.722685+0000","last_unstale":"2024-03-11T19:57:46.881778+0000","last_undegraded":"2024-03-11T19:57:46.881778+0000","last_fullsized":"2024-03-11T19:57:46.881778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:47:13.747776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.576246+0000","last_change":"2024-03-11T19:57:47.576246+0000","last_active":"2024-03-11T19:57:47.576246+0000","last_peered":"2024-03-11T19:57:47.576246+0000","last_clean":"2024-03-11T19:57:47.576246+0000","last_became_active":"2024-03-11T19:57:47.575837+0000","last_became_peered":"2024-03-11T19:57:47.575837+0000","last_unstale":"2024-03-11T19:57:47.576246+0000","last_undegraded":"2024-03-11T19:57:47.576246+0000","last_fullsized":"2024-03-11T19:57:47.576246+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:00.914062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687185+0000","last_change":"2024-03-11T19:56:45.714194+0000","last_active":"2024-03-11T19:57:46.687185+0000","last_peered":"2024-03-11T19:57:46.687185+0000","last_clean":"2024-03-11T19:57:46.687185+0000","last_became_active":"2024-03-11T19:56:45.694559+0000","last_became_peered":"2024-03-11T19:56:45.694559+0000","last_unstale":"2024-03-11T19:57:46.687185+0000","last_undegraded":"2024-03-11T19:57:46.687185+0000","last_fullsized":"2024-03-11T19:57:46.687185+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:01:20.307915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023254+0000","last_change":"2024-03-11T19:55:45.715157+0000","last_active":"2024-03-11T19:57:47.023254+0000","last_peered":"2024-03-11T19:57:47.023254+0000","last_clean":"2024-03-11T19:57:47.023254+0000","last_became_active":"2024-03-11T19:55:45.714536+0000","last_became_peered":"2024-03-11T19:55:45.714536+0000","last_unstale":"2024-03-11T19:57:47.023254+0000","last_undegraded":"2024-03-11T19:57:47.023254+0000","last_fullsized":"2024-03-11T19:57:47.023254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:02.416736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.682631+0000","last_change":"2024-03-11T19:55:50.102157+0000","last_active":"2024-03-11T19:57:46.682631+0000","last_peered":"2024-03-11T19:57:46.682631+0000","last_clean":"2024-03-11T19:57:46.682631+0000","last_became_active":"2024-03-11T19:55:50.101715+0000","last_became_peered":"2024-03-11T19:55:50.101715+0000","last_unstale":"2024-03-11T19:57:46.682631+0000","last_undegraded":"2024-03-11T19:57:46.682631+0000","last_fullsized":"2024-03-11T19:57:46.682631+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:12:18.780708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687126+0000","last_change":"2024-03-11T19:56:45.698585+0000","last_active":"2024-03-11T19:57:46.687126+0000","last_peered":"2024-03-11T19:57:46.687126+0000","last_clean":"2024-03-11T19:57:46.687126+0000","last_became_active":"2024-03-11T19:56:45.692929+0000","last_became_peered":"2024-03-11T19:56:45.692929+0000","last_unstale":"2024-03-11T19:57:46.687126+0000","last_undegraded":"2024-03-11T19:57:46.687126+0000","last_fullsized":"2024-03-11T19:57:46.687126+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:43:11.158934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680889+0000","last_change":"2024-03-11T19:55:45.709973+0000","last_active":"2024-03-11T19:57:46.680889+0000","last_peered":"2024-03-11T19:57:46.680889+0000","last_clean":"2024-03-11T19:57:46.680889+0000","last_became_active":"2024-03-11T19:55:45.709645+0000","last_became_peered":"2024-03-11T19:55:45.709645+0000","last_unstale":"2024-03-11T19:57:46.680889+0000","last_undegraded":"2024-03-11T19:57:46.680889+0000","last_fullsized":"2024-03-11T19:57:46.680889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:16.516982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802538+0000","last_change":"2024-03-11T19:55:50.094093+0000","last_active":"2024-03-11T19:57:46.802538+0000","last_peered":"2024-03-11T19:57:46.802538+0000","last_clean":"2024-03-11T19:57:46.802538+0000","last_became_active":"2024-03-11T19:55:50.093653+0000","last_became_peered":"2024-03-11T19:55:50.093653+0000","last_unstale":"2024-03-11T19:57:46.802538+0000","last_undegraded":"2024-03-11T19:57:46.802538+0000","last_fullsized":"2024-03-11T19:57:46.802538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:28:41.668001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":149,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:58:04.797796+0000","last_change":"2024-03-11T19:55:44.671547+0000","last_active":"2024-03-11T19:58:04.797796+0000","last_peered":"2024-03-11T19:58:04.797796+0000","last_clean":"2024-03-11T19:58:04.797796+0000","last_became_active":"2024-03-11T19:55:44.671211+0000","last_became_peered":"2024-03-11T19:55:44.671211+0000","last_unstale":"2024-03-11T19:58:04.797796+0000","last_undegraded":"2024-03-11T19:58:04.797796+0000","last_fullsized":"2024-03-11T19:58:04.797796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:18:46.261250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685626+0000","last_change":"2024-03-11T19:55:45.707968+0000","last_active":"2024-03-11T19:57:46.685626+0000","last_peered":"2024-03-11T19:57:46.685626+0000","last_clean":"2024-03-11T19:57:46.685626+0000","last_became_active":"2024-03-11T19:55:45.706998+0000","last_became_peered":"2024-03-11T19:55:45.706998+0000","last_unstale":"2024-03-11T19:57:46.685626+0000","last_undegraded":"2024-03-11T19:57:46.685626+0000","last_fullsized":"2024-03-11T19:57:46.685626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:49:35.711123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802348+0000","last_change":"2024-03-11T19:55:50.104310+0000","last_active":"2024-03-11T19:57:46.802348+0000","last_peered":"2024-03-11T19:57:46.802348+0000","last_clean":"2024-03-11T19:57:46.802348+0000","last_became_active":"2024-03-11T19:55:50.103831+0000","last_became_peered":"2024-03-11T19:55:50.103831+0000","last_unstale":"2024-03-11T19:57:46.802348+0000","last_undegraded":"2024-03-11T19:57:46.802348+0000","last_fullsized":"2024-03-11T19:57:46.802348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:33.213987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683083+0000","last_change":"2024-03-11T19:55:44.677615+0000","last_active":"2024-03-11T19:57:46.683083+0000","last_peered":"2024-03-11T19:57:46.683083+0000","last_clean":"2024-03-11T19:57:46.683083+0000","last_became_active":"2024-03-11T19:55:44.677153+0000","last_became_peered":"2024-03-11T19:55:44.677153+0000","last_unstale":"2024-03-11T19:57:46.683083+0000","last_undegraded":"2024-03-11T19:57:46.683083+0000","last_fullsized":"2024-03-11T19:57:46.683083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:52:59.153222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680692+0000","last_change":"2024-03-11T19:55:45.731827+0000","last_active":"2024-03-11T19:57:46.680692+0000","last_peered":"2024-03-11T19:57:46.680692+0000","last_clean":"2024-03-11T19:57:46.680692+0000","last_became_active":"2024-03-11T19:55:45.731460+0000","last_became_peered":"2024-03-11T19:55:45.731460+0000","last_unstale":"2024-03-11T19:57:46.680692+0000","last_undegraded":"2024-03-11T19:57:46.680692+0000","last_fullsized":"2024-03-11T19:57:46.680692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:12.277198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686677+0000","last_change":"2024-03-11T19:55:50.109812+0000","last_active":"2024-03-11T19:57:46.686677+0000","last_peered":"2024-03-11T19:57:46.686677+0000","last_clean":"2024-03-11T19:57:46.686677+0000","last_became_active":"2024-03-11T19:55:50.109420+0000","last_became_peered":"2024-03-11T19:55:50.109420+0000","last_unstale":"2024-03-11T19:57:46.686677+0000","last_undegraded":"2024-03-11T19:57:46.686677+0000","last_fullsized":"2024-03-11T19:57:46.686677+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:08:30.105941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683574+0000","last_change":"2024-03-11T19:55:44.683173+0000","last_active":"2024-03-11T19:57:46.683574+0000","last_peered":"2024-03-11T19:57:46.683574+0000","last_clean":"2024-03-11T19:57:46.683574+0000","last_became_active":"2024-03-11T19:55:44.682778+0000","last_became_peered":"2024-03-11T19:55:44.682778+0000","last_unstale":"2024-03-11T19:57:46.683574+0000","last_undegraded":"2024-03-11T19:57:46.683574+0000","last_fullsized":"2024-03-11T19:57:46.683574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:44:15.119864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572517+0000","last_change":"2024-03-11T19:55:45.721833+0000","last_active":"2024-03-11T19:57:47.572517+0000","last_peered":"2024-03-11T19:57:47.572517+0000","last_clean":"2024-03-11T19:57:47.572517+0000","last_became_active":"2024-03-11T19:55:45.721490+0000","last_became_peered":"2024-03-11T19:55:45.721490+0000","last_unstale":"2024-03-11T19:57:47.572517+0000","last_undegraded":"2024-03-11T19:57:47.572517+0000","last_fullsized":"2024-03-11T19:57:47.572517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:09.824912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881479+0000","last_change":"2024-03-11T19:55:50.108424+0000","last_active":"2024-03-11T19:57:46.881479+0000","last_peered":"2024-03-11T19:57:46.881479+0000","last_clean":"2024-03-11T19:57:46.881479+0000","last_became_active":"2024-03-11T19:55:50.107072+0000","last_became_peered":"2024-03-11T19:55:50.107072+0000","last_unstale":"2024-03-11T19:57:46.881479+0000","last_undegraded":"2024-03-11T19:57:46.881479+0000","last_fullsized":"2024-03-11T19:57:46.881479+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:31.751597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680227+0000","last_change":"2024-03-11T19:55:44.671570+0000","last_active":"2024-03-11T19:57:46.680227+0000","last_peered":"2024-03-11T19:57:46.680227+0000","last_clean":"2024-03-11T19:57:46.680227+0000","last_became_active":"2024-03-11T19:55:44.671007+0000","last_became_peered":"2024-03-11T19:55:44.671007+0000","last_unstale":"2024-03-11T19:57:46.680227+0000","last_undegraded":"2024-03-11T19:57:46.680227+0000","last_fullsized":"2024-03-11T19:57:46.680227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:23.865011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.691569+0000","last_change":"2024-03-11T19:55:45.733048+0000","last_active":"2024-03-11T19:57:46.691569+0000","last_peered":"2024-03-11T19:57:46.691569+0000","last_clean":"2024-03-11T19:57:46.691569+0000","last_became_active":"2024-03-11T19:55:45.731656+0000","last_became_peered":"2024-03-11T19:55:45.731656+0000","last_unstale":"2024-03-11T19:57:46.691569+0000","last_undegraded":"2024-03-11T19:57:46.691569+0000","last_fullsized":"2024-03-11T19:57:46.691569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:28:56.433327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572334+0000","last_change":"2024-03-11T19:55:50.105532+0000","last_active":"2024-03-11T19:57:47.572334+0000","last_peered":"2024-03-11T19:57:47.572334+0000","last_clean":"2024-03-11T19:57:47.572334+0000","last_became_active":"2024-03-11T19:55:50.102200+0000","last_became_peered":"2024-03-11T19:55:50.102200+0000","last_unstale":"2024-03-11T19:57:47.572334+0000","last_undegraded":"2024-03-11T19:57:47.572334+0000","last_fullsized":"2024-03-11T19:57:47.572334+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:37.060472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":254,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:58:06.800696+0000","last_change":"2024-03-11T19:55:44.678224+0000","last_active":"2024-03-11T19:58:06.800696+0000","last_peered":"2024-03-11T19:58:06.800696+0000","last_clean":"2024-03-11T19:58:06.800696+0000","last_became_active":"2024-03-11T19:55:44.677936+0000","last_became_peered":"2024-03-11T19:55:44.677936+0000","last_unstale":"2024-03-11T19:58:06.800696+0000","last_undegraded":"2024-03-11T19:58:06.800696+0000","last_fullsized":"2024-03-11T19:58:06.800696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:20:42.486501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":218,"num_read_kb":218,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023625+0000","last_change":"2024-03-11T19:55:45.709917+0000","last_active":"2024-03-11T19:57:47.023625+0000","last_peered":"2024-03-11T19:57:47.023625+0000","last_clean":"2024-03-11T19:57:47.023625+0000","last_became_active":"2024-03-11T19:55:45.709318+0000","last_became_peered":"2024-03-11T19:55:45.709318+0000","last_unstale":"2024-03-11T19:57:47.023625+0000","last_undegraded":"2024-03-11T19:57:47.023625+0000","last_fullsized":"2024-03-11T19:57:47.023625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:08.298158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.883428+0000","last_change":"2024-03-11T19:57:46.882937+0000","last_active":"2024-03-11T19:57:46.883428+0000","last_peered":"2024-03-11T19:57:46.883428+0000","last_clean":"2024-03-11T19:57:46.883428+0000","last_became_active":"2024-03-11T19:57:46.882408+0000","last_became_peered":"2024-03-11T19:57:46.882408+0000","last_unstale":"2024-03-11T19:57:46.883428+0000","last_undegraded":"2024-03-11T19:57:46.883428+0000","last_fullsized":"2024-03-11T19:57:46.883428+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:58:27.842775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685774+0000","last_change":"2024-03-11T19:55:44.676004+0000","last_active":"2024-03-11T19:57:46.685774+0000","last_peered":"2024-03-11T19:57:46.685774+0000","last_clean":"2024-03-11T19:57:46.685774+0000","last_became_active":"2024-03-11T19:55:44.675640+0000","last_became_peered":"2024-03-11T19:55:44.675640+0000","last_unstale":"2024-03-11T19:57:46.685774+0000","last_undegraded":"2024-03-11T19:57:46.685774+0000","last_fullsized":"2024-03-11T19:57:46.685774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:48:15.096780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.882070+0000","last_change":"2024-03-11T19:56:45.702569+0000","last_active":"2024-03-11T19:57:46.882070+0000","last_peered":"2024-03-11T19:57:46.882070+0000","last_clean":"2024-03-11T19:57:46.882070+0000","last_became_active":"2024-03-11T19:56:45.701890+0000","last_became_peered":"2024-03-11T19:56:45.701890+0000","last_unstale":"2024-03-11T19:57:46.882070+0000","last_undegraded":"2024-03-11T19:57:46.882070+0000","last_fullsized":"2024-03-11T19:57:46.882070+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:40.434712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684317+0000","last_change":"2024-03-11T19:55:50.096136+0000","last_active":"2024-03-11T19:57:46.684317+0000","last_peered":"2024-03-11T19:57:46.684317+0000","last_clean":"2024-03-11T19:57:46.684317+0000","last_became_active":"2024-03-11T19:55:50.095670+0000","last_became_peered":"2024-03-11T19:55:50.095670+0000","last_unstale":"2024-03-11T19:57:46.684317+0000","last_undegraded":"2024-03-11T19:57:46.684317+0000","last_fullsized":"2024-03-11T19:57:46.684317+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:40.943273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680839+0000","last_change":"2024-03-11T19:55:44.681547+0000","last_active":"2024-03-11T19:57:46.680839+0000","last_peered":"2024-03-11T19:57:46.680839+0000","last_clean":"2024-03-11T19:57:46.680839+0000","last_became_active":"2024-03-11T19:55:44.681237+0000","last_became_peered":"2024-03-11T19:55:44.681237+0000","last_unstale":"2024-03-11T19:57:46.680839+0000","last_undegraded":"2024-03-11T19:57:46.680839+0000","last_fullsized":"2024-03-11T19:57:46.680839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:08:09.509915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023459+0000","last_change":"2024-03-11T19:55:45.705025+0000","last_active":"2024-03-11T19:57:47.023459+0000","last_peered":"2024-03-11T19:57:47.023459+0000","last_clean":"2024-03-11T19:57:47.023459+0000","last_became_active":"2024-03-11T19:55:45.702680+0000","last_became_peered":"2024-03-11T19:55:45.702680+0000","last_unstale":"2024-03-11T19:57:47.023459+0000","last_undegraded":"2024-03-11T19:57:47.023459+0000","last_fullsized":"2024-03-11T19:57:47.023459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:10:28.245002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.689568+0000","last_change":"2024-03-11T19:55:50.099648+0000","last_active":"2024-03-11T19:57:46.689568+0000","last_peered":"2024-03-11T19:57:46.689568+0000","last_clean":"2024-03-11T19:57:46.689568+0000","last_became_active":"2024-03-11T19:55:50.098516+0000","last_became_peered":"2024-03-11T19:55:50.098516+0000","last_unstale":"2024-03-11T19:57:46.689568+0000","last_undegraded":"2024-03-11T19:57:46.689568+0000","last_fullsized":"2024-03-11T19:57:46.689568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:19.644046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":291,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:58:06.801497+0000","last_change":"2024-03-11T19:55:44.672460+0000","last_active":"2024-03-11T19:58:06.801497+0000","last_peered":"2024-03-11T19:58:06.801497+0000","last_clean":"2024-03-11T19:58:06.801497+0000","last_became_active":"2024-03-11T19:55:44.672123+0000","last_became_peered":"2024-03-11T19:55:44.672123+0000","last_unstale":"2024-03-11T19:58:06.801497+0000","last_undegraded":"2024-03-11T19:58:06.801497+0000","last_fullsized":"2024-03-11T19:58:06.801497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:43.940638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61471,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":225,"num_read_kb":173,"num_write":37,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685900+0000","last_change":"2024-03-11T19:55:45.702933+0000","last_active":"2024-03-11T19:57:46.685900+0000","last_peered":"2024-03-11T19:57:46.685900+0000","last_clean":"2024-03-11T19:57:46.685900+0000","last_became_active":"2024-03-11T19:55:45.702585+0000","last_became_peered":"2024-03-11T19:55:45.702585+0000","last_unstale":"2024-03-11T19:57:46.685900+0000","last_undegraded":"2024-03-11T19:57:46.685900+0000","last_fullsized":"2024-03-11T19:57:46.685900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:54:38.762156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.681155+0000","last_change":"2024-03-11T19:55:50.102571+0000","last_active":"2024-03-11T19:57:46.681155+0000","last_peered":"2024-03-11T19:57:46.681155+0000","last_clean":"2024-03-11T19:57:46.681155+0000","last_became_active":"2024-03-11T19:55:50.102099+0000","last_became_peered":"2024-03-11T19:55:50.102099+0000","last_unstale":"2024-03-11T19:57:46.681155+0000","last_undegraded":"2024-03-11T19:57:46.681155+0000","last_fullsized":"2024-03-11T19:57:46.681155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:17.631378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802459+0000","last_change":"2024-03-11T19:55:44.671027+0000","last_active":"2024-03-11T19:57:46.802459+0000","last_peered":"2024-03-11T19:57:46.802459+0000","last_clean":"2024-03-11T19:57:46.802459+0000","last_became_active":"2024-03-11T19:55:44.669696+0000","last_became_peered":"2024-03-11T19:55:44.669696+0000","last_unstale":"2024-03-11T19:57:46.802459+0000","last_undegraded":"2024-03-11T19:57:46.802459+0000","last_fullsized":"2024-03-11T19:57:46.802459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:48:20.169638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684067+0000","last_change":"2024-03-11T19:55:45.729119+0000","last_active":"2024-03-11T19:57:46.684067+0000","last_peered":"2024-03-11T19:57:46.684067+0000","last_clean":"2024-03-11T19:57:46.684067+0000","last_became_active":"2024-03-11T19:55:45.727442+0000","last_became_peered":"2024-03-11T19:55:45.727442+0000","last_unstale":"2024-03-11T19:57:46.684067+0000","last_undegraded":"2024-03-11T19:57:46.684067+0000","last_fullsized":"2024-03-11T19:57:46.684067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:14:51.483199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686945+0000","last_change":"2024-03-11T19:55:50.104326+0000","last_active":"2024-03-11T19:57:46.686945+0000","last_peered":"2024-03-11T19:57:46.686945+0000","last_clean":"2024-03-11T19:57:46.686945+0000","last_became_active":"2024-03-11T19:55:50.103881+0000","last_became_peered":"2024-03-11T19:55:50.103881+0000","last_unstale":"2024-03-11T19:57:46.686945+0000","last_undegraded":"2024-03-11T19:57:46.686945+0000","last_fullsized":"2024-03-11T19:57:46.686945+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:12:38.819591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680670+0000","last_change":"2024-03-11T19:55:44.684900+0000","last_active":"2024-03-11T19:57:46.680670+0000","last_peered":"2024-03-11T19:57:46.680670+0000","last_clean":"2024-03-11T19:57:46.680670+0000","last_became_active":"2024-03-11T19:55:44.684590+0000","last_became_peered":"2024-03-11T19:55:44.684590+0000","last_unstale":"2024-03-11T19:57:46.680670+0000","last_undegraded":"2024-03-11T19:57:46.680670+0000","last_fullsized":"2024-03-11T19:57:46.680670+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:20:07.744366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684990+0000","last_change":"2024-03-11T19:55:45.704787+0000","last_active":"2024-03-11T19:57:46.684990+0000","last_peered":"2024-03-11T19:57:46.684990+0000","last_clean":"2024-03-11T19:57:46.684990+0000","last_became_active":"2024-03-11T19:55:45.704405+0000","last_became_peered":"2024-03-11T19:55:45.704405+0000","last_unstale":"2024-03-11T19:57:46.684990+0000","last_undegraded":"2024-03-11T19:57:46.684990+0000","last_fullsized":"2024-03-11T19:57:46.684990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:58:05.849570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.690120+0000","last_change":"2024-03-11T19:55:50.104793+0000","last_active":"2024-03-11T19:57:46.690120+0000","last_peered":"2024-03-11T19:57:46.690120+0000","last_clean":"2024-03-11T19:57:46.690120+0000","last_became_active":"2024-03-11T19:55:50.104415+0000","last_became_peered":"2024-03-11T19:55:50.104415+0000","last_unstale":"2024-03-11T19:57:46.690120+0000","last_undegraded":"2024-03-11T19:57:46.690120+0000","last_fullsized":"2024-03-11T19:57:46.690120+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:03:53.307313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":147,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:58:06.828460+0000","last_change":"2024-03-11T19:55:44.669403+0000","last_active":"2024-03-11T19:58:06.828460+0000","last_peered":"2024-03-11T19:58:06.828460+0000","last_clean":"2024-03-11T19:58:06.828460+0000","last_became_active":"2024-03-11T19:55:44.668908+0000","last_became_peered":"2024-03-11T19:55:44.668908+0000","last_unstale":"2024-03-11T19:58:06.828460+0000","last_undegraded":"2024-03-11T19:58:06.828460+0000","last_fullsized":"2024-03-11T19:58:06.828460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:56:36.831506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684407+0000","last_change":"2024-03-11T19:55:45.700029+0000","last_active":"2024-03-11T19:57:46.684407+0000","last_peered":"2024-03-11T19:57:46.684407+0000","last_clean":"2024-03-11T19:57:46.684407+0000","last_became_active":"2024-03-11T19:55:45.697296+0000","last_became_peered":"2024-03-11T19:55:45.697296+0000","last_unstale":"2024-03-11T19:57:46.684407+0000","last_undegraded":"2024-03-11T19:57:46.684407+0000","last_fullsized":"2024-03-11T19:57:46.684407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:59:35.025845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.577523+0000","last_change":"2024-03-11T19:57:47.577523+0000","last_active":"2024-03-11T19:57:47.577523+0000","last_peered":"2024-03-11T19:57:47.577523+0000","last_clean":"2024-03-11T19:57:47.577523+0000","last_became_active":"2024-03-11T19:57:47.575852+0000","last_became_peered":"2024-03-11T19:57:47.575852+0000","last_unstale":"2024-03-11T19:57:47.577523+0000","last_undegraded":"2024-03-11T19:57:47.577523+0000","last_fullsized":"2024-03-11T19:57:47.577523+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:00:25.108706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685404+0000","last_change":"2024-03-11T19:55:44.677491+0000","last_active":"2024-03-11T19:57:46.685404+0000","last_peered":"2024-03-11T19:57:46.685404+0000","last_clean":"2024-03-11T19:57:46.685404+0000","last_became_active":"2024-03-11T19:55:44.677176+0000","last_became_peered":"2024-03-11T19:55:44.677176+0000","last_unstale":"2024-03-11T19:57:46.685404+0000","last_undegraded":"2024-03-11T19:57:46.685404+0000","last_fullsized":"2024-03-11T19:57:46.685404+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:21:51.002538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680376+0000","last_change":"2024-03-11T19:55:45.703310+0000","last_active":"2024-03-11T19:57:46.680376+0000","last_peered":"2024-03-11T19:57:46.680376+0000","last_clean":"2024-03-11T19:57:46.680376+0000","last_became_active":"2024-03-11T19:55:45.702933+0000","last_became_peered":"2024-03-11T19:55:45.702933+0000","last_unstale":"2024-03-11T19:57:46.680376+0000","last_undegraded":"2024-03-11T19:57:46.680376+0000","last_fullsized":"2024-03-11T19:57:46.680376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:07:58.787949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680521+0000","last_change":"2024-03-11T19:55:50.103849+0000","last_active":"2024-03-11T19:57:46.680521+0000","last_peered":"2024-03-11T19:57:46.680521+0000","last_clean":"2024-03-11T19:57:46.680521+0000","last_became_active":"2024-03-11T19:55:50.103360+0000","last_became_peered":"2024-03-11T19:55:50.103360+0000","last_unstale":"2024-03-11T19:57:46.680521+0000","last_undegraded":"2024-03-11T19:57:46.680521+0000","last_fullsized":"2024-03-11T19:57:46.680521+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:38:05.151041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685914+0000","last_change":"2024-03-11T19:55:44.669839+0000","last_active":"2024-03-11T19:57:46.685914+0000","last_peered":"2024-03-11T19:57:46.685914+0000","last_clean":"2024-03-11T19:57:46.685914+0000","last_became_active":"2024-03-11T19:55:44.669494+0000","last_became_peered":"2024-03-11T19:55:44.669494+0000","last_unstale":"2024-03-11T19:57:46.685914+0000","last_undegraded":"2024-03-11T19:57:46.685914+0000","last_fullsized":"2024-03-11T19:57:46.685914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:00.313847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687060+0000","last_change":"2024-03-11T19:56:45.693864+0000","last_active":"2024-03-11T19:57:46.687060+0000","last_peered":"2024-03-11T19:57:46.687060+0000","last_clean":"2024-03-11T19:57:46.687060+0000","last_became_active":"2024-03-11T19:56:45.693091+0000","last_became_peered":"2024-03-11T19:56:45.693091+0000","last_unstale":"2024-03-11T19:57:46.687060+0000","last_undegraded":"2024-03-11T19:57:46.687060+0000","last_fullsized":"2024-03-11T19:57:46.687060+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:26:37.233485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685560+0000","last_change":"2024-03-11T19:55:50.143916+0000","last_active":"2024-03-11T19:57:46.685560+0000","last_peered":"2024-03-11T19:57:46.685560+0000","last_clean":"2024-03-11T19:57:46.685560+0000","last_became_active":"2024-03-11T19:55:50.143341+0000","last_became_peered":"2024-03-11T19:55:50.143341+0000","last_unstale":"2024-03-11T19:57:46.685560+0000","last_undegraded":"2024-03-11T19:57:46.685560+0000","last_fullsized":"2024-03-11T19:57:46.685560+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:40.637529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.691467+0000","last_change":"2024-03-11T19:55:44.688468+0000","last_active":"2024-03-11T19:57:46.691467+0000","last_peered":"2024-03-11T19:57:46.691467+0000","last_clean":"2024-03-11T19:57:46.691467+0000","last_became_active":"2024-03-11T19:55:44.688065+0000","last_became_peered":"2024-03-11T19:55:44.688065+0000","last_unstale":"2024-03-11T19:57:46.691467+0000","last_undegraded":"2024-03-11T19:57:46.691467+0000","last_fullsized":"2024-03-11T19:57:46.691467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:29:23.392749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683940+0000","last_change":"2024-03-11T19:55:45.728970+0000","last_active":"2024-03-11T19:57:46.683940+0000","last_peered":"2024-03-11T19:57:46.683940+0000","last_clean":"2024-03-11T19:57:46.683940+0000","last_became_active":"2024-03-11T19:55:45.728516+0000","last_became_peered":"2024-03-11T19:55:45.728516+0000","last_unstale":"2024-03-11T19:57:46.683940+0000","last_undegraded":"2024-03-11T19:57:46.683940+0000","last_fullsized":"2024-03-11T19:57:46.683940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:19.242610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684235+0000","last_change":"2024-03-11T19:55:50.111884+0000","last_active":"2024-03-11T19:57:46.684235+0000","last_peered":"2024-03-11T19:57:46.684235+0000","last_clean":"2024-03-11T19:57:46.684235+0000","last_became_active":"2024-03-11T19:55:50.110802+0000","last_became_peered":"2024-03-11T19:55:50.110802+0000","last_unstale":"2024-03-11T19:57:46.684235+0000","last_undegraded":"2024-03-11T19:57:46.684235+0000","last_fullsized":"2024-03-11T19:57:46.684235+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:35.866368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":146,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:58:05.879436+0000","last_change":"2024-03-11T19:55:44.677156+0000","last_active":"2024-03-11T19:58:05.879436+0000","last_peered":"2024-03-11T19:58:05.879436+0000","last_clean":"2024-03-11T19:58:05.879436+0000","last_became_active":"2024-03-11T19:55:44.676656+0000","last_became_peered":"2024-03-11T19:55:44.676656+0000","last_unstale":"2024-03-11T19:58:05.879436+0000","last_undegraded":"2024-03-11T19:58:05.879436+0000","last_fullsized":"2024-03-11T19:58:05.879436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:40:27.521669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":109,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683235+0000","last_change":"2024-03-11T19:55:45.715545+0000","last_active":"2024-03-11T19:57:46.683235+0000","last_peered":"2024-03-11T19:57:46.683235+0000","last_clean":"2024-03-11T19:57:46.683235+0000","last_became_active":"2024-03-11T19:55:45.715042+0000","last_became_peered":"2024-03-11T19:55:45.715042+0000","last_unstale":"2024-03-11T19:57:46.683235+0000","last_undegraded":"2024-03-11T19:57:46.683235+0000","last_fullsized":"2024-03-11T19:57:46.683235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:06:33.362235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685165+0000","last_change":"2024-03-11T19:55:50.145603+0000","last_active":"2024-03-11T19:57:46.685165+0000","last_peered":"2024-03-11T19:57:46.685165+0000","last_clean":"2024-03-11T19:57:46.685165+0000","last_became_active":"2024-03-11T19:55:50.143689+0000","last_became_peered":"2024-03-11T19:55:50.143689+0000","last_unstale":"2024-03-11T19:57:46.685165+0000","last_undegraded":"2024-03-11T19:57:46.685165+0000","last_fullsized":"2024-03-11T19:57:46.685165+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:37:57.289983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680431+0000","last_change":"2024-03-11T19:55:45.702011+0000","last_active":"2024-03-11T19:57:46.680431+0000","last_peered":"2024-03-11T19:57:46.680431+0000","last_clean":"2024-03-11T19:57:46.680431+0000","last_became_active":"2024-03-11T19:55:45.701673+0000","last_became_peered":"2024-03-11T19:55:45.701673+0000","last_unstale":"2024-03-11T19:57:46.680431+0000","last_undegraded":"2024-03-11T19:57:46.680431+0000","last_fullsized":"2024-03-11T19:57:46.680431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:17.028276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":146,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:58:04.823486+0000","last_change":"2024-03-11T19:55:44.689121+0000","last_active":"2024-03-11T19:58:04.823486+0000","last_peered":"2024-03-11T19:58:04.823486+0000","last_clean":"2024-03-11T19:58:04.823486+0000","last_became_active":"2024-03-11T19:55:44.688453+0000","last_became_peered":"2024-03-11T19:55:44.688453+0000","last_unstale":"2024-03-11T19:58:04.823486+0000","last_undegraded":"2024-03-11T19:58:04.823486+0000","last_fullsized":"2024-03-11T19:58:04.823486+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:31.940675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":109,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.698656+0000","last_change":"2024-03-11T19:57:46.698213+0000","last_active":"2024-03-11T19:57:46.698656+0000","last_peered":"2024-03-11T19:57:46.698656+0000","last_clean":"2024-03-11T19:57:46.698656+0000","last_became_active":"2024-03-11T19:57:46.693710+0000","last_became_peered":"2024-03-11T19:57:46.693710+0000","last_unstale":"2024-03-11T19:57:46.698656+0000","last_undegraded":"2024-03-11T19:57:46.698656+0000","last_fullsized":"2024-03-11T19:57:46.698656+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:06:00.214092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.691362+0000","last_change":"2024-03-11T19:56:45.703746+0000","last_active":"2024-03-11T19:57:46.691362+0000","last_peered":"2024-03-11T19:57:46.691362+0000","last_clean":"2024-03-11T19:57:46.691362+0000","last_became_active":"2024-03-11T19:56:45.701252+0000","last_became_peered":"2024-03-11T19:56:45.701252+0000","last_unstale":"2024-03-11T19:57:46.691362+0000","last_undegraded":"2024-03-11T19:57:46.691362+0000","last_fullsized":"2024-03-11T19:57:46.691362+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:16:50.219593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684249+0000","last_change":"2024-03-11T19:55:44.672644+0000","last_active":"2024-03-11T19:57:46.684249+0000","last_peered":"2024-03-11T19:57:46.684249+0000","last_clean":"2024-03-11T19:57:46.684249+0000","last_became_active":"2024-03-11T19:55:44.671645+0000","last_became_peered":"2024-03-11T19:55:44.671645+0000","last_unstale":"2024-03-11T19:57:46.684249+0000","last_undegraded":"2024-03-11T19:57:46.684249+0000","last_fullsized":"2024-03-11T19:57:46.684249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:25:46.637768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":367,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684193+0000","last_change":"2024-03-11T19:54:20.915891+0000","last_active":"2024-03-11T19:57:46.684193+0000","last_peered":"2024-03-11T19:57:46.684193+0000","last_clean":"2024-03-11T19:57:46.684193+0000","last_became_active":"2024-03-11T19:54:20.915450+0000","last_became_peered":"2024-03-11T19:54:20.915450+0000","last_unstale":"2024-03-11T19:57:46.684193+0000","last_undegraded":"2024-03-11T19:57:46.684193+0000","last_fullsized":"2024-03-11T19:57:46.684193+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:50:29.052858+0000","last_clean_scrub_stamp":"2024-03-11T19:50:29.052858+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:47.966076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680928+0000","last_change":"2024-03-11T19:55:50.144053+0000","last_active":"2024-03-11T19:57:46.680928+0000","last_peered":"2024-03-11T19:57:46.680928+0000","last_clean":"2024-03-11T19:57:46.680928+0000","last_became_active":"2024-03-11T19:55:50.143509+0000","last_became_peered":"2024-03-11T19:55:50.143509+0000","last_unstale":"2024-03-11T19:57:46.680928+0000","last_undegraded":"2024-03-11T19:57:46.680928+0000","last_fullsized":"2024-03-11T19:57:46.680928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:57:15.448812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.683348+0000","last_change":"2024-03-11T19:55:45.712621+0000","last_active":"2024-03-11T19:57:46.683348+0000","last_peered":"2024-03-11T19:57:46.683348+0000","last_clean":"2024-03-11T19:57:46.683348+0000","last_became_active":"2024-03-11T19:55:45.712161+0000","last_became_peered":"2024-03-11T19:55:45.712161+0000","last_unstale":"2024-03-11T19:57:46.683348+0000","last_undegraded":"2024-03-11T19:57:46.683348+0000","last_fullsized":"2024-03-11T19:57:46.683348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:34:52.911944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684355+0000","last_change":"2024-03-11T19:55:44.674606+0000","last_active":"2024-03-11T19:57:46.684355+0000","last_peered":"2024-03-11T19:57:46.684355+0000","last_clean":"2024-03-11T19:57:46.684355+0000","last_became_active":"2024-03-11T19:55:44.674355+0000","last_became_peered":"2024-03-11T19:55:44.674355+0000","last_unstale":"2024-03-11T19:57:46.684355+0000","last_undegraded":"2024-03-11T19:57:46.684355+0000","last_fullsized":"2024-03-11T19:57:46.684355+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:31:01.662582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686781+0000","last_change":"2024-03-11T19:55:50.101468+0000","last_active":"2024-03-11T19:57:46.686781+0000","last_peered":"2024-03-11T19:57:46.686781+0000","last_clean":"2024-03-11T19:57:46.686781+0000","last_became_active":"2024-03-11T19:55:50.101040+0000","last_became_peered":"2024-03-11T19:55:50.101040+0000","last_unstale":"2024-03-11T19:57:46.686781+0000","last_undegraded":"2024-03-11T19:57:46.686781+0000","last_fullsized":"2024-03-11T19:57:46.686781+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:55.354176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686046+0000","last_change":"2024-03-11T19:55:44.681467+0000","last_active":"2024-03-11T19:57:46.686046+0000","last_peered":"2024-03-11T19:57:46.686046+0000","last_clean":"2024-03-11T19:57:46.686046+0000","last_became_active":"2024-03-11T19:55:44.681176+0000","last_became_peered":"2024-03-11T19:55:44.681176+0000","last_unstale":"2024-03-11T19:57:46.686046+0000","last_undegraded":"2024-03-11T19:57:46.686046+0000","last_fullsized":"2024-03-11T19:57:46.686046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:27:38.019116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685401+0000","last_change":"2024-03-11T19:55:45.709798+0000","last_active":"2024-03-11T19:57:46.685401+0000","last_peered":"2024-03-11T19:57:46.685401+0000","last_clean":"2024-03-11T19:57:46.685401+0000","last_became_active":"2024-03-11T19:55:45.709277+0000","last_became_peered":"2024-03-11T19:55:45.709277+0000","last_unstale":"2024-03-11T19:57:46.685401+0000","last_undegraded":"2024-03-11T19:57:46.685401+0000","last_fullsized":"2024-03-11T19:57:46.685401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:58.852316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.572588+0000","last_change":"2024-03-11T19:55:50.146601+0000","last_active":"2024-03-11T19:57:47.572588+0000","last_peered":"2024-03-11T19:57:47.572588+0000","last_clean":"2024-03-11T19:57:47.572588+0000","last_became_active":"2024-03-11T19:55:50.144584+0000","last_became_peered":"2024-03-11T19:55:50.144584+0000","last_unstale":"2024-03-11T19:57:47.572588+0000","last_undegraded":"2024-03-11T19:57:47.572588+0000","last_fullsized":"2024-03-11T19:57:47.572588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:00.365713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'14","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.690894+0000","last_change":"2024-03-11T19:55:44.689109+0000","last_active":"2024-03-11T19:57:46.690894+0000","last_peered":"2024-03-11T19:57:46.690894+0000","last_clean":"2024-03-11T19:57:46.690894+0000","last_became_active":"2024-03-11T19:55:44.687323+0000","last_became_peered":"2024-03-11T19:55:44.687323+0000","last_unstale":"2024-03-11T19:57:46.690894+0000","last_undegraded":"2024-03-11T19:57:46.690894+0000","last_fullsized":"2024-03-11T19:57:46.690894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:24.613901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573872+0000","last_change":"2024-03-11T19:55:45.726033+0000","last_active":"2024-03-11T19:57:47.573872+0000","last_peered":"2024-03-11T19:57:47.573872+0000","last_clean":"2024-03-11T19:57:47.573872+0000","last_became_active":"2024-03-11T19:55:45.725353+0000","last_became_peered":"2024-03-11T19:55:45.725353+0000","last_unstale":"2024-03-11T19:57:47.573872+0000","last_undegraded":"2024-03-11T19:57:47.573872+0000","last_fullsized":"2024-03-11T19:57:47.573872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:36:24.341870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802159+0000","last_change":"2024-03-11T19:55:50.089496+0000","last_active":"2024-03-11T19:57:46.802159+0000","last_peered":"2024-03-11T19:57:46.802159+0000","last_clean":"2024-03-11T19:57:46.802159+0000","last_became_active":"2024-03-11T19:55:50.089072+0000","last_became_peered":"2024-03-11T19:55:50.089072+0000","last_unstale":"2024-03-11T19:57:46.802159+0000","last_undegraded":"2024-03-11T19:57:46.802159+0000","last_fullsized":"2024-03-11T19:57:46.802159+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:27:32.539333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685271+0000","last_change":"2024-03-11T19:55:44.673147+0000","last_active":"2024-03-11T19:57:46.685271+0000","last_peered":"2024-03-11T19:57:46.685271+0000","last_clean":"2024-03-11T19:57:46.685271+0000","last_became_active":"2024-03-11T19:55:44.672787+0000","last_became_peered":"2024-03-11T19:55:44.672787+0000","last_unstale":"2024-03-11T19:57:46.685271+0000","last_undegraded":"2024-03-11T19:57:46.685271+0000","last_fullsized":"2024-03-11T19:57:46.685271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:11:23.864419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.882032+0000","last_change":"2024-03-11T19:55:45.732376+0000","last_active":"2024-03-11T19:57:46.882032+0000","last_peered":"2024-03-11T19:57:46.882032+0000","last_clean":"2024-03-11T19:57:46.882032+0000","last_became_active":"2024-03-11T19:55:45.731956+0000","last_became_peered":"2024-03-11T19:55:45.731956+0000","last_unstale":"2024-03-11T19:57:46.882032+0000","last_undegraded":"2024-03-11T19:57:46.882032+0000","last_fullsized":"2024-03-11T19:57:46.882032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:43:48.423678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685199+0000","last_change":"2024-03-11T19:55:50.144080+0000","last_active":"2024-03-11T19:57:46.685199+0000","last_peered":"2024-03-11T19:57:46.685199+0000","last_clean":"2024-03-11T19:57:46.685199+0000","last_became_active":"2024-03-11T19:55:50.143585+0000","last_became_peered":"2024-03-11T19:55:50.143585+0000","last_unstale":"2024-03-11T19:57:46.685199+0000","last_undegraded":"2024-03-11T19:57:46.685199+0000","last_fullsized":"2024-03-11T19:57:46.685199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:47:51.388823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023178+0000","last_change":"2024-03-11T19:55:44.677354+0000","last_active":"2024-03-11T19:57:47.023178+0000","last_peered":"2024-03-11T19:57:47.023178+0000","last_clean":"2024-03-11T19:57:47.023178+0000","last_became_active":"2024-03-11T19:55:44.677013+0000","last_became_peered":"2024-03-11T19:55:44.677013+0000","last_unstale":"2024-03-11T19:57:47.023178+0000","last_undegraded":"2024-03-11T19:57:47.023178+0000","last_fullsized":"2024-03-11T19:57:47.023178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:50:48.855320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685452+0000","last_change":"2024-03-11T19:55:45.702278+0000","last_active":"2024-03-11T19:57:46.685452+0000","last_peered":"2024-03-11T19:57:46.685452+0000","last_clean":"2024-03-11T19:57:46.685452+0000","last_became_active":"2024-03-11T19:55:45.701843+0000","last_became_peered":"2024-03-11T19:55:45.701843+0000","last_unstale":"2024-03-11T19:57:46.685452+0000","last_undegraded":"2024-03-11T19:57:46.685452+0000","last_fullsized":"2024-03-11T19:57:46.685452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:55:28.351093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.690903+0000","last_change":"2024-03-11T19:55:50.106083+0000","last_active":"2024-03-11T19:57:46.690903+0000","last_peered":"2024-03-11T19:57:46.690903+0000","last_clean":"2024-03-11T19:57:46.690903+0000","last_became_active":"2024-03-11T19:55:50.105591+0000","last_became_peered":"2024-03-11T19:55:50.105591+0000","last_unstale":"2024-03-11T19:57:46.690903+0000","last_undegraded":"2024-03-11T19:57:46.690903+0000","last_fullsized":"2024-03-11T19:57:46.690903+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:34:58.713672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.573135+0000","last_change":"2024-03-11T19:55:44.688128+0000","last_active":"2024-03-11T19:57:47.573135+0000","last_peered":"2024-03-11T19:57:47.573135+0000","last_clean":"2024-03-11T19:57:47.573135+0000","last_became_active":"2024-03-11T19:55:44.684165+0000","last_became_peered":"2024-03-11T19:55:44.684165+0000","last_unstale":"2024-03-11T19:57:47.573135+0000","last_undegraded":"2024-03-11T19:57:47.573135+0000","last_fullsized":"2024-03-11T19:57:47.573135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:37:19.415960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686783+0000","last_change":"2024-03-11T19:55:45.718968+0000","last_active":"2024-03-11T19:57:46.686783+0000","last_peered":"2024-03-11T19:57:46.686783+0000","last_clean":"2024-03-11T19:57:46.686783+0000","last_became_active":"2024-03-11T19:55:45.718625+0000","last_became_peered":"2024-03-11T19:55:45.718625+0000","last_unstale":"2024-03-11T19:57:46.686783+0000","last_undegraded":"2024-03-11T19:57:46.686783+0000","last_fullsized":"2024-03-11T19:57:46.686783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:27.197227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684534+0000","last_change":"2024-03-11T19:55:50.097617+0000","last_active":"2024-03-11T19:57:46.684534+0000","last_peered":"2024-03-11T19:57:46.684534+0000","last_clean":"2024-03-11T19:57:46.684534+0000","last_became_active":"2024-03-11T19:55:50.097137+0000","last_became_peered":"2024-03-11T19:55:50.097137+0000","last_unstale":"2024-03-11T19:57:46.684534+0000","last_undegraded":"2024-03-11T19:57:46.684534+0000","last_fullsized":"2024-03-11T19:57:46.684534+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:19:20.995137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802304+0000","last_change":"2024-03-11T19:55:44.669944+0000","last_active":"2024-03-11T19:57:46.802304+0000","last_peered":"2024-03-11T19:57:46.802304+0000","last_clean":"2024-03-11T19:57:46.802304+0000","last_became_active":"2024-03-11T19:55:44.669309+0000","last_became_peered":"2024-03-11T19:55:44.669309+0000","last_unstale":"2024-03-11T19:57:46.802304+0000","last_undegraded":"2024-03-11T19:57:46.802304+0000","last_fullsized":"2024-03-11T19:57:46.802304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:06:41.489579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.574541+0000","last_change":"2024-03-11T19:55:45.731348+0000","last_active":"2024-03-11T19:57:47.574541+0000","last_peered":"2024-03-11T19:57:47.574541+0000","last_clean":"2024-03-11T19:57:47.574541+0000","last_became_active":"2024-03-11T19:55:45.730957+0000","last_became_peered":"2024-03-11T19:55:45.730957+0000","last_unstale":"2024-03-11T19:57:47.574541+0000","last_undegraded":"2024-03-11T19:57:47.574541+0000","last_fullsized":"2024-03-11T19:57:47.574541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:06:50.422511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684484+0000","last_change":"2024-03-11T19:55:50.144050+0000","last_active":"2024-03-11T19:57:46.684484+0000","last_peered":"2024-03-11T19:57:46.684484+0000","last_clean":"2024-03-11T19:57:46.684484+0000","last_became_active":"2024-03-11T19:55:50.143591+0000","last_became_peered":"2024-03-11T19:55:50.143591+0000","last_unstale":"2024-03-11T19:57:46.684484+0000","last_undegraded":"2024-03-11T19:57:46.684484+0000","last_fullsized":"2024-03-11T19:57:46.684484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:27:02.006665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685567+0000","last_change":"2024-03-11T19:55:44.671973+0000","last_active":"2024-03-11T19:57:46.685567+0000","last_peered":"2024-03-11T19:57:46.685567+0000","last_clean":"2024-03-11T19:57:46.685567+0000","last_became_active":"2024-03-11T19:55:44.671661+0000","last_became_peered":"2024-03-11T19:55:44.671661+0000","last_unstale":"2024-03-11T19:57:46.685567+0000","last_undegraded":"2024-03-11T19:57:46.685567+0000","last_fullsized":"2024-03-11T19:57:46.685567+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:45:35.747386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684815+0000","last_change":"2024-03-11T19:55:45.705545+0000","last_active":"2024-03-11T19:57:46.684815+0000","last_peered":"2024-03-11T19:57:46.684815+0000","last_clean":"2024-03-11T19:57:46.684815+0000","last_became_active":"2024-03-11T19:55:45.705083+0000","last_became_peered":"2024-03-11T19:55:45.705083+0000","last_unstale":"2024-03-11T19:57:46.684815+0000","last_undegraded":"2024-03-11T19:57:46.684815+0000","last_fullsized":"2024-03-11T19:57:46.684815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:02:58.219026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023054+0000","last_change":"2024-03-11T19:55:50.097951+0000","last_active":"2024-03-11T19:57:47.023054+0000","last_peered":"2024-03-11T19:57:47.023054+0000","last_clean":"2024-03-11T19:57:47.023054+0000","last_became_active":"2024-03-11T19:55:50.097446+0000","last_became_peered":"2024-03-11T19:55:50.097446+0000","last_unstale":"2024-03-11T19:57:47.023054+0000","last_undegraded":"2024-03-11T19:57:47.023054+0000","last_fullsized":"2024-03-11T19:57:47.023054+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:12.105801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'7","reported_seq":152,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:58:05.842794+0000","last_change":"2024-03-11T19:55:44.673136+0000","last_active":"2024-03-11T19:58:05.842794+0000","last_peered":"2024-03-11T19:58:05.842794+0000","last_clean":"2024-03-11T19:58:05.842794+0000","last_became_active":"2024-03-11T19:55:44.672467+0000","last_became_peered":"2024-03-11T19:55:44.672467+0000","last_unstale":"2024-03-11T19:58:05.842794+0000","last_undegraded":"2024-03-11T19:58:05.842794+0000","last_fullsized":"2024-03-11T19:58:05.842794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:46:05.931253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":113,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685653+0000","last_change":"2024-03-11T19:55:45.710561+0000","last_active":"2024-03-11T19:57:46.685653+0000","last_peered":"2024-03-11T19:57:46.685653+0000","last_clean":"2024-03-11T19:57:46.685653+0000","last_became_active":"2024-03-11T19:55:45.710164+0000","last_became_peered":"2024-03-11T19:55:45.710164+0000","last_unstale":"2024-03-11T19:57:46.685653+0000","last_undegraded":"2024-03-11T19:57:46.685653+0000","last_fullsized":"2024-03-11T19:57:46.685653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:00:33.977595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.880353+0000","last_change":"2024-03-11T19:55:50.113390+0000","last_active":"2024-03-11T19:57:46.880353+0000","last_peered":"2024-03-11T19:57:46.880353+0000","last_clean":"2024-03-11T19:57:46.880353+0000","last_became_active":"2024-03-11T19:55:50.112656+0000","last_became_peered":"2024-03-11T19:55:50.112656+0000","last_unstale":"2024-03-11T19:57:46.880353+0000","last_undegraded":"2024-03-11T19:57:46.880353+0000","last_fullsized":"2024-03-11T19:57:46.880353+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:13:11.952585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.681043+0000","last_change":"2024-03-11T19:55:44.684907+0000","last_active":"2024-03-11T19:57:46.681043+0000","last_peered":"2024-03-11T19:57:46.681043+0000","last_clean":"2024-03-11T19:57:46.681043+0000","last_became_active":"2024-03-11T19:55:44.684600+0000","last_became_peered":"2024-03-11T19:55:44.684600+0000","last_unstale":"2024-03-11T19:57:46.681043+0000","last_undegraded":"2024-03-11T19:57:46.681043+0000","last_fullsized":"2024-03-11T19:57:46.681043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:12:33.666756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685219+0000","last_change":"2024-03-11T19:55:45.723431+0000","last_active":"2024-03-11T19:57:46.685219+0000","last_peered":"2024-03-11T19:57:46.685219+0000","last_clean":"2024-03-11T19:57:46.685219+0000","last_became_active":"2024-03-11T19:55:45.723135+0000","last_became_peered":"2024-03-11T19:55:45.723135+0000","last_unstale":"2024-03-11T19:57:46.685219+0000","last_undegraded":"2024-03-11T19:57:46.685219+0000","last_fullsized":"2024-03-11T19:57:46.685219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:12:35.654094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680159+0000","last_change":"2024-03-11T19:55:50.092876+0000","last_active":"2024-03-11T19:57:46.680159+0000","last_peered":"2024-03-11T19:57:46.680159+0000","last_clean":"2024-03-11T19:57:46.680159+0000","last_became_active":"2024-03-11T19:55:50.092478+0000","last_became_peered":"2024-03-11T19:55:50.092478+0000","last_unstale":"2024-03-11T19:57:46.680159+0000","last_undegraded":"2024-03-11T19:57:46.680159+0000","last_fullsized":"2024-03-11T19:57:46.680159+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:49.056657+0000","last_clean_scrub_stamp":"2024-03-11T19:55:49.056657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:39.248325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685058+0000","last_change":"2024-03-11T19:56:45.711269+0000","last_active":"2024-03-11T19:57:46.685058+0000","last_peered":"2024-03-11T19:57:46.685058+0000","last_clean":"2024-03-11T19:57:46.685058+0000","last_became_active":"2024-03-11T19:56:45.704657+0000","last_became_peered":"2024-03-11T19:56:45.704657+0000","last_unstale":"2024-03-11T19:57:46.685058+0000","last_undegraded":"2024-03-11T19:57:46.685058+0000","last_fullsized":"2024-03-11T19:57:46.685058+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:43.569699+0000","last_clean_scrub_stamp":"2024-03-11T19:55:43.569699+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:48:51.549965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.685966+0000","last_change":"2024-03-11T19:55:45.702924+0000","last_active":"2024-03-11T19:57:46.685966+0000","last_peered":"2024-03-11T19:57:46.685966+0000","last_clean":"2024-03-11T19:57:46.685966+0000","last_became_active":"2024-03-11T19:55:45.702530+0000","last_became_peered":"2024-03-11T19:55:45.702530+0000","last_unstale":"2024-03-11T19:57:46.685966+0000","last_undegraded":"2024-03-11T19:57:46.685966+0000","last_fullsized":"2024-03-11T19:57:46.685966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:14.679964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.686802+0000","last_change":"2024-03-11T19:55:45.706123+0000","last_active":"2024-03-11T19:57:46.686802+0000","last_peered":"2024-03-11T19:57:46.686802+0000","last_clean":"2024-03-11T19:57:46.686802+0000","last_became_active":"2024-03-11T19:55:45.705706+0000","last_became_peered":"2024-03-11T19:55:45.705706+0000","last_unstale":"2024-03-11T19:57:46.686802+0000","last_undegraded":"2024-03-11T19:57:46.686802+0000","last_fullsized":"2024-03-11T19:57:46.686802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:42:58.989823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802706+0000","last_change":"2024-03-11T19:55:45.703188+0000","last_active":"2024-03-11T19:57:46.802706+0000","last_peered":"2024-03-11T19:57:46.802706+0000","last_clean":"2024-03-11T19:57:46.802706+0000","last_became_active":"2024-03-11T19:55:45.702803+0000","last_became_peered":"2024-03-11T19:55:45.702803+0000","last_unstale":"2024-03-11T19:57:46.802706+0000","last_undegraded":"2024-03-11T19:57:46.802706+0000","last_fullsized":"2024-03-11T19:57:46.802706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:39:20.922963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802270+0000","last_change":"2024-03-11T19:55:45.728856+0000","last_active":"2024-03-11T19:57:46.802270+0000","last_peered":"2024-03-11T19:57:46.802270+0000","last_clean":"2024-03-11T19:57:46.802270+0000","last_became_active":"2024-03-11T19:55:45.728526+0000","last_became_peered":"2024-03-11T19:55:45.728526+0000","last_unstale":"2024-03-11T19:57:46.802270+0000","last_undegraded":"2024-03-11T19:57:46.802270+0000","last_fullsized":"2024-03-11T19:57:46.802270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:25:30.836073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.881301+0000","last_change":"2024-03-11T19:55:45.725075+0000","last_active":"2024-03-11T19:57:46.881301+0000","last_peered":"2024-03-11T19:57:46.881301+0000","last_clean":"2024-03-11T19:57:46.881301+0000","last_became_active":"2024-03-11T19:55:45.724627+0000","last_became_peered":"2024-03-11T19:55:45.724627+0000","last_unstale":"2024-03-11T19:57:46.881301+0000","last_undegraded":"2024-03-11T19:57:46.881301+0000","last_fullsized":"2024-03-11T19:57:46.881301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:01.039660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684719+0000","last_change":"2024-03-11T19:55:45.718879+0000","last_active":"2024-03-11T19:57:46.684719+0000","last_peered":"2024-03-11T19:57:46.684719+0000","last_clean":"2024-03-11T19:57:46.684719+0000","last_became_active":"2024-03-11T19:55:45.718535+0000","last_became_peered":"2024-03-11T19:55:45.718535+0000","last_unstale":"2024-03-11T19:57:46.684719+0000","last_undegraded":"2024-03-11T19:57:46.684719+0000","last_fullsized":"2024-03-11T19:57:46.684719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:48.236368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687368+0000","last_change":"2024-03-11T19:55:45.703432+0000","last_active":"2024-03-11T19:57:46.687368+0000","last_peered":"2024-03-11T19:57:46.687368+0000","last_clean":"2024-03-11T19:57:46.687368+0000","last_became_active":"2024-03-11T19:55:45.702669+0000","last_became_peered":"2024-03-11T19:55:45.702669+0000","last_unstale":"2024-03-11T19:57:46.687368+0000","last_undegraded":"2024-03-11T19:57:46.687368+0000","last_fullsized":"2024-03-11T19:57:46.687368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:18:02.496981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.802765+0000","last_change":"2024-03-11T19:55:45.708733+0000","last_active":"2024-03-11T19:57:46.802765+0000","last_peered":"2024-03-11T19:57:46.802765+0000","last_clean":"2024-03-11T19:57:46.802765+0000","last_became_active":"2024-03-11T19:55:45.708369+0000","last_became_peered":"2024-03-11T19:55:45.708369+0000","last_unstale":"2024-03-11T19:57:46.802765+0000","last_undegraded":"2024-03-11T19:57:46.802765+0000","last_fullsized":"2024-03-11T19:57:46.802765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:44:36.619802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.574156+0000","last_change":"2024-03-11T19:55:45.725698+0000","last_active":"2024-03-11T19:57:47.574156+0000","last_peered":"2024-03-11T19:57:47.574156+0000","last_clean":"2024-03-11T19:57:47.574156+0000","last_became_active":"2024-03-11T19:55:45.724872+0000","last_became_peered":"2024-03-11T19:55:45.724872+0000","last_unstale":"2024-03-11T19:57:47.574156+0000","last_undegraded":"2024-03-11T19:57:47.574156+0000","last_fullsized":"2024-03-11T19:57:47.574156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:55.982969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.680314+0000","last_change":"2024-03-11T19:55:45.726923+0000","last_active":"2024-03-11T19:57:46.680314+0000","last_peered":"2024-03-11T19:57:46.680314+0000","last_clean":"2024-03-11T19:57:46.680314+0000","last_became_active":"2024-03-11T19:55:45.726589+0000","last_became_peered":"2024-03-11T19:55:45.726589+0000","last_unstale":"2024-03-11T19:57:46.680314+0000","last_undegraded":"2024-03-11T19:57:46.680314+0000","last_fullsized":"2024-03-11T19:57:46.680314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:29:15.197359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.690624+0000","last_change":"2024-03-11T19:56:45.698623+0000","last_active":"2024-03-11T19:57:46.690624+0000","last_peered":"2024-03-11T19:57:46.690624+0000","last_clean":"2024-03-11T19:57:46.690624+0000","last_became_active":"2024-03-11T19:56:45.697805+0000","last_became_peered":"2024-03-11T19:56:45.697805+0000","last_unstale":"2024-03-11T19:57:46.690624+0000","last_undegraded":"2024-03-11T19:57:46.690624+0000","last_fullsized":"2024-03-11T19:57:46.690624+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:16:31.624856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:47.023245+0000","last_change":"2024-03-11T19:55:45.729141+0000","last_active":"2024-03-11T19:57:47.023245+0000","last_peered":"2024-03-11T19:57:47.023245+0000","last_clean":"2024-03-11T19:57:47.023245+0000","last_became_active":"2024-03-11T19:55:45.728781+0000","last_became_peered":"2024-03-11T19:55:45.728781+0000","last_unstale":"2024-03-11T19:57:47.023245+0000","last_undegraded":"2024-03-11T19:57:47.023245+0000","last_fullsized":"2024-03-11T19:57:47.023245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:02.939181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.684123+0000","last_change":"2024-03-11T19:55:45.723892+0000","last_active":"2024-03-11T19:57:46.684123+0000","last_peered":"2024-03-11T19:57:46.684123+0000","last_clean":"2024-03-11T19:57:46.684123+0000","last_became_active":"2024-03-11T19:55:45.723544+0000","last_became_peered":"2024-03-11T19:55:45.723544+0000","last_unstale":"2024-03-11T19:57:46.684123+0000","last_undegraded":"2024-03-11T19:57:46.684123+0000","last_fullsized":"2024-03-11T19:57:46.684123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:09:08.720854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T19:57:46.687496+0000","last_change":"2024-03-11T19:56:45.699717+0000","last_active":"2024-03-11T19:57:46.687496+0000","last_peered":"2024-03-11T19:57:46.687496+0000","last_clean":"2024-03-11T19:57:46.687496+0000","last_became_active":"2024-03-11T19:56:45.699346+0000","last_became_peered":"2024-03-11T19:56:45.699346+0000","last_unstale":"2024-03-11T19:57:46.687496+0000","last_undegraded":"2024-03-11T19:57:46.687496+0000","last_fullsized":"2024-03-11T19:57:46.687496+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T19:55:44.652279+0000","last_clean_scrub_stamp":"2024-03-11T19:55:44.652279+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:20:52.920371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68957,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1124,"num_read_kb":638,"num_write":250,"num_write_kb":145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":582,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":206871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":217,"ondisk_log_size":217,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383724,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1069056,"data_stored":521390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32904,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712248,"statfs":{"total":95995035648,"available":95961341952,"internally_reserved":0,"allocated":1761280,"data_stored":1172659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1093632,"data_stored":524302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070526,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":520002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332163,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36984,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708168,"statfs":{"total":95995035648,"available":95957164032,"internally_reserved":0,"allocated":1683456,"data_stored":1117222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495686,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":515710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1130496,"data_stored":578211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":517698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36464,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708688,"statfs":{"total":95995035648,"available":95957696512,"internally_reserved":0,"allocated":1150976,"data_stored":579495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182489,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":515418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920851,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":519784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757326,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36992,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708160,"statfs":{"total":95995035648,"available":95957155840,"internally_reserved":0,"allocated":1687552,"data_stored":1108708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T19:58:11.550 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T19:58:11.550 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T19:58:11.550 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T19:58:11.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T19:58:11.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:11 smithi097 ceph-mon[31898]: pgmap v396: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:11.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:11 smithi097 ceph-mon[31898]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:58:11.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:11 smithi114 ceph-mon[36376]: pgmap v396: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:11.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:11 smithi114 ceph-mon[36376]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:58:12.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:11 smithi161 ceph-mon[36332]: pgmap v396: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:12.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:11 smithi161 ceph-mon[36332]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T19:58:12.040 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:58:13.799 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:13 smithi097 ceph-mon[31898]: pgmap v397: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:13.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:13 smithi114 ceph-mon[36376]: pgmap v397: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:14.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:13 smithi161 ceph-mon[36332]: pgmap v397: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:14.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.316+0000 7f9dbae00700 1 -- 172.21.15.97:0/931656633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9db41084e0 msgr2=0x7f9db41088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:14.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.316+0000 7f9dbae00700 1 --2- 172.21.15.97:0/931656633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9db41084e0 0x7f9db41088c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9da4009990 tx=0x7f9da4009ca0 comp rx=0 tx=0).stop 2024-03-11T19:58:14.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.316+0000 7f9dbae00700 1 -- 172.21.15.97:0/931656633 shutdown_connections 2024-03-11T19:58:14.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.316+0000 7f9dbae00700 1 --2- 172.21.15.97:0/931656633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9db41084e0 0x7f9db41088c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:14.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.316+0000 7f9dbae00700 1 --2- 172.21.15.97:0/931656633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9db410d5c0 0x7f9db4115190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:14.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.316+0000 7f9dbae00700 1 --2- 172.21.15.97:0/931656633 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9db4108e90 0x7f9db410cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:14.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.316+0000 7f9dbae00700 1 -- 172.21.15.97:0/931656633 >> 172.21.15.97:0/931656633 conn(0x7f9db4075440 msgr2=0x7f9db4077860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:14.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.317+0000 7f9dbae00700 1 -- 172.21.15.97:0/931656633 shutdown_connections 2024-03-11T19:58:14.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.317+0000 7f9dbae00700 1 -- 172.21.15.97:0/931656633 wait complete. 2024-03-11T19:58:14.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.318+0000 7f9dbae00700 1 Processor -- start 2024-03-11T19:58:14.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.318+0000 7f9dbae00700 1 -- start start 2024-03-11T19:58:14.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.318+0000 7f9dbae00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9db41084e0 0x7f9db41a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:14.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.318+0000 7f9dbae00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9db4108e90 0x7f9db41a4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:14.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.318+0000 7f9dbae00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9db410d5c0 0x7f9db419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:14.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.318+0000 7f9dbae00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db419ec50 con 0x7f9db4108e90 2024-03-11T19:58:14.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.318+0000 7f9dbae00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db419ed90 con 0x7f9db410d5c0 2024-03-11T19:58:14.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.318+0000 7f9dbae00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db419ef50 con 0x7f9db41084e0 2024-03-11T19:58:14.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.319+0000 7f9dab400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9db4108e90 0x7f9db41a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:14.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.319+0000 7f9dab400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9db4108e90 0x7f9db41a4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38566/0 (socket says 172.21.15.97:38566) 2024-03-11T19:58:14.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.319+0000 7f9dab400700 1 -- 172.21.15.97:0/3859138939 learned_addr learned my addr 172.21.15.97:0/3859138939 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:14.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.319+0000 7f9db3e00700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9db41084e0 0x7f9db41a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:14.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.319+0000 7f9db8e00700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9db410d5c0 0x7f9db419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:14.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.319+0000 7f9db3e00700 1 -- 172.21.15.97:0/3859138939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9db410d5c0 msgr2=0x7f9db419e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:14.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.319+0000 7f9db3e00700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9db410d5c0 0x7f9db419e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:14.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.319+0000 7f9db3e00700 1 -- 172.21.15.97:0/3859138939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9db4108e90 msgr2=0x7f9db41a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:14.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.319+0000 7f9db3e00700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9db4108e90 0x7f9db41a4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:14.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.319+0000 7f9db3e00700 1 -- 172.21.15.97:0/3859138939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da40095f0 con 0x7f9db41084e0 2024-03-11T19:58:14.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.319+0000 7f9dab400700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9db4108e90 0x7f9db41a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:14.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.319+0000 7f9db3e00700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9db41084e0 0x7f9db41a4430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9da40059d0 tx=0x7f9da4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:14.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.320+0000 7f9db1600700 1 -- 172.21.15.97:0/3859138939 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da401d070 con 0x7f9db41084e0 2024-03-11T19:58:14.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.320+0000 7f9dbae00700 1 -- 172.21.15.97:0/3859138939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db419f1d0 con 0x7f9db41084e0 2024-03-11T19:58:14.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.320+0000 7f9db1600700 1 -- 172.21.15.97:0/3859138939 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9da400b610 con 0x7f9db41084e0 2024-03-11T19:58:14.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.320+0000 7f9db1600700 1 -- 172.21.15.97:0/3859138939 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da4022e80 con 0x7f9db41084e0 2024-03-11T19:58:14.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.320+0000 7f9dbae00700 1 -- 172.21.15.97:0/3859138939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db41ab4d0 con 0x7f9db41084e0 2024-03-11T19:58:14.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.321+0000 7f9dbae00700 1 -- 172.21.15.97:0/3859138939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db4110880 con 0x7f9db41084e0 2024-03-11T19:58:14.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.323+0000 7f9db1600700 1 -- 172.21.15.97:0/3859138939 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9da4022500 con 0x7f9db41084e0 2024-03-11T19:58:14.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.324+0000 7f9db1600700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9d980a1b40 0x7f9d980a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:14.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.324+0000 7f9db1600700 1 -- 172.21.15.97:0/3859138939 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f9da40cedc0 con 0x7f9db41084e0 2024-03-11T19:58:14.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.325+0000 7f9dab400700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9d980a1b40 0x7f9d980a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:14.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.325+0000 7f9dab400700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9d980a1b40 0x7f9d980a4000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9db41075e0 tx=0x7f9d9c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:14.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.327+0000 7f9db1600700 1 -- 172.21.15.97:0/3859138939 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9da4097c40 con 0x7f9db41084e0 2024-03-11T19:58:14.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.592+0000 7f9dbae00700 1 -- 172.21.15.97:0/3859138939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9db41099c0 con 0x7f9db41084e0 2024-03-11T19:58:14.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.593+0000 7f9db1600700 1 -- 172.21.15.97:0/3859138939 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f9da409b300 con 0x7f9db41084e0 2024-03-11T19:58:14.591 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:14.591 INFO:teuthology.orchestra.run.smithi097.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T19:58:14.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.596+0000 7f9dbae00700 1 -- 172.21.15.97:0/3859138939 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9d980a1b40 msgr2=0x7f9d980a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:14.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.596+0000 7f9dbae00700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9d980a1b40 0x7f9d980a4000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9db41075e0 tx=0x7f9d9c00b040 comp rx=0 tx=0).stop 2024-03-11T19:58:14.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.596+0000 7f9dbae00700 1 -- 172.21.15.97:0/3859138939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9db41084e0 msgr2=0x7f9db41a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:14.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.596+0000 7f9dbae00700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9db41084e0 0x7f9db41a4430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9da40059d0 tx=0x7f9da4004730 comp rx=0 tx=0).stop 2024-03-11T19:58:14.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.596+0000 7f9dbae00700 1 -- 172.21.15.97:0/3859138939 shutdown_connections 2024-03-11T19:58:14.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.596+0000 7f9dbae00700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9d980a1b40 0x7f9d980a4000 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:14.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.596+0000 7f9dbae00700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9db410d5c0 0x7f9db419e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:14.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.596+0000 7f9dbae00700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9db41084e0 0x7f9db41a4430 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:14.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.596+0000 7f9dbae00700 1 --2- 172.21.15.97:0/3859138939 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9db4108e90 0x7f9db41a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:14.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.596+0000 7f9dbae00700 1 -- 172.21.15.97:0/3859138939 >> 172.21.15.97:0/3859138939 conn(0x7f9db4075440 msgr2=0x7f9db4077710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:14.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.596+0000 7f9dbae00700 1 -- 172.21.15.97:0/3859138939 shutdown_connections 2024-03-11T19:58:14.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:14.596+0000 7f9dbae00700 1 -- 172.21.15.97:0/3859138939 wait complete. 2024-03-11T19:58:14.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:14 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3859138939' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T19:58:14.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:14 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3859138939' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T19:58:15.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:14 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3859138939' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T19:58:15.574 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T19:58:15.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:58:15.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.797+0000 7fe429e00640 1 -- 172.21.15.97:0/89360779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4241031d0 msgr2=0x7fe4241035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:15.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.797+0000 7fe429e00640 1 --2- 172.21.15.97:0/89360779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4241031d0 0x7fe4241035f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe418009900 tx=0x7fe41802bc40 comp rx=0 tx=0).stop 2024-03-11T19:58:15.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.797+0000 7fe429e00640 1 -- 172.21.15.97:0/89360779 shutdown_connections 2024-03-11T19:58:15.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.797+0000 7fe429e00640 1 --2- 172.21.15.97:0/89360779 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe424104d80 0x7fe42410b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:15.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.797+0000 7fe429e00640 1 --2- 172.21.15.97:0/89360779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4241043c0 0x7fe424104840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:15.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.797+0000 7fe429e00640 1 --2- 172.21.15.97:0/89360779 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4241031d0 0x7fe4241035f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:15.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.797+0000 7fe429e00640 1 -- 172.21.15.97:0/89360779 >> 172.21.15.97:0/89360779 conn(0x7fe4240fe760 msgr2=0x7fe424100b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:15.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.797+0000 7fe429e00640 1 -- 172.21.15.97:0/89360779 shutdown_connections 2024-03-11T19:58:15.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.797+0000 7fe429e00640 1 -- 172.21.15.97:0/89360779 wait complete. 2024-03-11T19:58:15.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.798+0000 7fe429e00640 1 Processor -- start 2024-03-11T19:58:15.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.798+0000 7fe429e00640 1 -- start start 2024-03-11T19:58:15.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.798+0000 7fe429e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4241031d0 0x7fe424199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:15.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.798+0000 7fe429e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4241043c0 0x7fe4241997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:15.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.798+0000 7fe429e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe424104d80 0x7fe4241a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:15.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.798+0000 7fe429e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4241a0d70 con 0x7fe4241043c0 2024-03-11T19:58:15.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.798+0000 7fe429e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4241a0ee0 con 0x7fe424104d80 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.798+0000 7fe429e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4241a10a0 con 0x7fe4241031d0 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.799+0000 7fe423e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4241043c0 0x7fe4241997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.799+0000 7fe423e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4241043c0 0x7fe4241997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38584/0 (socket says 172.21.15.97:38584) 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.799+0000 7fe423e00640 1 -- 172.21.15.97:0/3643600117 learned_addr learned my addr 172.21.15.97:0/3643600117 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.799+0000 7fe429400640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe424104d80 0x7fe4241a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.799+0000 7fe428a00640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4241031d0 0x7fe424199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.799+0000 7fe423e00640 1 -- 172.21.15.97:0/3643600117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4241031d0 msgr2=0x7fe424199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.799+0000 7fe423e00640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4241031d0 0x7fe424199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.799+0000 7fe423e00640 1 -- 172.21.15.97:0/3643600117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe424104d80 msgr2=0x7fe4241a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.799+0000 7fe423e00640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe424104d80 0x7fe4241a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.799+0000 7fe423e00640 1 -- 172.21.15.97:0/3643600117 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe418009560 con 0x7fe4241043c0 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.799+0000 7fe423e00640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4241043c0 0x7fe4241997c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe41000e9f0 tx=0x7fe41000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.800+0000 7fe421600640 1 -- 172.21.15.97:0/3643600117 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410009700 con 0x7fe4241043c0 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.800+0000 7fe421600640 1 -- 172.21.15.97:0/3643600117 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe41000cd80 con 0x7fe4241043c0 2024-03-11T19:58:15.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.800+0000 7fe429e00640 1 -- 172.21.15.97:0/3643600117 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4241a1380 con 0x7fe4241043c0 2024-03-11T19:58:15.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.800+0000 7fe421600640 1 -- 172.21.15.97:0/3643600117 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410018920 con 0x7fe4241043c0 2024-03-11T19:58:15.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.800+0000 7fe429e00640 1 -- 172.21.15.97:0/3643600117 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4241a1890 con 0x7fe4241043c0 2024-03-11T19:58:15.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.801+0000 7fe429e00640 1 -- 172.21.15.97:0/3643600117 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe42410a490 con 0x7fe4241043c0 2024-03-11T19:58:15.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.803+0000 7fe421600640 1 -- 172.21.15.97:0/3643600117 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe410018a80 con 0x7fe4241043c0 2024-03-11T19:58:15.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.805+0000 7fe421600640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe40c0a1a30 0x7fe40c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:15.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.805+0000 7fe421600640 1 -- 172.21.15.97:0/3643600117 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fe410014070 con 0x7fe4241043c0 2024-03-11T19:58:15.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.805+0000 7fe428a00640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe40c0a1a30 0x7fe40c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:15.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.806+0000 7fe428a00640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe40c0a1a30 0x7fe40c0a3ef0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe41802c250 tx=0x7fe41802c7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:15.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:15.809+0000 7fe421600640 1 -- 172.21.15.97:0/3643600117 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe410052d20 con 0x7fe4241043c0 2024-03-11T19:58:15.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:15 smithi097 ceph-mon[31898]: pgmap v398: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:15.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:15 smithi114 ceph-mon[36376]: pgmap v398: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:16.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.009+0000 7fe429e00640 1 -- 172.21.15.97:0/3643600117 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe42402ce40 con 0x7fe4241043c0 2024-03-11T19:58:16.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.012+0000 7fe421600640 1 -- 172.21.15.97:0/3643600117 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24838 (secure 0 0 0) 0x7fe410096a30 con 0x7fe4241043c0 2024-03-11T19:58:16.011 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:16.011 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":93,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:57:46.674203+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:58:16.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.014+0000 7fe429e00640 1 -- 172.21.15.97:0/3643600117 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe40c0a1a30 msgr2=0x7fe40c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:16.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.015+0000 7fe429e00640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe40c0a1a30 0x7fe40c0a3ef0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe41802c250 tx=0x7fe41802c7f0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.015+0000 7fe429e00640 1 -- 172.21.15.97:0/3643600117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4241043c0 msgr2=0x7fe4241997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:16.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.015+0000 7fe429e00640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4241043c0 0x7fe4241997c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe41000e9f0 tx=0x7fe41000eeb0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.015+0000 7fe429e00640 1 -- 172.21.15.97:0/3643600117 shutdown_connections 2024-03-11T19:58:16.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.015+0000 7fe429e00640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe424104d80 0x7fe4241a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.015+0000 7fe429e00640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe40c0a1a30 0x7fe40c0a3ef0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.015+0000 7fe429e00640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe4241043c0 0x7fe4241997c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.015+0000 7fe429e00640 1 --2- 172.21.15.97:0/3643600117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4241031d0 0x7fe424199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.015+0000 7fe429e00640 1 -- 172.21.15.97:0/3643600117 >> 172.21.15.97:0/3643600117 conn(0x7fe4240fe760 msgr2=0x7fe424107660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:16.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.015+0000 7fe429e00640 1 -- 172.21.15.97:0/3643600117 shutdown_connections 2024-03-11T19:58:16.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.015+0000 7fe429e00640 1 -- 172.21.15.97:0/3643600117 wait complete. 2024-03-11T19:58:16.034 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:50:29.051985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:55:43.424076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:55:44.652160+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T19:55:48.598077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T19:58:16.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:58:16.035 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:15 smithi161 ceph-mon[36332]: pgmap v398: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:16.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.248+0000 7fb31ec00640 1 -- 172.21.15.97:0/432247964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb318109350 msgr2=0x7fb31810fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:16.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.248+0000 7fb31ec00640 1 --2- 172.21.15.97:0/432247964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb318109350 0x7fb31810fbe0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb310009ca0 tx=0x7fb31002c0f0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.249+0000 7fb31ec00640 1 -- 172.21.15.97:0/432247964 shutdown_connections 2024-03-11T19:58:16.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.249+0000 7fb31ec00640 1 --2- 172.21.15.97:0/432247964 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb318109350 0x7fb31810fbe0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.249+0000 7fb31ec00640 1 --2- 172.21.15.97:0/432247964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb318108990 0x7fb318108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.249+0000 7fb31ec00640 1 --2- 172.21.15.97:0/432247964 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb3181077a0 0x7fb318107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.249+0000 7fb31ec00640 1 -- 172.21.15.97:0/432247964 >> 172.21.15.97:0/432247964 conn(0x7fb318076740 msgr2=0x7fb318078b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:16.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.249+0000 7fb31ec00640 1 -- 172.21.15.97:0/432247964 shutdown_connections 2024-03-11T19:58:16.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.249+0000 7fb31ec00640 1 -- 172.21.15.97:0/432247964 wait complete. 2024-03-11T19:58:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.250+0000 7fb31ec00640 1 Processor -- start 2024-03-11T19:58:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.250+0000 7fb31ec00640 1 -- start start 2024-03-11T19:58:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.250+0000 7fb31ec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3181077a0 0x7fb31819d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.250+0000 7fb31ec00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb318108990 0x7fb31819dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.250+0000 7fb31ec00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb318109350 0x7fb3181a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.250+0000 7fb31ec00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3181a5310 con 0x7fb318108990 2024-03-11T19:58:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.250+0000 7fb31ec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3181a5480 con 0x7fb3181077a0 2024-03-11T19:58:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.250+0000 7fb31ec00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3181a5640 con 0x7fb318109350 2024-03-11T19:58:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.250+0000 7fb30f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb318108990 0x7fb31819dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.250+0000 7fb30f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb318108990 0x7fb31819dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38602/0 (socket says 172.21.15.97:38602) 2024-03-11T19:58:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.250+0000 7fb30f400640 1 -- 172.21.15.97:0/2756290841 learned_addr learned my addr 172.21.15.97:0/2756290841 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:16.250 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.250+0000 7fb31cc00640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb318109350 0x7fb3181a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:16.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.250+0000 7fb317e00640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3181077a0 0x7fb31819d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:16.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.251+0000 7fb30f400640 1 -- 172.21.15.97:0/2756290841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb318109350 msgr2=0x7fb3181a4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:16.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.251+0000 7fb30f400640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb318109350 0x7fb3181a4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.251+0000 7fb30f400640 1 -- 172.21.15.97:0/2756290841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3181077a0 msgr2=0x7fb31819d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:16.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.251+0000 7fb30f400640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3181077a0 0x7fb31819d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.251+0000 7fb30f400640 1 -- 172.21.15.97:0/2756290841 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb308009490 con 0x7fb318108990 2024-03-11T19:58:16.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.251+0000 7fb30f400640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb318108990 0x7fb31819dd60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb30000e9f0 tx=0x7fb30000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:16.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.251+0000 7fb315600640 1 -- 172.21.15.97:0/2756290841 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb300009700 con 0x7fb318108990 2024-03-11T19:58:16.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.251+0000 7fb31ec00640 1 -- 172.21.15.97:0/2756290841 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb310009900 con 0x7fb318108990 2024-03-11T19:58:16.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.251+0000 7fb315600640 1 -- 172.21.15.97:0/2756290841 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb30000cd80 con 0x7fb318108990 2024-03-11T19:58:16.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.251+0000 7fb315600640 1 -- 172.21.15.97:0/2756290841 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb300018920 con 0x7fb318108990 2024-03-11T19:58:16.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.251+0000 7fb31ec00640 1 -- 172.21.15.97:0/2756290841 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3181a5ca0 con 0x7fb318108990 2024-03-11T19:58:16.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.252+0000 7fb31ec00640 1 -- 172.21.15.97:0/2756290841 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31810eac0 con 0x7fb318108990 2024-03-11T19:58:16.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.254+0000 7fb315600640 1 -- 172.21.15.97:0/2756290841 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb300018a80 con 0x7fb318108990 2024-03-11T19:58:16.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.256+0000 7fb315600640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb2fc0a1a30 0x7fb2fc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:16.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.256+0000 7fb315600640 1 -- 172.21.15.97:0/2756290841 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fb300014070 con 0x7fb318108990 2024-03-11T19:58:16.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.256+0000 7fb317e00640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb2fc0a1a30 0x7fb2fc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:16.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.257+0000 7fb317e00640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb2fc0a1a30 0x7fb2fc0a3ef0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb3181087f0 tx=0x7fb308005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:16.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.260+0000 7fb315600640 1 -- 172.21.15.97:0/2756290841 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb300052d20 con 0x7fb318108990 2024-03-11T19:58:16.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.456+0000 7fb31ec00640 1 -- 172.21.15.97:0/2756290841 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb31802ce40 con 0x7fb318108990 2024-03-11T19:58:16.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.457+0000 7fb315600640 1 -- 172.21.15.97:0/2756290841 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb300096a30 con 0x7fb318108990 2024-03-11T19:58:16.456 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-11T19:58:16.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.460+0000 7fb31ec00640 1 -- 172.21.15.97:0/2756290841 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb2fc0a1a30 msgr2=0x7fb2fc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:16.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.460+0000 7fb31ec00640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb2fc0a1a30 0x7fb2fc0a3ef0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb3181087f0 tx=0x7fb308005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:16.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.460+0000 7fb31ec00640 1 -- 172.21.15.97:0/2756290841 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb318108990 msgr2=0x7fb31819dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:16.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.460+0000 7fb31ec00640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb318108990 0x7fb31819dd60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb30000e9f0 tx=0x7fb30000eeb0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.460+0000 7fb31ec00640 1 -- 172.21.15.97:0/2756290841 shutdown_connections 2024-03-11T19:58:16.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.460+0000 7fb31ec00640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb318109350 0x7fb3181a4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.460+0000 7fb31ec00640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb2fc0a1a30 0x7fb2fc0a3ef0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.460+0000 7fb31ec00640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb318108990 0x7fb31819dd60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.460+0000 7fb31ec00640 1 --2- 172.21.15.97:0/2756290841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3181077a0 0x7fb31819d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.460+0000 7fb31ec00640 1 -- 172.21.15.97:0/2756290841 >> 172.21.15.97:0/2756290841 conn(0x7fb318076740 msgr2=0x7fb31810bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:16.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.460+0000 7fb31ec00640 1 -- 172.21.15.97:0/2756290841 shutdown_connections 2024-03-11T19:58:16.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.460+0000 7fb31ec00640 1 -- 172.21.15.97:0/2756290841 wait complete. 2024-03-11T19:58:16.478 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:58:16.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.689+0000 7fefc8600640 1 -- 172.21.15.97:0/2410671655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefc0102360 msgr2=0x7fefc01047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:16.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.689+0000 7fefc8600640 1 --2- 172.21.15.97:0/2410671655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefc0102360 0x7fefc01047d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fefac009900 tx=0x7fefac02bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:16.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.689+0000 7fefc8600640 1 -- 172.21.15.97:0/2410671655 shutdown_connections 2024-03-11T19:58:16.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.689+0000 7fefc8600640 1 --2- 172.21.15.97:0/2410671655 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fefc0104d10 0x7fefc0105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.689+0000 7fefc8600640 1 --2- 172.21.15.97:0/2410671655 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefc0102360 0x7fefc01047d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.689+0000 7fefc8600640 1 --2- 172.21.15.97:0/2410671655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefc00ff9f0 0x7fefc0101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.689+0000 7fefc8600640 1 -- 172.21.15.97:0/2410671655 >> 172.21.15.97:0/2410671655 conn(0x7fefc00fb610 msgr2=0x7fefc00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:16.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.689+0000 7fefc8600640 1 -- 172.21.15.97:0/2410671655 shutdown_connections 2024-03-11T19:58:16.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.690+0000 7fefc8600640 1 -- 172.21.15.97:0/2410671655 wait complete. 2024-03-11T19:58:16.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.690+0000 7fefc8600640 1 Processor -- start 2024-03-11T19:58:16.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.691+0000 7fefc8600640 1 -- start start 2024-03-11T19:58:16.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.691+0000 7fefc8600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fefc00ff9f0 0x7fefc019d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:16.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.691+0000 7fefc8600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefc0102360 0x7fefc019dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:16.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.691+0000 7fefc8600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefc0104d10 0x7fefc01a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:16.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.691+0000 7fefc8600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc01a5320 con 0x7fefc00ff9f0 2024-03-11T19:58:16.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.691+0000 7fefc8600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc01a5490 con 0x7fefc0102360 2024-03-11T19:58:16.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.691+0000 7fefc8600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc01a5650 con 0x7fefc0104d10 2024-03-11T19:58:16.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.691+0000 7fefc5c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fefc00ff9f0 0x7fefc019d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:16.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.691+0000 7fefc5c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fefc00ff9f0 0x7fefc019d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38610/0 (socket says 172.21.15.97:38610) 2024-03-11T19:58:16.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.691+0000 7fefc5c00640 1 -- 172.21.15.97:0/153541514 learned_addr learned my addr 172.21.15.97:0/153541514 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:16.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.691+0000 7fefc5200640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefc0102360 0x7fefc019dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:16.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.692+0000 7fefc6600640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefc0104d10 0x7fefc01a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:16.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.692+0000 7fefc5200640 1 -- 172.21.15.97:0/153541514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefc0104d10 msgr2=0x7fefc01a4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:16.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.692+0000 7fefc5200640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefc0104d10 0x7fefc01a4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.692+0000 7fefc5200640 1 -- 172.21.15.97:0/153541514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fefc00ff9f0 msgr2=0x7fefc019d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:16.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.692+0000 7fefc5200640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fefc00ff9f0 0x7fefc019d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.692+0000 7fefc5200640 1 -- 172.21.15.97:0/153541514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac009560 con 0x7fefc0102360 2024-03-11T19:58:16.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.692+0000 7fefc5c00640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fefc00ff9f0 0x7fefc019d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:16.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.692+0000 7fefc5200640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefc0102360 0x7fefc019dd70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fefac000c00 tx=0x7fefac005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:16.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.693+0000 7fefbe000640 1 -- 172.21.15.97:0/153541514 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac0043d0 con 0x7fefc0102360 2024-03-11T19:58:16.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.693+0000 7fefc8600640 1 -- 172.21.15.97:0/153541514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc01a58d0 con 0x7fefc0102360 2024-03-11T19:58:16.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.693+0000 7fefbe000640 1 -- 172.21.15.97:0/153541514 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fefac0052d0 con 0x7fefc0102360 2024-03-11T19:58:16.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.693+0000 7fefbe000640 1 -- 172.21.15.97:0/153541514 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac003d40 con 0x7fefc0102360 2024-03-11T19:58:16.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.693+0000 7fefc8600640 1 -- 172.21.15.97:0/153541514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc01a5e00 con 0x7fefc0102360 2024-03-11T19:58:16.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.694+0000 7fefc8600640 1 -- 172.21.15.97:0/153541514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc00fd100 con 0x7fefc0102360 2024-03-11T19:58:16.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.696+0000 7fefbe000640 1 -- 172.21.15.97:0/153541514 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fefac005440 con 0x7fefc0102360 2024-03-11T19:58:16.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.697+0000 7fefbe000640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fefa80a1ab0 0x7fefa80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:16.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.698+0000 7fefbe000640 1 -- 172.21.15.97:0/153541514 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fefac0ef840 con 0x7fefc0102360 2024-03-11T19:58:16.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.698+0000 7fefc5c00640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fefa80a1ab0 0x7fefa80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:16.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.698+0000 7fefc5c00640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fefa80a1ab0 0x7fefa80a3f70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fefb4011550 tx=0x7fefb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:16.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.702+0000 7fefbe000640 1 -- 172.21.15.97:0/153541514 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fefac0b8630 con 0x7fefc0102360 2024-03-11T19:58:16.803 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:16 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3643600117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:16.803 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:16 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2756290841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:58:16.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.903+0000 7fefc8600640 1 -- 172.21.15.97:0/153541514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fefc002ce40 con 0x7fefc0102360 2024-03-11T19:58:16.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.903+0000 7fefbe000640 1 -- 172.21.15.97:0/153541514 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fefac0bbcf0 con 0x7fefc0102360 2024-03-11T19:58:16.902 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T19:58:16.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.906+0000 7fefc8600640 1 -- 172.21.15.97:0/153541514 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fefa80a1ab0 msgr2=0x7fefa80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:16.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.906+0000 7fefc8600640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fefa80a1ab0 0x7fefa80a3f70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fefb4011550 tx=0x7fefb4005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:16.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.906+0000 7fefc8600640 1 -- 172.21.15.97:0/153541514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefc0102360 msgr2=0x7fefc019dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:16.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.906+0000 7fefc8600640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefc0102360 0x7fefc019dd70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fefac000c00 tx=0x7fefac005e50 comp rx=0 tx=0).stop 2024-03-11T19:58:16.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.906+0000 7fefc8600640 1 -- 172.21.15.97:0/153541514 shutdown_connections 2024-03-11T19:58:16.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.906+0000 7fefc8600640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fefc0104d10 0x7fefc01a4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.906+0000 7fefc8600640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fefc0102360 0x7fefc019dd70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.906+0000 7fefc8600640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fefa80a1ab0 0x7fefa80a3f70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.906+0000 7fefc8600640 1 --2- 172.21.15.97:0/153541514 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fefc00ff9f0 0x7fefc019d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:16.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.906+0000 7fefc8600640 1 -- 172.21.15.97:0/153541514 >> 172.21.15.97:0/153541514 conn(0x7fefc00fb610 msgr2=0x7fefc01002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:16.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.906+0000 7fefc8600640 1 -- 172.21.15.97:0/153541514 shutdown_connections 2024-03-11T19:58:16.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:16.906+0000 7fefc8600640 1 -- 172.21.15.97:0/153541514 wait complete. 2024-03-11T19:58:16.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:58:16.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:16 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3643600117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:16.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:16 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2756290841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:58:17.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:16 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3643600117' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:17.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:16 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2756290841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:58:17.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.121+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3039637309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9ac104490 msgr2=0x7fe9ac104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:17.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.121+0000 7fe9b0c00640 1 --2- 172.21.15.97:0/3039637309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9ac104490 0x7fe9ac104910 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009860 tx=0x7fe99c02bb70 comp rx=0 tx=0).stop 2024-03-11T19:58:17.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.121+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3039637309 shutdown_connections 2024-03-11T19:58:17.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.121+0000 7fe9b0c00640 1 --2- 172.21.15.97:0/3039637309 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9ac104e50 0x7fe9ac10b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.121+0000 7fe9b0c00640 1 --2- 172.21.15.97:0/3039637309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9ac104490 0x7fe9ac104910 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.121+0000 7fe9b0c00640 1 --2- 172.21.15.97:0/3039637309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9ac1032a0 0x7fe9ac1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.121+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3039637309 >> 172.21.15.97:0/3039637309 conn(0x7fe9ac0fe830 msgr2=0x7fe9ac100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:17.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.121+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3039637309 shutdown_connections 2024-03-11T19:58:17.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.121+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3039637309 wait complete. 2024-03-11T19:58:17.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.122+0000 7fe9b0c00640 1 Processor -- start 2024-03-11T19:58:17.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.122+0000 7fe9b0c00640 1 -- start start 2024-03-11T19:58:17.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.122+0000 7fe9b0c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9ac1032a0 0x7fe9ac199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:17.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.122+0000 7fe9b0c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9ac104490 0x7fe9ac199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:17.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.122+0000 7fe9b0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9ac104e50 0x7fe9ac1a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:17.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.122+0000 7fe9b0c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac1a0e00 con 0x7fe9ac104e50 2024-03-11T19:58:17.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.122+0000 7fe9b0c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac1a0f70 con 0x7fe9ac104490 2024-03-11T19:58:17.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.122+0000 7fe9b0c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac1a1130 con 0x7fe9ac1032a0 2024-03-11T19:58:17.122 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.123+0000 7fe9abe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9ac104e50 0x7fe9ac1a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.123+0000 7fe9ab400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9ac1032a0 0x7fe9ac199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.123+0000 7fe9aaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9ac104490 0x7fe9ac199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.123+0000 7fe9abe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9ac104e50 0x7fe9ac1a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38616/0 (socket says 172.21.15.97:38616) 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.123+0000 7fe9abe00640 1 -- 172.21.15.97:0/3271085323 learned_addr learned my addr 172.21.15.97:0/3271085323 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.123+0000 7fe9aaa00640 1 -- 172.21.15.97:0/3271085323 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9ac1032a0 msgr2=0x7fe9ac199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.123+0000 7fe9aaa00640 1 --2- 172.21.15.97:0/3271085323 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9ac1032a0 0x7fe9ac199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.123+0000 7fe9aaa00640 1 -- 172.21.15.97:0/3271085323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9ac104e50 msgr2=0x7fe9ac1a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.123+0000 7fe9aaa00640 1 --2- 172.21.15.97:0/3271085323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9ac104e50 0x7fe9ac1a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.123+0000 7fe9aaa00640 1 -- 172.21.15.97:0/3271085323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe994009490 con 0x7fe9ac104490 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.123+0000 7fe9aaa00640 1 --2- 172.21.15.97:0/3271085323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9ac104490 0x7fe9ac199850 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe99c02c0b0 tx=0x7fe99c0045a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.124+0000 7fe9a3e00640 1 -- 172.21.15.97:0/3271085323 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c02c510 con 0x7fe9ac104490 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.124+0000 7fe9a3e00640 1 -- 172.21.15.97:0/3271085323 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe99c02cb50 con 0x7fe9ac104490 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.124+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3271085323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a4009900 con 0x7fe9ac104490 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.124+0000 7fe9a3e00640 1 -- 172.21.15.97:0/3271085323 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c032600 con 0x7fe9ac104490 2024-03-11T19:58:17.123 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.124+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3271085323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe99c009490 con 0x7fe9ac104490 2024-03-11T19:58:17.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.125+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3271085323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ac10a690 con 0x7fe9ac104490 2024-03-11T19:58:17.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.127+0000 7fe9a3e00640 1 -- 172.21.15.97:0/3271085323 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe99c03b040 con 0x7fe9ac104490 2024-03-11T19:58:17.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.128+0000 7fe9a3e00640 1 --2- 172.21.15.97:0/3271085323 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9900a1a30 0x7fe9900a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:17.126 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.128+0000 7fe9a3e00640 1 -- 172.21.15.97:0/3271085323 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fe99c0ef740 con 0x7fe9ac104490 2024-03-11T19:58:17.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.128+0000 7fe9ab400640 1 --2- 172.21.15.97:0/3271085323 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9900a1a30 0x7fe9900a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:17.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.129+0000 7fe9ab400640 1 --2- 172.21.15.97:0/3271085323 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9900a1a30 0x7fe9900a3ef0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac1042f0 tx=0x7fe994005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:17.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.130+0000 7fe9a3e00640 1 -- 172.21.15.97:0/3271085323 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe99c0b8640 con 0x7fe9ac104490 2024-03-11T19:58:17.321 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.323+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3271085323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe9ac10ef00 con 0x7fe9ac104490 2024-03-11T19:58:17.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.324+0000 7fe9a3e00640 1 -- 172.21.15.97:0/3271085323 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe99c041020 con 0x7fe9ac104490 2024-03-11T19:58:17.322 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-11T19:58:17.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.326+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3271085323 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9900a1a30 msgr2=0x7fe9900a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:17.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.326+0000 7fe9b0c00640 1 --2- 172.21.15.97:0/3271085323 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9900a1a30 0x7fe9900a3ef0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac1042f0 tx=0x7fe994005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:17.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.326+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3271085323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9ac104490 msgr2=0x7fe9ac199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:17.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.326+0000 7fe9b0c00640 1 --2- 172.21.15.97:0/3271085323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9ac104490 0x7fe9ac199850 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe99c02c0b0 tx=0x7fe99c0045a0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.326+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3271085323 shutdown_connections 2024-03-11T19:58:17.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.326+0000 7fe9b0c00640 1 --2- 172.21.15.97:0/3271085323 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe9900a1a30 0x7fe9900a3ef0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.326+0000 7fe9b0c00640 1 --2- 172.21.15.97:0/3271085323 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe9ac104e50 0x7fe9ac1a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.326+0000 7fe9b0c00640 1 --2- 172.21.15.97:0/3271085323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9ac104490 0x7fe9ac199850 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.326+0000 7fe9b0c00640 1 --2- 172.21.15.97:0/3271085323 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9ac1032a0 0x7fe9ac199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.326+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3271085323 >> 172.21.15.97:0/3271085323 conn(0x7fe9ac0fe830 msgr2=0x7fe9ac107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:17.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.327+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3271085323 shutdown_connections 2024-03-11T19:58:17.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.327+0000 7fe9b0c00640 1 -- 172.21.15.97:0/3271085323 wait complete. 2024-03-11T19:58:17.344 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T19:58:17.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.544+0000 7f7befe00640 1 -- 172.21.15.97:0/15001288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bf0073b60 msgr2=0x7f7bf0073fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:17.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.544+0000 7f7befe00640 1 --2- 172.21.15.97:0/15001288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bf0073b60 0x7f7bf0073fc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009900 tx=0x7f7bd802bcd0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.544+0000 7f7befe00640 1 -- 172.21.15.97:0/15001288 shutdown_connections 2024-03-11T19:58:17.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.544+0000 7f7befe00640 1 --2- 172.21.15.97:0/15001288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bf0108f30 0x7f7bf010b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.544+0000 7f7befe00640 1 --2- 172.21.15.97:0/15001288 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bf0073b60 0x7f7bf0073fc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.544+0000 7f7befe00640 1 --2- 172.21.15.97:0/15001288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bf0075140 0x7f7bf0073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.544+0000 7f7befe00640 1 -- 172.21.15.97:0/15001288 >> 172.21.15.97:0/15001288 conn(0x7f7bf00fc5f0 msgr2=0x7f7bf00fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:17.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.544+0000 7f7befe00640 1 -- 172.21.15.97:0/15001288 shutdown_connections 2024-03-11T19:58:17.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.544+0000 7f7befe00640 1 -- 172.21.15.97:0/15001288 wait complete. 2024-03-11T19:58:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.545+0000 7f7befe00640 1 Processor -- start 2024-03-11T19:58:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.545+0000 7f7befe00640 1 -- start start 2024-03-11T19:58:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.545+0000 7f7befe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bf0073b60 0x7f7bf01992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.545+0000 7f7befe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bf0075140 0x7f7bf0199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.545+0000 7f7befe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bf0108f30 0x7f7bf01a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7befe00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf01a0dd0 con 0x7f7bf0073b60 2024-03-11T19:58:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7befe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf01a0f40 con 0x7f7bf0108f30 2024-03-11T19:58:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7befe00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf01a1100 con 0x7f7bf0075140 2024-03-11T19:58:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7beea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bf0073b60 0x7f7bf01992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:17.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7beea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bf0073b60 0x7f7bf01992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38636/0 (socket says 172.21.15.97:38636) 2024-03-11T19:58:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7beea00640 1 -- 172.21.15.97:0/1018134830 learned_addr learned my addr 172.21.15.97:0/1018134830 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7bef400640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bf0108f30 0x7f7bf01a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7be6000640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bf0075140 0x7f7bf0199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7bef400640 1 -- 172.21.15.97:0/1018134830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bf0075140 msgr2=0x7f7bf0199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7bef400640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bf0075140 0x7f7bf0199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7bef400640 1 -- 172.21.15.97:0/1018134830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bf0073b60 msgr2=0x7f7bf01992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7bef400640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bf0073b60 0x7f7bf01992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7bef400640 1 -- 172.21.15.97:0/1018134830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd8009560 con 0x7f7bf0108f30 2024-03-11T19:58:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.546+0000 7f7beea00640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bf0073b60 0x7f7bf01992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.547+0000 7f7bef400640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bf0108f30 0x7f7bf01a0890 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009fd0 tx=0x7f7be800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:17.547 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.547+0000 7f7be7e00640 1 -- 172.21.15.97:0/1018134830 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be800ce20 con 0x7f7bf0108f30 2024-03-11T19:58:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.547+0000 7f7befe00640 1 -- 172.21.15.97:0/1018134830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf01a13e0 con 0x7f7bf0108f30 2024-03-11T19:58:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.547+0000 7f7be7e00640 1 -- 172.21.15.97:0/1018134830 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7be8005020 con 0x7f7bf0108f30 2024-03-11T19:58:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.547+0000 7f7be7e00640 1 -- 172.21.15.97:0/1018134830 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8010690 con 0x7f7bf0108f30 2024-03-11T19:58:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.547+0000 7f7befe00640 1 -- 172.21.15.97:0/1018134830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf01a18c0 con 0x7f7bf0108f30 2024-03-11T19:58:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.548+0000 7f7befe00640 1 -- 172.21.15.97:0/1018134830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf0107cf0 con 0x7f7bf0108f30 2024-03-11T19:58:17.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.550+0000 7f7be7e00640 1 -- 172.21.15.97:0/1018134830 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7be8005500 con 0x7f7bf0108f30 2024-03-11T19:58:17.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.551+0000 7f7be7e00640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7bd00a1a30 0x7f7bd00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:17.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.551+0000 7f7be7e00640 1 -- 172.21.15.97:0/1018134830 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f7be8014070 con 0x7f7bf0108f30 2024-03-11T19:58:17.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.551+0000 7f7beea00640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7bd00a1a30 0x7f7bd00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:17.550 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.552+0000 7f7beea00640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7bd00a1a30 0x7f7bd00a3ef0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7be0011550 tx=0x7f7be0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:17.551 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.553+0000 7f7be7e00640 1 -- 172.21.15.97:0/1018134830 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7be8096d00 con 0x7f7bf0108f30 2024-03-11T19:58:17.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.756+0000 7f7befe00640 1 -- 172.21.15.97:0/1018134830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7bf002cda0 con 0x7f7bf0108f30 2024-03-11T19:58:17.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.757+0000 7f7be7e00640 1 -- 172.21.15.97:0/1018134830 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7be8005190 con 0x7f7bf0108f30 2024-03-11T19:58:17.755 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T19:58:17.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.759+0000 7f7befe00640 1 -- 172.21.15.97:0/1018134830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7bd00a1a30 msgr2=0x7f7bd00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:17.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.759+0000 7f7befe00640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7bd00a1a30 0x7f7bd00a3ef0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7be0011550 tx=0x7f7be0005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:17.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.759+0000 7f7befe00640 1 -- 172.21.15.97:0/1018134830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bf0108f30 msgr2=0x7f7bf01a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:17.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.759+0000 7f7befe00640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bf0108f30 0x7f7bf01a0890 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009fd0 tx=0x7f7be800ef10 comp rx=0 tx=0).stop 2024-03-11T19:58:17.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.759+0000 7f7befe00640 1 -- 172.21.15.97:0/1018134830 shutdown_connections 2024-03-11T19:58:17.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.759+0000 7f7befe00640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bf0108f30 0x7f7bf01a0890 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.759+0000 7f7befe00640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bf0075140 0x7f7bf0199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.759+0000 7f7befe00640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7bd00a1a30 0x7f7bd00a3ef0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.759+0000 7f7befe00640 1 --2- 172.21.15.97:0/1018134830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bf0073b60 0x7f7bf01992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.759+0000 7f7befe00640 1 -- 172.21.15.97:0/1018134830 >> 172.21.15.97:0/1018134830 conn(0x7f7bf00fc5f0 msgr2=0x7f7bf0104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:17.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.759+0000 7f7befe00640 1 -- 172.21.15.97:0/1018134830 shutdown_connections 2024-03-11T19:58:17.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.759+0000 7f7befe00640 1 -- 172.21.15.97:0/1018134830 wait complete. 2024-03-11T19:58:17.772 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:17 smithi097 ceph-mon[31898]: pgmap v399: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:17.773 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:17 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/153541514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:58:17.773 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:17 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3271085323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:58:17.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:58:17.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.972+0000 7f887b200640 1 -- 172.21.15.97:0/3112894159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f887c100b40 msgr2=0x7f887c100fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:17.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.972+0000 7f887b200640 1 --2- 172.21.15.97:0/3112894159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f887c100b40 0x7f887c100fc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8874009900 tx=0x7f887402bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:17.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.972+0000 7f887b200640 1 -- 172.21.15.97:0/3112894159 shutdown_connections 2024-03-11T19:58:17.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.972+0000 7f887b200640 1 --2- 172.21.15.97:0/3112894159 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f887c100b40 0x7f887c100fc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.972+0000 7f887b200640 1 --2- 172.21.15.97:0/3112894159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c1001a0 0x7f887c100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.972+0000 7f887b200640 1 --2- 172.21.15.97:0/3112894159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f887c0ff840 0x7f887c0ffc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.972+0000 7f887b200640 1 -- 172.21.15.97:0/3112894159 >> 172.21.15.97:0/3112894159 conn(0x7f887c0fb460 msgr2=0x7f887c0fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:17.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.972+0000 7f887b200640 1 -- 172.21.15.97:0/3112894159 shutdown_connections 2024-03-11T19:58:17.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.972+0000 7f887b200640 1 -- 172.21.15.97:0/3112894159 wait complete. 2024-03-11T19:58:17.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.973+0000 7f887b200640 1 Processor -- start 2024-03-11T19:58:17.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.973+0000 7f887b200640 1 -- start start 2024-03-11T19:58:17.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.973+0000 7f887b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c0ff840 0x7f887c199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:17.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.973+0000 7f887b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f887c1001a0 0x7f887c199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:17.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.973+0000 7f887b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f887c100b40 0x7f887c1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:17.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.973+0000 7f887b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c1a0d00 con 0x7f887c1001a0 2024-03-11T19:58:17.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.973+0000 7f887b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c1a0e70 con 0x7f887c0ff840 2024-03-11T19:58:17.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.974+0000 7f887b200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c1a1030 con 0x7f887c100b40 2024-03-11T19:58:17.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.974+0000 7f8879e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c0ff840 0x7f887c199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:17.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.974+0000 7f8871400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f887c1001a0 0x7f887c199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:17.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.974+0000 7f8879e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c0ff840 0x7f887c199210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:34206/0 (socket says 172.21.15.97:34206) 2024-03-11T19:58:17.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.974+0000 7f8879e00640 1 -- 172.21.15.97:0/3609908710 learned_addr learned my addr 172.21.15.97:0/3609908710 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:17.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.974+0000 7f887a800640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f887c100b40 0x7f887c1a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:17.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.974+0000 7f8879e00640 1 -- 172.21.15.97:0/3609908710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f887c100b40 msgr2=0x7f887c1a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:17.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.974+0000 7f8879e00640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f887c100b40 0x7f887c1a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.974+0000 7f8879e00640 1 -- 172.21.15.97:0/3609908710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f887c1001a0 msgr2=0x7f887c199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:17.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.974+0000 7f8879e00640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f887c1001a0 0x7f887c199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:17.974 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.974+0000 7f8879e00640 1 -- 172.21.15.97:0/3609908710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8874009560 con 0x7f887c0ff840 2024-03-11T19:58:17.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.974+0000 7f887a800640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f887c100b40 0x7f887c1a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:17.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.975+0000 7f8879e00640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c0ff840 0x7f887c199210 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f886c00eb90 tx=0x7f886c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:17.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.975+0000 7f8873400640 1 -- 172.21.15.97:0/3609908710 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c0099b0 con 0x7f887c0ff840 2024-03-11T19:58:17.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.975+0000 7f887b200640 1 -- 172.21.15.97:0/3609908710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f887c1a1310 con 0x7f887c0ff840 2024-03-11T19:58:17.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.975+0000 7f8873400640 1 -- 172.21.15.97:0/3609908710 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f886c00ee90 con 0x7f887c0ff840 2024-03-11T19:58:17.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.975+0000 7f8873400640 1 -- 172.21.15.97:0/3609908710 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c0107a0 con 0x7f887c0ff840 2024-03-11T19:58:17.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.975+0000 7f887b200640 1 -- 172.21.15.97:0/3609908710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887c1a17d0 con 0x7f887c0ff840 2024-03-11T19:58:17.975 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.976+0000 7f887b200640 1 -- 172.21.15.97:0/3609908710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887c10a590 con 0x7f887c0ff840 2024-03-11T19:58:17.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.978+0000 7f8873400640 1 -- 172.21.15.97:0/3609908710 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f886c005000 con 0x7f887c0ff840 2024-03-11T19:58:17.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.979+0000 7f8873400640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f88600a1a30 0x7f88600a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:17.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.979+0000 7f8873400640 1 -- 172.21.15.97:0/3609908710 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f886c014070 con 0x7f887c0ff840 2024-03-11T19:58:17.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.979+0000 7f8871400640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f88600a1a30 0x7f88600a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:17.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.980+0000 7f8871400640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f88600a1a30 0x7f88600a3ef0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f887c19a720 tx=0x7f8864005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:17.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:17.981+0000 7f8873400640 1 -- 172.21.15.97:0/3609908710 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f886c0971d0 con 0x7f887c0ff840 2024-03-11T19:58:17.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:17 smithi114 ceph-mon[36376]: pgmap v399: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:17.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:17 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/153541514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:58:17.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:17 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3271085323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:58:18.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:17 smithi161 ceph-mon[36332]: pgmap v399: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:18.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:17 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/153541514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:58:18.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:17 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3271085323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:58:18.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.214+0000 7f887b200640 1 -- 172.21.15.97:0/3609908710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f887c10ec60 con 0x7f887c0ff840 2024-03-11T19:58:18.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.215+0000 7f8873400640 1 -- 172.21.15.97:0/3609908710 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11401 (secure 0 0 0) 0x7f886c09a890 con 0x7f887c0ff840 2024-03-11T19:58:18.214 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:18.214 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:55:45.149335+0000","modified":"2024-03-11T19:56:19.631638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24583,"mds_2":24547,"mds_3":24545,"mds_4":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24583":{"gid":24583,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.97:6841/2531110572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2531110572},{"type":"v1","addr":"172.21.15.97:6841","nonce":2531110572}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6839/1624532248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":1624532248},{"type":"v1","addr":"172.21.15.114:6839","nonce":1624532248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:58:18.214 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-03-11T19:58:18.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.218+0000 7f887b200640 1 -- 172.21.15.97:0/3609908710 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f88600a1a30 msgr2=0x7f88600a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:18.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.218+0000 7f887b200640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f88600a1a30 0x7f88600a3ef0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f887c19a720 tx=0x7f8864005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:18.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.218+0000 7f887b200640 1 -- 172.21.15.97:0/3609908710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c0ff840 msgr2=0x7f887c199210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:18.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.218+0000 7f887b200640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c0ff840 0x7f887c199210 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f886c00eb90 tx=0x7f886c00c6a0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.218+0000 7f887b200640 1 -- 172.21.15.97:0/3609908710 shutdown_connections 2024-03-11T19:58:18.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.218+0000 7f887b200640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f887c100b40 0x7f887c1a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.218+0000 7f887b200640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f88600a1a30 0x7f88600a3ef0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.218+0000 7f887b200640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f887c1001a0 0x7f887c199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.218+0000 7f887b200640 1 --2- 172.21.15.97:0/3609908710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f887c0ff840 0x7f887c199210 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.218+0000 7f887b200640 1 -- 172.21.15.97:0/3609908710 >> 172.21.15.97:0/3609908710 conn(0x7f887c0fb460 msgr2=0x7f887c0fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:18.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.218+0000 7f887b200640 1 -- 172.21.15.97:0/3609908710 shutdown_connections 2024-03-11T19:58:18.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.218+0000 7f887b200640 1 -- 172.21.15.97:0/3609908710 wait complete. 2024-03-11T19:58:18.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:58:18.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.426+0000 7f321f200640 1 -- 172.21.15.97:0/1097876001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3220101a40 msgr2=0x7f3220103e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:18.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.426+0000 7f321f200640 1 --2- 172.21.15.97:0/1097876001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3220101a40 0x7f3220103e30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3208009900 tx=0x7f320802bc40 comp rx=0 tx=0).stop 2024-03-11T19:58:18.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.426+0000 7f321f200640 1 -- 172.21.15.97:0/1097876001 shutdown_connections 2024-03-11T19:58:18.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.426+0000 7f321f200640 1 --2- 172.21.15.97:0/1097876001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3220106ca0 0x7f32201090f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.426+0000 7f321f200640 1 --2- 172.21.15.97:0/1097876001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3220104370 0x7f3220106760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.426+0000 7f321f200640 1 --2- 172.21.15.97:0/1097876001 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3220101a40 0x7f3220103e30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.426+0000 7f321f200640 1 -- 172.21.15.97:0/1097876001 >> 172.21.15.97:0/1097876001 conn(0x7f32200fb430 msgr2=0x7f32200fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:18.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.426+0000 7f321f200640 1 -- 172.21.15.97:0/1097876001 shutdown_connections 2024-03-11T19:58:18.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.426+0000 7f321f200640 1 -- 172.21.15.97:0/1097876001 wait complete. 2024-03-11T19:58:18.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.427+0000 7f321f200640 1 Processor -- start 2024-03-11T19:58:18.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.427+0000 7f321f200640 1 -- start start 2024-03-11T19:58:18.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.427+0000 7f321f200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3220101a40 0x7f32201013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:18.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.427+0000 7f321f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3220104370 0x7f32200ff9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:18.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.427+0000 7f321f200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3220106ca0 0x7f32200fff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:18.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.427+0000 7f321f200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3220100690 con 0x7f3220101a40 2024-03-11T19:58:18.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.427+0000 7f321f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3220100800 con 0x7f3220104370 2024-03-11T19:58:18.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.427+0000 7f321f200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32201a1130 con 0x7f3220106ca0 2024-03-11T19:58:18.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.428+0000 7f321de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3220101a40 0x7f32201013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:18.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.428+0000 7f321de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3220101a40 0x7f32201013b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38664/0 (socket says 172.21.15.97:38664) 2024-03-11T19:58:18.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.428+0000 7f321de00640 1 -- 172.21.15.97:0/379164803 learned_addr learned my addr 172.21.15.97:0/379164803 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:18.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.428+0000 7f321d400640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3220104370 0x7f32200ff9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:18.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.428+0000 7f321e800640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3220106ca0 0x7f32200fff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:18.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.428+0000 7f321de00640 1 -- 172.21.15.97:0/379164803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3220106ca0 msgr2=0x7f32200fff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:18.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.428+0000 7f321de00640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3220106ca0 0x7f32200fff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.428+0000 7f321de00640 1 -- 172.21.15.97:0/379164803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3220104370 msgr2=0x7f32200ff9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:18.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.428+0000 7f321de00640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3220104370 0x7f32200ff9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.428+0000 7f321de00640 1 -- 172.21.15.97:0/379164803 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3208009560 con 0x7f3220101a40 2024-03-11T19:58:18.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.428+0000 7f321e800640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3220106ca0 0x7f32200fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:18.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.428+0000 7f321de00640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3220101a40 0x7f32201013b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3208036040 tx=0x7f3208005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:18.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.429+0000 7f3216a00640 1 -- 172.21.15.97:0/379164803 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32080043d0 con 0x7f3220101a40 2024-03-11T19:58:18.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.429+0000 7f3216a00640 1 -- 172.21.15.97:0/379164803 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32080052d0 con 0x7f3220101a40 2024-03-11T19:58:18.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.429+0000 7f321f200640 1 -- 172.21.15.97:0/379164803 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32201a1380 con 0x7f3220101a40 2024-03-11T19:58:18.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.429+0000 7f3216a00640 1 -- 172.21.15.97:0/379164803 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3208003cc0 con 0x7f3220101a40 2024-03-11T19:58:18.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.429+0000 7f321f200640 1 -- 172.21.15.97:0/379164803 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32201a18e0 con 0x7f3220101a40 2024-03-11T19:58:18.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.430+0000 7f321f200640 1 -- 172.21.15.97:0/379164803 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32200fcf20 con 0x7f3220101a40 2024-03-11T19:58:18.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.432+0000 7f3216a00640 1 -- 172.21.15.97:0/379164803 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3208005440 con 0x7f3220101a40 2024-03-11T19:58:18.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.434+0000 7f3216a00640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f32040a1ab0 0x7f32040a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:18.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.434+0000 7f3216a00640 1 -- 172.21.15.97:0/379164803 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f32080efa90 con 0x7f3220101a40 2024-03-11T19:58:18.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.434+0000 7f321d400640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f32040a1ab0 0x7f32040a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:18.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.435+0000 7f321d400640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f32040a1ab0 0x7f32040a3f70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3220101120 tx=0x7f3210005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:18.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.438+0000 7f3216a00640 1 -- 172.21.15.97:0/379164803 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f32080b8880 con 0x7f3220101a40 2024-03-11T19:58:18.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.632+0000 7f321f200640 1 -- 172.21.15.97:0/379164803 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f322002cdd0 con 0x7f3220101a40 2024-03-11T19:58:18.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.636+0000 7f3216a00640 1 -- 172.21.15.97:0/379164803 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24838 (secure 0 0 0) 0x7f32080bbf40 con 0x7f3220101a40 2024-03-11T19:58:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:18.635 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":93,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:57:46.674203+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:58:18.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.639+0000 7f321f200640 1 -- 172.21.15.97:0/379164803 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f32040a1ab0 msgr2=0x7f32040a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:18.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.639+0000 7f321f200640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f32040a1ab0 0x7f32040a3f70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3220101120 tx=0x7f3210005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:18.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.639+0000 7f321f200640 1 -- 172.21.15.97:0/379164803 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3220101a40 msgr2=0x7f32201013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:18.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.639+0000 7f321f200640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3220101a40 0x7f32201013b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3208036040 tx=0x7f3208005e80 comp rx=0 tx=0).stop 2024-03-11T19:58:18.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.639+0000 7f321f200640 1 -- 172.21.15.97:0/379164803 shutdown_connections 2024-03-11T19:58:18.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.639+0000 7f321f200640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3220106ca0 0x7f32200fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.639+0000 7f321f200640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3220104370 0x7f32200ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.639+0000 7f321f200640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f32040a1ab0 0x7f32040a3f70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.639+0000 7f321f200640 1 --2- 172.21.15.97:0/379164803 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3220101a40 0x7f32201013b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.639+0000 7f321f200640 1 -- 172.21.15.97:0/379164803 >> 172.21.15.97:0/379164803 conn(0x7f32200fb430 msgr2=0x7f32201024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:18.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.639+0000 7f321f200640 1 -- 172.21.15.97:0/379164803 shutdown_connections 2024-03-11T19:58:18.638 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.639+0000 7f321f200640 1 -- 172.21.15.97:0/379164803 wait complete. 2024-03-11T19:58:18.651 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:18 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1018134830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:58:18.655 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:50:29.051985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:55:43.424076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:55:44.652160+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T19:55:48.598077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T19:58:18.655 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:58:18.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.851+0000 7ffaa9000640 1 -- 172.21.15.97:0/1478940046 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa4104490 msgr2=0x7ffaa4104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:18.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.851+0000 7ffaa9000640 1 --2- 172.21.15.97:0/1478940046 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa4104490 0x7ffaa4104910 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009900 tx=0x7ffa8c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:18.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.851+0000 7ffaa9000640 1 -- 172.21.15.97:0/1478940046 shutdown_connections 2024-03-11T19:58:18.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.851+0000 7ffaa9000640 1 --2- 172.21.15.97:0/1478940046 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaa4104e50 0x7ffaa410b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.851+0000 7ffaa9000640 1 --2- 172.21.15.97:0/1478940046 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa4104490 0x7ffaa4104910 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.851+0000 7ffaa9000640 1 --2- 172.21.15.97:0/1478940046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa41032a0 0x7ffaa41036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.851+0000 7ffaa9000640 1 -- 172.21.15.97:0/1478940046 >> 172.21.15.97:0/1478940046 conn(0x7ffaa40fe830 msgr2=0x7ffaa4100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:18.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.851+0000 7ffaa9000640 1 -- 172.21.15.97:0/1478940046 shutdown_connections 2024-03-11T19:58:18.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.852+0000 7ffaa9000640 1 -- 172.21.15.97:0/1478940046 wait complete. 2024-03-11T19:58:18.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.852+0000 7ffaa9000640 1 Processor -- start 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.853+0000 7ffaa9000640 1 -- start start 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.853+0000 7ffaa9000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa41032a0 0x7ffaa4199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.853+0000 7ffaa9000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaa4104490 0x7ffaa4199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.853+0000 7ffaa9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa4104e50 0x7ffaa41a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.853+0000 7ffaa9000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa41a0e30 con 0x7ffaa41032a0 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.853+0000 7ffaa9000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa41a0fa0 con 0x7ffaa4104e50 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.853+0000 7ffaa9000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa41a1160 con 0x7ffaa4104490 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.853+0000 7ffaa3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa41032a0 0x7ffaa4199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.853+0000 7ffa9be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaa4104490 0x7ffaa4199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.853+0000 7ffaa3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa4104e50 0x7ffaa41a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.853+0000 7ffaa3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa41032a0 0x7ffaa4199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38694/0 (socket says 172.21.15.97:38694) 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.853+0000 7ffaa3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa4104e50 0x7ffaa41a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:34250/0 (socket says 172.21.15.97:34250) 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.853+0000 7ffaa3400640 1 -- 172.21.15.97:0/2977470521 learned_addr learned my addr 172.21.15.97:0/2977470521 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.854+0000 7ffaa3e00640 1 -- 172.21.15.97:0/2977470521 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaa4104490 msgr2=0x7ffaa4199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:18.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.854+0000 7ffaa3e00640 1 --2- 172.21.15.97:0/2977470521 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaa4104490 0x7ffaa4199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.854+0000 7ffaa3e00640 1 -- 172.21.15.97:0/2977470521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa41032a0 msgr2=0x7ffaa4199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:18.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.854+0000 7ffaa3e00640 1 --2- 172.21.15.97:0/2977470521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa41032a0 0x7ffaa4199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:18.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.854+0000 7ffaa3e00640 1 -- 172.21.15.97:0/2977470521 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8c009560 con 0x7ffaa4104e50 2024-03-11T19:58:18.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.854+0000 7ffaa3e00640 1 --2- 172.21.15.97:0/2977470521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa4104e50 0x7ffaa41a08f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009fd0 tx=0x7ffa9c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:18.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.854+0000 7ffaa0c00640 1 -- 172.21.15.97:0/2977470521 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c00ce20 con 0x7ffaa4104e50 2024-03-11T19:58:18.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.854+0000 7ffaa0c00640 1 -- 172.21.15.97:0/2977470521 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa9c005020 con 0x7ffaa4104e50 2024-03-11T19:58:18.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.854+0000 7ffaa9000640 1 -- 172.21.15.97:0/2977470521 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa41a1440 con 0x7ffaa4104e50 2024-03-11T19:58:18.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.854+0000 7ffaa0c00640 1 -- 172.21.15.97:0/2977470521 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c010690 con 0x7ffaa4104e50 2024-03-11T19:58:18.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.854+0000 7ffaa9000640 1 -- 172.21.15.97:0/2977470521 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa41a1920 con 0x7ffaa4104e50 2024-03-11T19:58:18.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.855+0000 7ffaa9000640 1 -- 172.21.15.97:0/2977470521 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa410a560 con 0x7ffaa4104e50 2024-03-11T19:58:18.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.857+0000 7ffaa0c00640 1 -- 172.21.15.97:0/2977470521 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ffa9c005500 con 0x7ffaa4104e50 2024-03-11T19:58:18.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.858+0000 7ffaa0c00640 1 --2- 172.21.15.97:0/2977470521 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ffa840a1a30 0x7ffa840a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:18.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.859+0000 7ffaa0c00640 1 -- 172.21.15.97:0/2977470521 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7ffa9c014070 con 0x7ffaa4104e50 2024-03-11T19:58:18.857 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.859+0000 7ffaa3400640 1 --2- 172.21.15.97:0/2977470521 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ffa840a1a30 0x7ffa840a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:18.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.859+0000 7ffaa3400640 1 --2- 172.21.15.97:0/2977470521 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ffa840a1a30 0x7ffa840a3ef0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffa940058c0 tx=0x7ffa94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:18.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:18.863+0000 7ffaa0c00640 1 -- 172.21.15.97:0/2977470521 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffa9c096d00 con 0x7ffaa4104e50 2024-03-11T19:58:18.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:18 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3609908710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:18.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:18 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/379164803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:18.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:18 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1018134830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:58:18.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:18 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3609908710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:18.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:18 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/379164803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:19.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:18 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1018134830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:58:19.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:18 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3609908710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:19.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:18 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/379164803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:19.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.045+0000 7ffaa9000640 1 -- 172.21.15.97:0/2977470521 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffaa402cda0 con 0x7ffaa4104e50 2024-03-11T19:58:19.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.046+0000 7ffaa0c00640 1 -- 172.21.15.97:0/2977470521 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffa9c005190 con 0x7ffaa4104e50 2024-03-11T19:58:19.044 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-11T19:58:19.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.048+0000 7ffaa9000640 1 -- 172.21.15.97:0/2977470521 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ffa840a1a30 msgr2=0x7ffa840a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:19.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.048+0000 7ffaa9000640 1 --2- 172.21.15.97:0/2977470521 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ffa840a1a30 0x7ffa840a3ef0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffa940058c0 tx=0x7ffa94005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:19.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.048+0000 7ffaa9000640 1 -- 172.21.15.97:0/2977470521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa4104e50 msgr2=0x7ffaa41a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:19.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.048+0000 7ffaa9000640 1 --2- 172.21.15.97:0/2977470521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa4104e50 0x7ffaa41a08f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009fd0 tx=0x7ffa9c00ef10 comp rx=0 tx=0).stop 2024-03-11T19:58:19.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.048+0000 7ffaa9000640 1 -- 172.21.15.97:0/2977470521 shutdown_connections 2024-03-11T19:58:19.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.048+0000 7ffaa9000640 1 --2- 172.21.15.97:0/2977470521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaa4104e50 0x7ffaa41a08f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.048+0000 7ffaa9000640 1 --2- 172.21.15.97:0/2977470521 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaa4104490 0x7ffaa4199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.048+0000 7ffaa9000640 1 --2- 172.21.15.97:0/2977470521 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ffa840a1a30 0x7ffa840a3ef0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.048+0000 7ffaa9000640 1 --2- 172.21.15.97:0/2977470521 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ffaa41032a0 0x7ffaa4199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.048+0000 7ffaa9000640 1 -- 172.21.15.97:0/2977470521 >> 172.21.15.97:0/2977470521 conn(0x7ffaa40fe830 msgr2=0x7ffaa4107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:19.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.048+0000 7ffaa9000640 1 -- 172.21.15.97:0/2977470521 shutdown_connections 2024-03-11T19:58:19.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.048+0000 7ffaa9000640 1 -- 172.21.15.97:0/2977470521 wait complete. 2024-03-11T19:58:19.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:58:19.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.268+0000 7fb77a200640 1 -- 172.21.15.97:0/3363055271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb774108990 msgr2=0x7fb774108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:19.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.268+0000 7fb77a200640 1 --2- 172.21.15.97:0/3363055271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb774108990 0x7fb774108e10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009860 tx=0x7fb75c02bb70 comp rx=0 tx=0).stop 2024-03-11T19:58:19.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.269+0000 7fb77a200640 1 -- 172.21.15.97:0/3363055271 shutdown_connections 2024-03-11T19:58:19.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.269+0000 7fb77a200640 1 --2- 172.21.15.97:0/3363055271 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb774109350 0x7fb77410fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.269+0000 7fb77a200640 1 --2- 172.21.15.97:0/3363055271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb774108990 0x7fb774108e10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.269+0000 7fb77a200640 1 --2- 172.21.15.97:0/3363055271 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7741077a0 0x7fb774107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.269+0000 7fb77a200640 1 -- 172.21.15.97:0/3363055271 >> 172.21.15.97:0/3363055271 conn(0x7fb774076740 msgr2=0x7fb774078b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:19.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.269+0000 7fb77a200640 1 -- 172.21.15.97:0/3363055271 shutdown_connections 2024-03-11T19:58:19.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.269+0000 7fb77a200640 1 -- 172.21.15.97:0/3363055271 wait complete. 2024-03-11T19:58:19.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.270+0000 7fb77a200640 1 Processor -- start 2024-03-11T19:58:19.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.270+0000 7fb77a200640 1 -- start start 2024-03-11T19:58:19.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.270+0000 7fb77a200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7741077a0 0x7fb77419d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:19.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.270+0000 7fb77a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb774108990 0x7fb77419dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:19.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.270+0000 7fb77a200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb774109350 0x7fb7741a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:19.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb77a200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7741a5350 con 0x7fb774108990 2024-03-11T19:58:19.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb77a200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7741a54c0 con 0x7fb7741077a0 2024-03-11T19:58:19.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb77a200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7741a5680 con 0x7fb774109350 2024-03-11T19:58:19.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb76be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb774108990 0x7fb77419dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb773400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7741077a0 0x7fb77419d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb76be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb774108990 0x7fb77419dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38712/0 (socket says 172.21.15.97:38712) 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb773400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7741077a0 0x7fb77419d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:34274/0 (socket says 172.21.15.97:34274) 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb76be00640 1 -- 172.21.15.97:0/2870850241 learned_addr learned my addr 172.21.15.97:0/2870850241 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb773e00640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb774109350 0x7fb7741a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb76be00640 1 -- 172.21.15.97:0/2870850241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb774109350 msgr2=0x7fb7741a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb76be00640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb774109350 0x7fb7741a4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb76be00640 1 -- 172.21.15.97:0/2870850241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7741077a0 msgr2=0x7fb77419d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb76be00640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7741077a0 0x7fb77419d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb76be00640 1 -- 172.21.15.97:0/2870850241 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb764009560 con 0x7fb774108990 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb773400640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7741077a0 0x7fb77419d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.271+0000 7fb76be00640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb774108990 0x7fb77419dda0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb75c005940 tx=0x7fb75c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.272+0000 7fb770c00640 1 -- 172.21.15.97:0/2870850241 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c02c510 con 0x7fb774108990 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.272+0000 7fb770c00640 1 -- 172.21.15.97:0/2870850241 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb75c02cb50 con 0x7fb774108990 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.272+0000 7fb77a200640 1 -- 172.21.15.97:0/2870850241 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76c009840 con 0x7fb774108990 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.272+0000 7fb770c00640 1 -- 172.21.15.97:0/2870850241 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c032600 con 0x7fb774108990 2024-03-11T19:58:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.272+0000 7fb77a200640 1 -- 172.21.15.97:0/2870850241 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75c009490 con 0x7fb774108990 2024-03-11T19:58:19.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.273+0000 7fb77a200640 1 -- 172.21.15.97:0/2870850241 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77410eb40 con 0x7fb774108990 2024-03-11T19:58:19.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.275+0000 7fb770c00640 1 -- 172.21.15.97:0/2870850241 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb75c03b040 con 0x7fb774108990 2024-03-11T19:58:19.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.276+0000 7fb770c00640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb7580a1a30 0x7fb7580a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:19.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.276+0000 7fb770c00640 1 -- 172.21.15.97:0/2870850241 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fb75c0ef4d0 con 0x7fb774108990 2024-03-11T19:58:19.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.276+0000 7fb773400640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb7580a1a30 0x7fb7580a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:19.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.277+0000 7fb773400640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb7580a1a30 0x7fb7580a3ef0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb764009530 tx=0x7fb764009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:19.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.281+0000 7fb770c00640 1 -- 172.21.15.97:0/2870850241 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb75c0b82c0 con 0x7fb774108990 2024-03-11T19:58:19.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.463+0000 7fb77a200640 1 -- 172.21.15.97:0/2870850241 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb77402cda0 con 0x7fb774108990 2024-03-11T19:58:19.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.464+0000 7fb770c00640 1 -- 172.21.15.97:0/2870850241 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb75c003710 con 0x7fb774108990 2024-03-11T19:58:19.462 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T19:58:19.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.467+0000 7fb77a200640 1 -- 172.21.15.97:0/2870850241 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb7580a1a30 msgr2=0x7fb7580a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:19.465 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.467+0000 7fb77a200640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb7580a1a30 0x7fb7580a3ef0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb764009530 tx=0x7fb764009260 comp rx=0 tx=0).stop 2024-03-11T19:58:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.467+0000 7fb77a200640 1 -- 172.21.15.97:0/2870850241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb774108990 msgr2=0x7fb77419dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.467+0000 7fb77a200640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb774108990 0x7fb77419dda0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb75c005940 tx=0x7fb75c0045e0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.467+0000 7fb77a200640 1 -- 172.21.15.97:0/2870850241 shutdown_connections 2024-03-11T19:58:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.467+0000 7fb77a200640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb774109350 0x7fb7741a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.467+0000 7fb77a200640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb7580a1a30 0x7fb7580a3ef0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.467+0000 7fb77a200640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb774108990 0x7fb77419dda0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.467+0000 7fb77a200640 1 --2- 172.21.15.97:0/2870850241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7741077a0 0x7fb77419d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.467+0000 7fb77a200640 1 -- 172.21.15.97:0/2870850241 >> 172.21.15.97:0/2870850241 conn(0x7fb774076740 msgr2=0x7fb77410bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.467+0000 7fb77a200640 1 -- 172.21.15.97:0/2870850241 shutdown_connections 2024-03-11T19:58:19.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.467+0000 7fb77a200640 1 -- 172.21.15.97:0/2870850241 wait complete. 2024-03-11T19:58:19.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:58:19.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.694+0000 7fcc14200640 1 -- 172.21.15.97:0/1085086996 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0c102380 msgr2=0x7fcc0c102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.694+0000 7fcc14200640 1 --2- 172.21.15.97:0/1085086996 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0c102380 0x7fcc0c102800 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009900 tx=0x7fcbf802bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.694+0000 7fcc14200640 1 -- 172.21.15.97:0/1085086996 shutdown_connections 2024-03-11T19:58:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.694+0000 7fcc14200640 1 --2- 172.21.15.97:0/1085086996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc0c102d40 0x7fcc0c1095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.694+0000 7fcc14200640 1 --2- 172.21.15.97:0/1085086996 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0c102380 0x7fcc0c102800 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.694+0000 7fcc14200640 1 --2- 172.21.15.97:0/1085086996 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc0c101190 0x7fcc0c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.694+0000 7fcc14200640 1 -- 172.21.15.97:0/1085086996 >> 172.21.15.97:0/1085086996 conn(0x7fcc0c0fc720 msgr2=0x7fcc0c0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:19.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.694+0000 7fcc14200640 1 -- 172.21.15.97:0/1085086996 shutdown_connections 2024-03-11T19:58:19.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.694+0000 7fcc14200640 1 -- 172.21.15.97:0/1085086996 wait complete. 2024-03-11T19:58:19.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.695+0000 7fcc14200640 1 Processor -- start 2024-03-11T19:58:19.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.695+0000 7fcc14200640 1 -- start start 2024-03-11T19:58:19.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.695+0000 7fcc14200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0c101190 0x7fcc0c074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:19.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.695+0000 7fcc14200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc0c102380 0x7fcc0c073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:19.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.695+0000 7fcc14200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc0c102d40 0x7fcc0c073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:19.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.695+0000 7fcc14200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c074210 con 0x7fcc0c101190 2024-03-11T19:58:19.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.695+0000 7fcc14200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c074380 con 0x7fcc0c102380 2024-03-11T19:58:19.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.695+0000 7fcc14200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c1a1220 con 0x7fcc0c102d40 2024-03-11T19:58:19.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.696+0000 7fcc11800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0c101190 0x7fcc0c074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:19.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.696+0000 7fcc11800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0c101190 0x7fcc0c074f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38730/0 (socket says 172.21.15.97:38730) 2024-03-11T19:58:19.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.696+0000 7fcc11800640 1 -- 172.21.15.97:0/1430583833 learned_addr learned my addr 172.21.15.97:0/1430583833 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:19.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.696+0000 7fcc12200640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc0c102d40 0x7fcc0c073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:19.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.696+0000 7fcc10e00640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc0c102380 0x7fcc0c073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:19.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.696+0000 7fcc11800640 1 -- 172.21.15.97:0/1430583833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc0c102d40 msgr2=0x7fcc0c073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:19.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.696+0000 7fcc11800640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc0c102d40 0x7fcc0c073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.696+0000 7fcc11800640 1 -- 172.21.15.97:0/1430583833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc0c102380 msgr2=0x7fcc0c073570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.696+0000 7fcc11800640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc0c102380 0x7fcc0c073570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.696+0000 7fcc11800640 1 -- 172.21.15.97:0/1430583833 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf8009560 con 0x7fcc0c101190 2024-03-11T19:58:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.696+0000 7fcc10e00640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc0c102380 0x7fcc0c073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.696+0000 7fcc11800640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0c101190 0x7fcc0c074f30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000d980 tx=0x7fcc0000de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.697+0000 7fcc0a000640 1 -- 172.21.15.97:0/1430583833 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00009c30 con 0x7fcc0c101190 2024-03-11T19:58:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.697+0000 7fcc14200640 1 -- 172.21.15.97:0/1430583833 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0c1a14d0 con 0x7fcc0c101190 2024-03-11T19:58:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.697+0000 7fcc0a000640 1 -- 172.21.15.97:0/1430583833 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc00009d90 con 0x7fcc0c101190 2024-03-11T19:58:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.697+0000 7fcc0a000640 1 -- 172.21.15.97:0/1430583833 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0000f7c0 con 0x7fcc0c101190 2024-03-11T19:58:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.697+0000 7fcc14200640 1 -- 172.21.15.97:0/1430583833 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0c1a19b0 con 0x7fcc0c101190 2024-03-11T19:58:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.698+0000 7fcc14200640 1 -- 172.21.15.97:0/1430583833 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0c1084b0 con 0x7fcc0c101190 2024-03-11T19:58:19.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.700+0000 7fcc0a000640 1 -- 172.21.15.97:0/1430583833 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcc00010460 con 0x7fcc0c101190 2024-03-11T19:58:19.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.701+0000 7fcc0a000640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcbf40a1a30 0x7fcbf40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:19.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.701+0000 7fcc0a000640 1 -- 172.21.15.97:0/1430583833 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fcc000cdae0 con 0x7fcc0c101190 2024-03-11T19:58:19.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.701+0000 7fcc10e00640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcbf40a1a30 0x7fcbf40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:19.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.702+0000 7fcc10e00640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcbf40a1a30 0x7fcbf40a3ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8006010 tx=0x7fcbf802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:19.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.705+0000 7fcc0a000640 1 -- 172.21.15.97:0/1430583833 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcc000968d0 con 0x7fcc0c101190 2024-03-11T19:58:19.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.904+0000 7fcc14200640 1 -- 172.21.15.97:0/1430583833 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcc0c10cbf0 con 0x7fcc0c101190 2024-03-11T19:58:19.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.905+0000 7fcc0a000640 1 -- 172.21.15.97:0/1430583833 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcc00099f90 con 0x7fcc0c101190 2024-03-11T19:58:19.904 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-11T19:58:19.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.908+0000 7fcc14200640 1 -- 172.21.15.97:0/1430583833 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcbf40a1a30 msgr2=0x7fcbf40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:19.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.908+0000 7fcc14200640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcbf40a1a30 0x7fcbf40a3ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8006010 tx=0x7fcbf802c7e0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.908+0000 7fcc14200640 1 -- 172.21.15.97:0/1430583833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0c101190 msgr2=0x7fcc0c074f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:19.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.908+0000 7fcc14200640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0c101190 0x7fcc0c074f30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000d980 tx=0x7fcc0000de40 comp rx=0 tx=0).stop 2024-03-11T19:58:19.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.908+0000 7fcc14200640 1 -- 172.21.15.97:0/1430583833 shutdown_connections 2024-03-11T19:58:19.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.908+0000 7fcc14200640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc0c102d40 0x7fcc0c073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.908+0000 7fcc14200640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc0c102380 0x7fcc0c073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.908+0000 7fcc14200640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcbf40a1a30 0x7fcbf40a3ef0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.908+0000 7fcc14200640 1 --2- 172.21.15.97:0/1430583833 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcc0c101190 0x7fcc0c074f30 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:19.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.908+0000 7fcc14200640 1 -- 172.21.15.97:0/1430583833 >> 172.21.15.97:0/1430583833 conn(0x7fcc0c0fc720 msgr2=0x7fcc0c105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:19.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.908+0000 7fcc14200640 1 -- 172.21.15.97:0/1430583833 shutdown_connections 2024-03-11T19:58:19.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:19.908+0000 7fcc14200640 1 -- 172.21.15.97:0/1430583833 wait complete. 2024-03-11T19:58:19.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:19 smithi097 ceph-mon[31898]: pgmap v400: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:19.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:19 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2977470521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:58:19.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:19 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2870850241' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:58:19.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T19:58:19.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:19 smithi114 ceph-mon[36376]: pgmap v400: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:19.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:19 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2977470521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:58:19.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:19 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2870850241' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:58:20.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:19 smithi161 ceph-mon[36332]: pgmap v400: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:20.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:19 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2977470521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:58:20.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:19 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2870850241' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:58:20.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.141+0000 7f129c800640 1 -- 172.21.15.97:0/554456586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12941045b0 msgr2=0x7f1294104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:20.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.141+0000 7f129c800640 1 --2- 172.21.15.97:0/554456586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12941045b0 0x7f1294104a30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1284009900 tx=0x7f128402bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:20.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.141+0000 7f129c800640 1 -- 172.21.15.97:0/554456586 shutdown_connections 2024-03-11T19:58:20.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.141+0000 7f129c800640 1 --2- 172.21.15.97:0/554456586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1294104f70 0x7f129410b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.141+0000 7f129c800640 1 --2- 172.21.15.97:0/554456586 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12941045b0 0x7f1294104a30 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.141+0000 7f129c800640 1 --2- 172.21.15.97:0/554456586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941033c0 0x7f12941037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.141+0000 7f129c800640 1 -- 172.21.15.97:0/554456586 >> 172.21.15.97:0/554456586 conn(0x7f12940fe930 msgr2=0x7f1294100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:20.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.141+0000 7f129c800640 1 -- 172.21.15.97:0/554456586 shutdown_connections 2024-03-11T19:58:20.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.141+0000 7f129c800640 1 -- 172.21.15.97:0/554456586 wait complete. 2024-03-11T19:58:20.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.143+0000 7f129c800640 1 Processor -- start 2024-03-11T19:58:20.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.143+0000 7f129c800640 1 -- start start 2024-03-11T19:58:20.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.143+0000 7f129c800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12941033c0 0x7f1294199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:20.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.143+0000 7f129c800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941045b0 0x7f12941999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:20.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.143+0000 7f129c800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1294104f70 0x7f12941a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:20.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.143+0000 7f129c800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12941a0f60 con 0x7f1294104f70 2024-03-11T19:58:20.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.143+0000 7f129c800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12941a10d0 con 0x7f12941045b0 2024-03-11T19:58:20.143 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.143+0000 7f129c800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12941a1290 con 0x7f12941033c0 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.143+0000 7f1299400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941045b0 0x7f12941999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.144+0000 7f1299400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941045b0 0x7f12941999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:34328/0 (socket says 172.21.15.97:34328) 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.144+0000 7f1299400640 1 -- 172.21.15.97:0/3857906068 learned_addr learned my addr 172.21.15.97:0/3857906068 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.144+0000 7f1299e00640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12941033c0 0x7f1294199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.144+0000 7f129a800640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1294104f70 0x7f12941a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.144+0000 7f1299400640 1 -- 172.21.15.97:0/3857906068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12941033c0 msgr2=0x7f1294199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.144+0000 7f1299400640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12941033c0 0x7f1294199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.144+0000 7f1299400640 1 -- 172.21.15.97:0/3857906068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1294104f70 msgr2=0x7f12941a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.144+0000 7f1299400640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1294104f70 0x7f12941a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.144+0000 7f1299400640 1 -- 172.21.15.97:0/3857906068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1288009490 con 0x7f12941045b0 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.144+0000 7f1299400640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941045b0 0x7f12941999b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f128402c6a0 tx=0x7f12840040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.145+0000 7f128ea00640 1 -- 172.21.15.97:0/3857906068 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12840044c0 con 0x7f12941045b0 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.145+0000 7f129c800640 1 -- 172.21.15.97:0/3857906068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1284009560 con 0x7f12941045b0 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.145+0000 7f128ea00640 1 -- 172.21.15.97:0/3857906068 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12840052d0 con 0x7f12941045b0 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.145+0000 7f128ea00640 1 -- 172.21.15.97:0/3857906068 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284003e80 con 0x7f12941045b0 2024-03-11T19:58:20.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.145+0000 7f129c800640 1 -- 172.21.15.97:0/3857906068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12941a18e0 con 0x7f12941045b0 2024-03-11T19:58:20.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.146+0000 7f129c800640 1 -- 172.21.15.97:0/3857906068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129410a680 con 0x7f12941045b0 2024-03-11T19:58:20.146 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.148+0000 7f128ea00640 1 -- 172.21.15.97:0/3857906068 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1284005440 con 0x7f12941045b0 2024-03-11T19:58:20.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.149+0000 7f128ea00640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f127c0a1ab0 0x7f127c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:20.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.149+0000 7f128ea00640 1 -- 172.21.15.97:0/3857906068 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f12840f0200 con 0x7f12941045b0 2024-03-11T19:58:20.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.149+0000 7f1299e00640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f127c0a1ab0 0x7f127c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:20.148 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.150+0000 7f1299e00640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f127c0a1ab0 0x7f127c0a3f70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1294104410 tx=0x7f1288005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:20.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.153+0000 7f128ea00640 1 -- 172.21.15.97:0/3857906068 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f12840b9110 con 0x7f12941045b0 2024-03-11T19:58:20.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.351+0000 7f129c800640 1 -- 172.21.15.97:0/3857906068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f129402ce40 con 0x7f12941045b0 2024-03-11T19:58:20.350 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.352+0000 7f128ea00640 1 -- 172.21.15.97:0/3857906068 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1284043020 con 0x7f12941045b0 2024-03-11T19:58:20.350 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T19:58:20.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.354+0000 7f129c800640 1 -- 172.21.15.97:0/3857906068 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f127c0a1ab0 msgr2=0x7f127c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:20.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.354+0000 7f129c800640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f127c0a1ab0 0x7f127c0a3f70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1294104410 tx=0x7f1288005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:20.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.355+0000 7f129c800640 1 -- 172.21.15.97:0/3857906068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941045b0 msgr2=0x7f12941999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:20.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.355+0000 7f129c800640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941045b0 0x7f12941999b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f128402c6a0 tx=0x7f12840040c0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.355+0000 7f129c800640 1 -- 172.21.15.97:0/3857906068 shutdown_connections 2024-03-11T19:58:20.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.355+0000 7f129c800640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f127c0a1ab0 0x7f127c0a3f70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.353 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.355+0000 7f129c800640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f1294104f70 0x7f12941a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.355+0000 7f129c800640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f12941045b0 0x7f12941999b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.355+0000 7f129c800640 1 --2- 172.21.15.97:0/3857906068 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12941033c0 0x7f1294199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.355+0000 7f129c800640 1 -- 172.21.15.97:0/3857906068 >> 172.21.15.97:0/3857906068 conn(0x7f12940fe930 msgr2=0x7f1294107850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:20.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.355+0000 7f129c800640 1 -- 172.21.15.97:0/3857906068 shutdown_connections 2024-03-11T19:58:20.354 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.355+0000 7f129c800640 1 -- 172.21.15.97:0/3857906068 wait complete. 2024-03-11T19:58:20.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:58:20.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.588+0000 7f2177000640 1 -- 172.21.15.97:0/2035308590 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2170104550 msgr2=0x7f2170106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:20.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.588+0000 7f2177000640 1 --2- 172.21.15.97:0/2035308590 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2170104550 0x7f2170106940 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2158009900 tx=0x7f215802bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:20.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.588+0000 7f2177000640 1 -- 172.21.15.97:0/2035308590 shutdown_connections 2024-03-11T19:58:20.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.588+0000 7f2177000640 1 --2- 172.21.15.97:0/2035308590 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2170106e80 0x7f21701092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.588+0000 7f2177000640 1 --2- 172.21.15.97:0/2035308590 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2170104550 0x7f2170106940 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.588+0000 7f2177000640 1 --2- 172.21.15.97:0/2035308590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170101c20 0x7f2170104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.588+0000 7f2177000640 1 -- 172.21.15.97:0/2035308590 >> 172.21.15.97:0/2035308590 conn(0x7f21700fb610 msgr2=0x7f21700fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:20.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.588+0000 7f2177000640 1 -- 172.21.15.97:0/2035308590 shutdown_connections 2024-03-11T19:58:20.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.588+0000 7f2177000640 1 -- 172.21.15.97:0/2035308590 wait complete. 2024-03-11T19:58:20.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.589+0000 7f2177000640 1 Processor -- start 2024-03-11T19:58:20.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.589+0000 7f2177000640 1 -- start start 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f2177000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2170101c20 0x7f21701972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f2177000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170104550 0x7f21701977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f2177000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2170106e80 0x7f217019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f2177000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217019ed90 con 0x7f2170101c20 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f2177000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217019ef00 con 0x7f2170106e80 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f2177000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217019f0c0 con 0x7f2170104550 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f216fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2170101c20 0x7f21701972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f216fe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2170101c20 0x7f21701972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38770/0 (socket says 172.21.15.97:38770) 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f216fe00640 1 -- 172.21.15.97:0/3535730502 learned_addr learned my addr 172.21.15.97:0/3535730502 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f2175000640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2170106e80 0x7f217019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f216f400640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170104550 0x7f21701977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f216fe00640 1 -- 172.21.15.97:0/3535730502 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170104550 msgr2=0x7f21701977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f216fe00640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170104550 0x7f21701977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f216fe00640 1 -- 172.21.15.97:0/3535730502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2170106e80 msgr2=0x7f217019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f216fe00640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2170106e80 0x7f217019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.590+0000 7f216fe00640 1 -- 172.21.15.97:0/3535730502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2168009840 con 0x7f2170101c20 2024-03-11T19:58:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.591+0000 7f216fe00640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2170101c20 0x7f21701972a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f216000e9f0 tx=0x7f216000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.591+0000 7f216cc00640 1 -- 172.21.15.97:0/3535730502 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2160009720 con 0x7f2170101c20 2024-03-11T19:58:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.591+0000 7f216cc00640 1 -- 172.21.15.97:0/3535730502 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2160005500 con 0x7f2170101c20 2024-03-11T19:58:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.591+0000 7f216cc00640 1 -- 172.21.15.97:0/3535730502 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2160010430 con 0x7f2170101c20 2024-03-11T19:58:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.591+0000 7f2177000640 1 -- 172.21.15.97:0/3535730502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2158009560 con 0x7f2170101c20 2024-03-11T19:58:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.591+0000 7f2177000640 1 -- 172.21.15.97:0/3535730502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f217019f6f0 con 0x7f2170101c20 2024-03-11T19:58:20.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.592+0000 7f2177000640 1 -- 172.21.15.97:0/3535730502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21700fd100 con 0x7f2170101c20 2024-03-11T19:58:20.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.594+0000 7f216cc00640 1 -- 172.21.15.97:0/3535730502 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2160005020 con 0x7f2170101c20 2024-03-11T19:58:20.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.595+0000 7f216cc00640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f21540a1ab0 0x7f21540a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:20.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.596+0000 7f216cc00640 1 -- 172.21.15.97:0/3535730502 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f2160014070 con 0x7f2170101c20 2024-03-11T19:58:20.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.596+0000 7f216f400640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f21540a1ab0 0x7f21540a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:20.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.596+0000 7f216f400640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f21540a1ab0 0x7f21540a3f70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2158005ab0 tx=0x7f2158005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:20.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.599+0000 7f216cc00640 1 -- 172.21.15.97:0/3535730502 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2160096950 con 0x7f2170101c20 2024-03-11T19:58:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.837+0000 7f2177000640 1 -- 172.21.15.97:0/3535730502 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f217010cbb0 con 0x7f2170101c20 2024-03-11T19:58:20.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.839+0000 7f216cc00640 1 -- 172.21.15.97:0/3535730502 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11401 (secure 0 0 0) 0x7f216009a010 con 0x7f2170101c20 2024-03-11T19:58:20.837 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:20.838 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:55:45.149335+0000","modified":"2024-03-11T19:56:19.631638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24583,"mds_2":24547,"mds_3":24545,"mds_4":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24583":{"gid":24583,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.97:6841/2531110572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2531110572},{"type":"v1","addr":"172.21.15.97:6841","nonce":2531110572}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6839/1624532248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":1624532248},{"type":"v1","addr":"172.21.15.114:6839","nonce":1624532248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:58:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-03-11T19:58:20.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.841+0000 7f2177000640 1 -- 172.21.15.97:0/3535730502 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f21540a1ab0 msgr2=0x7f21540a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:20.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.841+0000 7f2177000640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f21540a1ab0 0x7f21540a3f70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2158005ab0 tx=0x7f2158005fd0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.841+0000 7f2177000640 1 -- 172.21.15.97:0/3535730502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2170101c20 msgr2=0x7f21701972a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:20.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.841+0000 7f2177000640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2170101c20 0x7f21701972a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f216000e9f0 tx=0x7f216000edf0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.842+0000 7f2177000640 1 -- 172.21.15.97:0/3535730502 shutdown_connections 2024-03-11T19:58:20.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.842+0000 7f2177000640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2170106e80 0x7f217019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.842+0000 7f2177000640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2170104550 0x7f21701977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.842+0000 7f2177000640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f21540a1ab0 0x7f21540a3f70 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.842+0000 7f2177000640 1 --2- 172.21.15.97:0/3535730502 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2170101c20 0x7f21701972a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.842+0000 7f2177000640 1 -- 172.21.15.97:0/3535730502 >> 172.21.15.97:0/3535730502 conn(0x7f21700fb610 msgr2=0x7f2170102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.842+0000 7f2177000640 1 -- 172.21.15.97:0/3535730502 shutdown_connections 2024-03-11T19:58:20.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:20.842+0000 7f2177000640 1 -- 172.21.15.97:0/3535730502 wait complete. 2024-03-11T19:58:20.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:58:20.938 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:20 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1430583833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:58:20.938 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:20 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3857906068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:58:20.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:20 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1430583833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:58:20.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:20 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3857906068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:58:21.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:20 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1430583833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:58:21.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:20 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3857906068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:58:21.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.074+0000 7f88e8000640 1 -- 172.21.15.97:0/706164404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88e01045b0 msgr2=0x7f88e0104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:21.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.074+0000 7f88e8000640 1 --2- 172.21.15.97:0/706164404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88e01045b0 0x7f88e0104a30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009900 tx=0x7f88cc02bc50 comp rx=0 tx=0).stop 2024-03-11T19:58:21.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.074+0000 7f88e8000640 1 -- 172.21.15.97:0/706164404 shutdown_connections 2024-03-11T19:58:21.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.074+0000 7f88e8000640 1 --2- 172.21.15.97:0/706164404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88e0104f70 0x7f88e010b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.074+0000 7f88e8000640 1 --2- 172.21.15.97:0/706164404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88e01045b0 0x7f88e0104a30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.074+0000 7f88e8000640 1 --2- 172.21.15.97:0/706164404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88e01033c0 0x7f88e01037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.074+0000 7f88e8000640 1 -- 172.21.15.97:0/706164404 >> 172.21.15.97:0/706164404 conn(0x7f88e00fe970 msgr2=0x7f88e0100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:21.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.074+0000 7f88e8000640 1 -- 172.21.15.97:0/706164404 shutdown_connections 2024-03-11T19:58:21.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.074+0000 7f88e8000640 1 -- 172.21.15.97:0/706164404 wait complete. 2024-03-11T19:58:21.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.075+0000 7f88e8000640 1 Processor -- start 2024-03-11T19:58:21.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.075+0000 7f88e8000640 1 -- start start 2024-03-11T19:58:21.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e8000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88e01033c0 0x7f88e01994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e8000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88e01045b0 0x7f88e01999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e8000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88e0104f70 0x7f88e01a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e8000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e01a0fa0 con 0x7f88e0104f70 2024-03-11T19:58:21.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e8000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e01a1110 con 0x7f88e01033c0 2024-03-11T19:58:21.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e8000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e01a12d0 con 0x7f88e01045b0 2024-03-11T19:58:21.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88e0104f70 0x7f88e01a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88e0104f70 0x7f88e01a0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38782/0 (socket says 172.21.15.97:38782) 2024-03-11T19:58:21.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e6000640 1 -- 172.21.15.97:0/1541563311 learned_addr learned my addr 172.21.15.97:0/1541563311 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:21.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e5600640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88e01033c0 0x7f88e01994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e4c00640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88e01045b0 0x7f88e01999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e6000640 1 -- 172.21.15.97:0/1541563311 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88e01045b0 msgr2=0x7f88e01999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:21.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e6000640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88e01045b0 0x7f88e01999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e6000640 1 -- 172.21.15.97:0/1541563311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88e01033c0 msgr2=0x7f88e01994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:21.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e6000640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88e01033c0 0x7f88e01994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.076+0000 7f88e6000640 1 -- 172.21.15.97:0/1541563311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc009560 con 0x7f88e0104f70 2024-03-11T19:58:21.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.077+0000 7f88e6000640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88e0104f70 0x7f88e01a0a60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f88d800eaa0 tx=0x7f88d800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:21.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.077+0000 7f88de000640 1 -- 172.21.15.97:0/1541563311 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d800cdb0 con 0x7f88e0104f70 2024-03-11T19:58:21.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.077+0000 7f88de000640 1 -- 172.21.15.97:0/1541563311 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88d8005020 con 0x7f88e0104f70 2024-03-11T19:58:21.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.077+0000 7f88e8000640 1 -- 172.21.15.97:0/1541563311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e01a15b0 con 0x7f88e0104f70 2024-03-11T19:58:21.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.077+0000 7f88e8000640 1 -- 172.21.15.97:0/1541563311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e01a1a90 con 0x7f88e0104f70 2024-03-11T19:58:21.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.077+0000 7f88de000640 1 -- 172.21.15.97:0/1541563311 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d8011670 con 0x7f88e0104f70 2024-03-11T19:58:21.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.078+0000 7f88e8000640 1 -- 172.21.15.97:0/1541563311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e010a680 con 0x7f88e0104f70 2024-03-11T19:58:21.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.080+0000 7f88de000640 1 -- 172.21.15.97:0/1541563311 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f88d8005520 con 0x7f88e0104f70 2024-03-11T19:58:21.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.082+0000 7f88de000640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f88c80a1a30 0x7f88c80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.082+0000 7f88de000640 1 -- 172.21.15.97:0/1541563311 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f88d80cd930 con 0x7f88e0104f70 2024-03-11T19:58:21.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.082+0000 7f88e5600640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f88c80a1a30 0x7f88c80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.083+0000 7f88e5600640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f88c80a1a30 0x7f88c80a3ef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f88e0104410 tx=0x7f88d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:21.084 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.086+0000 7f88de000640 1 -- 172.21.15.97:0/1541563311 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f88d80967d0 con 0x7f88e0104f70 2024-03-11T19:58:21.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.283+0000 7f88e8000640 1 -- 172.21.15.97:0/1541563311 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88e010ee20 con 0x7f88e0104f70 2024-03-11T19:58:21.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.291+0000 7f88de000640 1 -- 172.21.15.97:0/1541563311 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24838 (secure 0 0 0) 0x7f88d8099e90 con 0x7f88e0104f70 2024-03-11T19:58:21.290 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:21.290 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":93,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:57:46.674203+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:58:21.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.294+0000 7f88e8000640 1 -- 172.21.15.97:0/1541563311 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f88c80a1a30 msgr2=0x7f88c80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:21.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.294+0000 7f88e8000640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f88c80a1a30 0x7f88c80a3ef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f88e0104410 tx=0x7f88d4005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:21.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.294+0000 7f88e8000640 1 -- 172.21.15.97:0/1541563311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88e0104f70 msgr2=0x7f88e01a0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:21.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.294+0000 7f88e8000640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88e0104f70 0x7f88e01a0a60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f88d800eaa0 tx=0x7f88d800ef60 comp rx=0 tx=0).stop 2024-03-11T19:58:21.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.294+0000 7f88e8000640 1 -- 172.21.15.97:0/1541563311 shutdown_connections 2024-03-11T19:58:21.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.294+0000 7f88e8000640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f88c80a1a30 0x7f88c80a3ef0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.294+0000 7f88e8000640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f88e0104f70 0x7f88e01a0a60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.294+0000 7f88e8000640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f88e01045b0 0x7f88e01999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.294+0000 7f88e8000640 1 --2- 172.21.15.97:0/1541563311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88e01033c0 0x7f88e01994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.294+0000 7f88e8000640 1 -- 172.21.15.97:0/1541563311 >> 172.21.15.97:0/1541563311 conn(0x7f88e00fe970 msgr2=0x7f88e0107850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:21.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.294+0000 7f88e8000640 1 -- 172.21.15.97:0/1541563311 shutdown_connections 2024-03-11T19:58:21.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.294+0000 7f88e8000640 1 -- 172.21.15.97:0/1541563311 wait complete. 2024-03-11T19:58:21.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:58:21.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.522+0000 7fe163e00640 1 -- 172.21.15.97:0/4000286179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe15c1045b0 msgr2=0x7fe15c104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.522+0000 7fe163e00640 1 --2- 172.21.15.97:0/4000286179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe15c1045b0 0x7fe15c104a30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe148009900 tx=0x7fe14802bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.522+0000 7fe163e00640 1 -- 172.21.15.97:0/4000286179 shutdown_connections 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.522+0000 7fe163e00640 1 --2- 172.21.15.97:0/4000286179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe15c104f70 0x7fe15c10b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.522+0000 7fe163e00640 1 --2- 172.21.15.97:0/4000286179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe15c1045b0 0x7fe15c104a30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.522+0000 7fe163e00640 1 --2- 172.21.15.97:0/4000286179 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe15c1033c0 0x7fe15c1037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.522+0000 7fe163e00640 1 -- 172.21.15.97:0/4000286179 >> 172.21.15.97:0/4000286179 conn(0x7fe15c0fe970 msgr2=0x7fe15c100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.522+0000 7fe163e00640 1 -- 172.21.15.97:0/4000286179 shutdown_connections 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.522+0000 7fe163e00640 1 -- 172.21.15.97:0/4000286179 wait complete. 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.523+0000 7fe163e00640 1 Processor -- start 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.523+0000 7fe163e00640 1 -- start start 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.523+0000 7fe163e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe15c1033c0 0x7fe15c199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.523+0000 7fe163e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe15c1045b0 0x7fe15c199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.523+0000 7fe163e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe15c104f70 0x7fe15c1a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe163e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c1a0f10 con 0x7fe15c104f70 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe163e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c1a1080 con 0x7fe15c1033c0 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe163e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c1a1240 con 0x7fe15c1045b0 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe161e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe15c104f70 0x7fe15c1a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe161e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe15c104f70 0x7fe15c1a09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38810/0 (socket says 172.21.15.97:38810) 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe161400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe15c1033c0 0x7fe15c199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe161e00640 1 -- 172.21.15.97:0/4008055119 learned_addr learned my addr 172.21.15.97:0/4008055119 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe160a00640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe15c1045b0 0x7fe15c199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe161e00640 1 -- 172.21.15.97:0/4008055119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe15c1045b0 msgr2=0x7fe15c199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe161e00640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe15c1045b0 0x7fe15c199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe161e00640 1 -- 172.21.15.97:0/4008055119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe15c1033c0 msgr2=0x7fe15c199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe161e00640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe15c1033c0 0x7fe15c199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe161e00640 1 -- 172.21.15.97:0/4008055119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe148009560 con 0x7fe15c104f70 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.524+0000 7fe161400640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe15c1033c0 0x7fe15c199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:21.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.525+0000 7fe161e00640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe15c104f70 0x7fe15c1a09d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe15400eb10 tx=0x7fe15400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:21.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.525+0000 7fe15a000640 1 -- 172.21.15.97:0/4008055119 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154004b20 con 0x7fe15c104f70 2024-03-11T19:58:21.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.525+0000 7fe163e00640 1 -- 172.21.15.97:0/4008055119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15c1a1520 con 0x7fe15c104f70 2024-03-11T19:58:21.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.525+0000 7fe15a000640 1 -- 172.21.15.97:0/4008055119 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe154004c80 con 0x7fe15c104f70 2024-03-11T19:58:21.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.525+0000 7fe163e00640 1 -- 172.21.15.97:0/4008055119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15c1a1a50 con 0x7fe15c104f70 2024-03-11T19:58:21.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.525+0000 7fe15a000640 1 -- 172.21.15.97:0/4008055119 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154018630 con 0x7fe15c104f70 2024-03-11T19:58:21.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.526+0000 7fe163e00640 1 -- 172.21.15.97:0/4008055119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15c10a6e0 con 0x7fe15c104f70 2024-03-11T19:58:21.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.528+0000 7fe15a000640 1 -- 172.21.15.97:0/4008055119 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe154018810 con 0x7fe15c104f70 2024-03-11T19:58:21.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.529+0000 7fe15a000640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe1440a1a30 0x7fe1440a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.530+0000 7fe15a000640 1 -- 172.21.15.97:0/4008055119 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fe154014070 con 0x7fe15c104f70 2024-03-11T19:58:21.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.530+0000 7fe161400640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe1440a1a30 0x7fe1440a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.530+0000 7fe161400640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe1440a1a30 0x7fe1440a3ef0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe15c104410 tx=0x7fe15000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:21.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.532+0000 7fe15a000640 1 -- 172.21.15.97:0/4008055119 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe154097270 con 0x7fe15c104f70 2024-03-11T19:58:21.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.765+0000 7fe163e00640 1 -- 172.21.15.97:0/4008055119 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe15c10edd0 con 0x7fe15c104f70 2024-03-11T19:58:21.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.767+0000 7fe15a000640 1 -- 172.21.15.97:0/4008055119 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11401 (secure 0 0 0) 0x7fe15409a930 con 0x7fe15c104f70 2024-03-11T19:58:21.766 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:21.766 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:55:45.149335+0000","modified":"2024-03-11T19:56:19.631638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24583,"mds_2":24547,"mds_3":24545,"mds_4":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24583":{"gid":24583,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.97:6841/2531110572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2531110572},{"type":"v1","addr":"172.21.15.97:6841","nonce":2531110572}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6839/1624532248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":1624532248},{"type":"v1","addr":"172.21.15.114:6839","nonce":1624532248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:58:21.766 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-03-11T19:58:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.772+0000 7fe163e00640 1 -- 172.21.15.97:0/4008055119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe1440a1a30 msgr2=0x7fe1440a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.772+0000 7fe163e00640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe1440a1a30 0x7fe1440a3ef0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe15c104410 tx=0x7fe15000b390 comp rx=0 tx=0).stop 2024-03-11T19:58:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.772+0000 7fe163e00640 1 -- 172.21.15.97:0/4008055119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe15c104f70 msgr2=0x7fe15c1a09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.772+0000 7fe163e00640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe15c104f70 0x7fe15c1a09d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe15400eb10 tx=0x7fe15400c2d0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.772+0000 7fe163e00640 1 -- 172.21.15.97:0/4008055119 shutdown_connections 2024-03-11T19:58:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.772+0000 7fe163e00640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fe1440a1a30 0x7fe1440a3ef0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.772+0000 7fe163e00640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe15c104f70 0x7fe15c1a09d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.772+0000 7fe163e00640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe15c1045b0 0x7fe15c199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.772+0000 7fe163e00640 1 --2- 172.21.15.97:0/4008055119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe15c1033c0 0x7fe15c199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.772+0000 7fe163e00640 1 -- 172.21.15.97:0/4008055119 >> 172.21.15.97:0/4008055119 conn(0x7fe15c0fe970 msgr2=0x7fe15c1078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.772+0000 7fe163e00640 1 -- 172.21.15.97:0/4008055119 shutdown_connections 2024-03-11T19:58:21.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.772+0000 7fe163e00640 1 -- 172.21.15.97:0/4008055119 wait complete. 2024-03-11T19:58:21.789 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-11T19:55:45.149335+0000', 'modified': '2024-03-11T19:56:19.631638+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24599, 'mds_1': 24583, 'mds_2': 24547, 'mds_3': 24545, 'mds_4': 24464}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24454': {'gid': 24454, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6833/2213740234', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6832', 'nonce': 2213740234}, {'type': 'v1', 'addr': '172.21.15.161:6833', 'nonce': 2213740234}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24464': {'gid': 24464, 'name': 'k', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.114:6833/2550912009', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6832', 'nonce': 2550912009}, {'type': 'v1', 'addr': '172.21.15.114:6833', 'nonce': 2550912009}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24499': {'gid': 24499, 'name': 'd', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.97:6837/267259401', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 267259401}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 267259401}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24512': {'gid': 24512, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.114:6835/2623690462', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 2623690462}, {'type': 'v1', 'addr': '172.21.15.114:6835', 'nonce': 2623690462}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24517': {'gid': 24517, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6835/3358622196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 3358622196}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 3358622196}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24529': {'gid': 24529, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6837/1600156948', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 1600156948}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 1600156948}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24545': {'gid': 24545, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.114:6837/1112816294', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6836', 'nonce': 1112816294}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1112816294}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.161:6839/565994162', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 565994162}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 565994162}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24583': {'gid': 24583, 'name': 'a', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.97:6841/2531110572', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6840', 'nonce': 2531110572}, {'type': 'v1', 'addr': '172.21.15.97:6841', 'nonce': 2531110572}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24599': {'gid': 24599, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6839/1624532248', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6838', 'nonce': 1624532248}, {'type': 'v1', 'addr': '172.21.15.114:6839', 'nonce': 1624532248}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-11T19:58:21.789 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-11T19:58:21.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-11T19:58:21.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:21 smithi114 ceph-mon[36376]: pgmap v401: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:21.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:21 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3535730502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:21.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:21 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1541563311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:21.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.991+0000 7f2251000640 1 -- 172.21.15.97:0/1755916543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c073be0 msgr2=0x7f224c074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:21.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.991+0000 7f2251000640 1 --2- 172.21.15.97:0/1755916543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c073be0 0x7f224c074060 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2238009900 tx=0x7f223802bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:21.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.991+0000 7f2251000640 1 -- 172.21.15.97:0/1755916543 shutdown_connections 2024-03-11T19:58:21.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.991+0000 7f2251000640 1 --2- 172.21.15.97:0/1755916543 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f224c108f40 0x7f224c10b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.991+0000 7f2251000640 1 --2- 172.21.15.97:0/1755916543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c073be0 0x7f224c074060 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.991+0000 7f2251000640 1 --2- 172.21.15.97:0/1755916543 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f224c0751c0 0x7f224c073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.991+0000 7f2251000640 1 -- 172.21.15.97:0/1755916543 >> 172.21.15.97:0/1755916543 conn(0x7f224c0fc5f0 msgr2=0x7f224c0fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:21.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.991+0000 7f2251000640 1 -- 172.21.15.97:0/1755916543 shutdown_connections 2024-03-11T19:58:21.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.991+0000 7f2251000640 1 -- 172.21.15.97:0/1755916543 wait complete. 2024-03-11T19:58:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.992+0000 7f2251000640 1 Processor -- start 2024-03-11T19:58:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.992+0000 7f2251000640 1 -- start start 2024-03-11T19:58:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.992+0000 7f2251000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f224c073be0 0x7f224c1992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.992+0000 7f2251000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c0751c0 0x7f224c199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.992+0000 7f2251000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f224c108f40 0x7f224c1a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.992+0000 7f2251000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c1a0de0 con 0x7f224c108f40 2024-03-11T19:58:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.992+0000 7f2251000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c1a0f50 con 0x7f224c0751c0 2024-03-11T19:58:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.992+0000 7f2251000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c1a1110 con 0x7f224c073be0 2024-03-11T19:58:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.992+0000 7f224aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c0751c0 0x7f224c199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.992+0000 7f224b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f224c073be0 0x7f224c1992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.992+0000 7f224be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f224c108f40 0x7f224c1a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.992+0000 7f224aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c0751c0 0x7f224c199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:34406/0 (socket says 172.21.15.97:34406) 2024-03-11T19:58:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.992+0000 7f224b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f224c073be0 0x7f224c1992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:39242/0 (socket says 172.21.15.97:39242) 2024-03-11T19:58:21.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.993+0000 7f224aa00640 1 -- 172.21.15.97:0/1476334856 learned_addr learned my addr 172.21.15.97:0/1476334856 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:21.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.993+0000 7f224aa00640 1 -- 172.21.15.97:0/1476334856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f224c073be0 msgr2=0x7f224c1992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:21.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.993+0000 7f224aa00640 1 --2- 172.21.15.97:0/1476334856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f224c073be0 0x7f224c1992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.993+0000 7f224aa00640 1 -- 172.21.15.97:0/1476334856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f224c108f40 msgr2=0x7f224c1a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:21.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.993+0000 7f224aa00640 1 --2- 172.21.15.97:0/1476334856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f224c108f40 0x7f224c1a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.993+0000 7f224aa00640 1 -- 172.21.15.97:0/1476334856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2238009560 con 0x7f224c0751c0 2024-03-11T19:58:21.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.993+0000 7f224aa00640 1 --2- 172.21.15.97:0/1476334856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c0751c0 0x7f224c199830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2238006010 tx=0x7f22380040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:21.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.993+0000 7f2243e00640 1 -- 172.21.15.97:0/1476334856 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22380044c0 con 0x7f224c0751c0 2024-03-11T19:58:21.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.993+0000 7f2243e00640 1 -- 172.21.15.97:0/1476334856 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22380052d0 con 0x7f224c0751c0 2024-03-11T19:58:21.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.993+0000 7f2251000640 1 -- 172.21.15.97:0/1476334856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224c1a1390 con 0x7f224c0751c0 2024-03-11T19:58:21.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.994+0000 7f2243e00640 1 -- 172.21.15.97:0/1476334856 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2238003e10 con 0x7f224c0751c0 2024-03-11T19:58:21.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.994+0000 7f2251000640 1 -- 172.21.15.97:0/1476334856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224c1a18a0 con 0x7f224c0751c0 2024-03-11T19:58:21.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f2255255740 1 Processor -- start 2024-03-11T19:58:21.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f2255255740 1 -- start start 2024-03-11T19:58:21.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f2255255740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406e760 0x5557a406eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f2255255740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a406f150 0x5557a4073c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f2243e00640 1 -- 172.21.15.97:0/1476334856 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2238005440 con 0x7f224c0751c0 2024-03-11T19:58:21.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f2255255740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a40742c0 0x5557a4076750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f2255255740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557a3fddb30 con 0x5557a406f150 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f2255255740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557a3fdb060 con 0x5557a40742c0 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f2255255740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557a3fd6840 con 0x5557a406e760 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f224b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a406f150 0x5557a4073c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f224b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a406f150 0x5557a4073c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38848/0 (socket says 172.21.15.97:38848) 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f224b400640 1 -- 172.21.15.97:0/1579848999 learned_addr learned my addr 172.21.15.97:0/1579848999 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f224be00640 1 --2- 172.21.15.97:0/1579848999 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406e760 0x5557a406eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.997+0000 7f224aa00640 1 --2- 172.21.15.97:0/1579848999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a40742c0 0x5557a4076750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.998+0000 7f224b400640 1 -- 172.21.15.97:0/1579848999 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406e760 msgr2=0x5557a406eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.998+0000 7f224b400640 1 --2- 172.21.15.97:0/1579848999 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406e760 0x5557a406eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.998+0000 7f224b400640 1 -- 172.21.15.97:0/1579848999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a40742c0 msgr2=0x5557a4076750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.998+0000 7f224b400640 1 --2- 172.21.15.97:0/1579848999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a40742c0 0x5557a4076750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.998+0000 7f224b400640 1 -- 172.21.15.97:0/1579848999 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557a3f9e4c0 con 0x5557a406f150 2024-03-11T19:58:21.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.998+0000 7f224aa00640 1 --2- 172.21.15.97:0/1579848999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a40742c0 0x5557a4076750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:21.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.998+0000 7f224be00640 1 --2- 172.21.15.97:0/1579848999 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406e760 0x5557a406eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:21.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.998+0000 7f224b400640 1 --2- 172.21.15.97:0/1579848999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a406f150 0x5557a4073c50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f223c00bd50 tx=0x7f223c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44981ee1abc79ea4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:21.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.998+0000 7f2240c00640 1 -- 172.21.15.97:0/1579848999 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c005220 con 0x5557a406f150 2024-03-11T19:58:21.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.998+0000 7f2240c00640 1 -- 172.21.15.97:0/1579848999 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f223c015070 con 0x5557a406f150 2024-03-11T19:58:21.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.998+0000 7f2240c00640 1 -- 172.21.15.97:0/1579848999 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c0116a0 con 0x5557a406f150 2024-03-11T19:58:22.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.999+0000 7f2243e00640 1 --2- 172.21.15.97:0/1476334856 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22300a1a30 0x7f22300a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.999+0000 7f2243e00640 1 -- 172.21.15.97:0/1476334856 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f22380efb50 con 0x7f224c0751c0 2024-03-11T19:58:22.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.999+0000 7f224be00640 1 --2- 172.21.15.97:0/1476334856 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22300a1a30 0x7f22300a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.999+0000 7f2255255740 1 -- 172.21.15.97:0/1579848999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a406f150 msgr2=0x5557a4073c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.999+0000 7f2255255740 1 --2- 172.21.15.97:0/1579848999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a406f150 0x5557a4073c50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f223c00bd50 tx=0x7f223c0107a0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.999+0000 7f2255255740 1 -- 172.21.15.97:0/1579848999 shutdown_connections 2024-03-11T19:58:22.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.999+0000 7f2255255740 1 --2- 172.21.15.97:0/1579848999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a40742c0 0x5557a4076750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.999+0000 7f2255255740 1 --2- 172.21.15.97:0/1579848999 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a406f150 0x5557a4073c50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.999+0000 7f2255255740 1 --2- 172.21.15.97:0/1579848999 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406e760 0x5557a406eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.999+0000 7f2255255740 1 -- 172.21.15.97:0/1579848999 >> 172.21.15.97:0/1579848999 conn(0x5557a4069cd0 msgr2=0x5557a406a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.999+0000 7f2255255740 1 -- 172.21.15.97:0/1579848999 shutdown_connections 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:21.999+0000 7f2255255740 1 -- 172.21.15.97:0/1579848999 wait complete. 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.000+0000 7f224be00640 1 --2- 172.21.15.97:0/1476334856 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22300a1a30 0x7f22300a3ef0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x5557a406f9b0 tx=0x7f2244009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.000+0000 7f2255255740 1 -- start start 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.000+0000 7f2255255740 1 Processor -- start 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.000+0000 7f2255255740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406e760 0x5557a40785b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.000+0000 7f2255255740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406f150 0x5557a4078af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f2255255740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a407fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f2255255740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557a3f7bea0 con 0x5557a40742c0 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f2255255740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557a3f7bb40 con 0x5557a406e760 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f2255255740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557a3d1ef20 con 0x5557a406f150 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f2255255740 10 client.0 _ll_register_callbacks cb 0x5557a3f8f950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f224be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a407fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f224b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406e760 0x5557a40785b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f224b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406e760 0x5557a40785b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:34426/0 (socket says 172.21.15.97:34426) 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f224b400640 1 -- 172.21.15.97:0/1282557090 learned_addr learned my addr 172.21.15.97:0/1282557090 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f224aa00640 1 --2- 172.21.15.97:0/1282557090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406f150 0x5557a4078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f224b400640 1 -- 172.21.15.97:0/1282557090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406f150 msgr2=0x5557a4078af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f224b400640 1 --2- 172.21.15.97:0/1282557090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406f150 0x5557a4078af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f224b400640 1 -- 172.21.15.97:0/1282557090 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 msgr2=0x5557a407fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f224b400640 1 --2- 172.21.15.97:0/1282557090 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a407fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f224b400640 1 -- 172.21.15.97:0/1282557090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223c00b980 con 0x5557a406e760 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.001+0000 7f224be00640 1 --2- 172.21.15.97:0/1282557090 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a407fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.002+0000 7f224b400640 1 --2- 172.21.15.97:0/1282557090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406e760 0x5557a40785b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f223c005d00 tx=0x7f223c01c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b7fcfdb3a507a58 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.002+0000 7f2235600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.002+0000 7f2235600640 1 -- 172.21.15.97:0/1282557090 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c021070 con 0x5557a406e760 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.002+0000 7f2255255740 1 -- 172.21.15.97:0/1282557090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5557a3ffac40 con 0x5557a406e760 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.002+0000 7f2235600640 1 -- 172.21.15.97:0/1282557090 <== mon.2 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f223c017070 con 0x5557a406e760 2024-03-11T19:58:22.004 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.002+0000 7f2235600640 1 -- 172.21.15.97:0/1282557090 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f223c01cba0 con 0x5557a406e760 2024-03-11T19:58:22.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.002+0000 7f2255255740 10 client.25190 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:22.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.002+0000 7f2255255740 1 -- 172.21.15.97:0/1282557090 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5557a3fd4ec0 con 0x5557a406e760 2024-03-11T19:58:22.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.002+0000 7f2235600640 1 -- 172.21.15.97:0/1282557090 <== mon.2 v2:172.21.15.114:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5557a3ffac40 con 0x5557a406e760 2024-03-11T19:58:22.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.002+0000 7f2235600640 1 -- 172.21.15.97:0/1282557090 <== mon.2 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f223c011440 con 0x5557a406e760 2024-03-11T19:58:22.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.003+0000 7f2255255740 10 client.25190 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:22.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.003+0000 7f2255255740 10 client.25190 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-11T19:58:22.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.003+0000 7f2255255740 20 client.25190 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:22.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.003+0000 7f2255255740 1 --2- 172.21.15.97:0/1282557090 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5557a4083d40 0x5557a4088e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.003+0000 7f2255255740 4 client.25190 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:58:22.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.003+0000 7f2255255740 1 -- 172.21.15.97:0/1282557090 --> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5557a3cd66c0 con 0x5557a4083d40 2024-03-11T19:58:22.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.003+0000 7f224aa00640 1 --2- 172.21.15.97:0/1282557090 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5557a4083d40 0x5557a4088e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.003+0000 7f224aa00640 1 --2- 172.21.15.97:0/1282557090 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5557a4083d40 0x5557a4088e70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=11cd51a1361f4f73 server_cookie=32e2eee9d5db8523 in_seq=0 out_seq=0 2024-03-11T19:58:22.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.003+0000 7f2235600640 10 client.25190 ms_handle_connect on v2:172.21.15.114:6832/2550912009 2024-03-11T19:58:22.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.004+0000 7f2235600640 1 -- 172.21.15.97:0/1282557090 <== mds.4 v2:172.21.15.114:6832/2550912009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5557a3cd66c0 con 0x5557a4083d40 2024-03-11T19:58:22.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.004+0000 7f2235600640 10 client.25190 handle_command_reply: tid=0 2024-03-11T19:58:22.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.004+0000 7f2255255740 1 client.25190 shutdown 2024-03-11T19:58:22.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.004+0000 7f2255255740 1 -- 172.21.15.97:0/1282557090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406e760 msgr2=0x5557a40785b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.004+0000 7f2255255740 1 --2- 172.21.15.97:0/1282557090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406e760 0x5557a40785b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f223c005d00 tx=0x7f223c01c6e0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.004+0000 7f2255255740 1 -- 172.21.15.97:0/1282557090 shutdown_connections 2024-03-11T19:58:22.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.004+0000 7f2255255740 1 --2- 172.21.15.97:0/1282557090 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5557a4083d40 0x5557a4088e70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.005+0000 7f2255255740 1 --2- 172.21.15.97:0/1282557090 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a407fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.005+0000 7f2255255740 1 --2- 172.21.15.97:0/1282557090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406f150 0x5557a4078af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.005+0000 7f2255255740 1 --2- 172.21.15.97:0/1282557090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406e760 0x5557a40785b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.005+0000 7f2255255740 1 -- 172.21.15.97:0/1282557090 >> 172.21.15.97:0/1282557090 conn(0x5557a4069cd0 msgr2=0x5557a4068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.005+0000 7f2235600640 0 client.25190 ms_handle_reset on v2:172.21.15.114:6832/2550912009 2024-03-11T19:58:22.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.005+0000 7f2255255740 1 -- 172.21.15.97:0/1282557090 shutdown_connections 2024-03-11T19:58:22.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.005+0000 7f2255255740 1 -- 172.21.15.97:0/1282557090 wait complete. 2024-03-11T19:58:22.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.005+0000 7f2255255740 20 client.25190 trim_cache size 0 max 16384 2024-03-11T19:58:22.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.021+0000 7f2255255740 1 Processor -- start 2024-03-11T19:58:22.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.021+0000 7f2255255740 1 -- start start 2024-03-11T19:58:22.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.021+0000 7f2255255740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406e760 0x5557a4068c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f2255255740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406f150 0x5557a407faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f2255255740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a408ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f2255255740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557a3cd66c0 con 0x5557a40742c0 2024-03-11T19:58:22.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f2255255740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557a3ffac40 con 0x5557a406f150 2024-03-11T19:58:22.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f2255255740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557a4080620 con 0x5557a406e760 2024-03-11T19:58:22.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f224be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a408ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f224b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406e760 0x5557a4068c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f224be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a408ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38856/0 (socket says 172.21.15.97:38856) 2024-03-11T19:58:22.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f224be00640 1 -- 172.21.15.97:0/2004289926 learned_addr learned my addr 172.21.15.97:0/2004289926 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f224aa00640 1 --2- 172.21.15.97:0/2004289926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406f150 0x5557a407faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f224aa00640 1 -- 172.21.15.97:0/2004289926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406e760 msgr2=0x5557a4068c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f224aa00640 1 --2- 172.21.15.97:0/2004289926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406e760 0x5557a4068c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f224aa00640 1 -- 172.21.15.97:0/2004289926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 msgr2=0x5557a408ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.022+0000 7f224aa00640 1 --2- 172.21.15.97:0/2004289926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a408ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.023+0000 7f224aa00640 1 -- 172.21.15.97:0/2004289926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557a3f718b0 con 0x5557a406f150 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.023+0000 7f224aa00640 1 --2- 172.21.15.97:0/2004289926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406f150 0x5557a407faf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f22380bc3a0 tx=0x7f22380a7040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=979a2b0fb69c09ad server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.023+0000 7f2237e00640 1 -- 172.21.15.97:0/2004289926 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22380bc000 con 0x5557a406f150 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.023+0000 7f2237e00640 1 -- 172.21.15.97:0/2004289926 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22380bcda0 con 0x5557a406f150 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.023+0000 7f2237e00640 1 -- 172.21.15.97:0/2004289926 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22380b8d10 con 0x5557a406f150 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.023+0000 7f2255255740 1 -- 172.21.15.97:0/2004289926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406f150 msgr2=0x5557a407faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.023+0000 7f2255255740 1 --2- 172.21.15.97:0/2004289926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406f150 0x5557a407faf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f22380bc3a0 tx=0x7f22380a7040 comp rx=0 tx=0).stop 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.024+0000 7f2255255740 1 -- 172.21.15.97:0/2004289926 shutdown_connections 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.024+0000 7f2255255740 1 --2- 172.21.15.97:0/2004289926 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a408ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.024+0000 7f2255255740 1 --2- 172.21.15.97:0/2004289926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406f150 0x5557a407faf0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.024+0000 7f2255255740 1 --2- 172.21.15.97:0/2004289926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406e760 0x5557a4068c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.024+0000 7f2255255740 1 -- 172.21.15.97:0/2004289926 >> 172.21.15.97:0/2004289926 conn(0x5557a4069cd0 msgr2=0x5557a4068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.024+0000 7f2255255740 1 -- 172.21.15.97:0/2004289926 shutdown_connections 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.024+0000 7f2255255740 1 -- 172.21.15.97:0/2004289926 wait complete. 2024-03-11T19:58:22.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.024+0000 7f2255255740 1 -- start start 2024-03-11T19:58:22.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.024+0000 7f2255255740 1 Processor -- start 2024-03-11T19:58:22.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f2255255740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406e760 0x5557a407bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f2255255740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406f150 0x5557a407c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f2255255740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a407f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f2255255740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557a407f6c0 con 0x5557a40742c0 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f2255255740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557a407f830 con 0x5557a406e760 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f2255255740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557a407f9f0 con 0x5557a406f150 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f2255255740 10 client.0 _ll_register_callbacks cb 0x7f223c00bba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f224be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a407f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f224aa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406f150 0x5557a407c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f224be00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a407f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38868/0 (socket says 172.21.15.97:38868) 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f224b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406e760 0x5557a407bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f224be00640 1 -- 172.21.15.97:0/39195980 learned_addr learned my addr 172.21.15.97:0/39195980 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f224aa00640 1 -- 172.21.15.97:0/39195980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406e760 msgr2=0x5557a407bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f224aa00640 1 --2- 172.21.15.97:0/39195980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406e760 0x5557a407bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f224aa00640 1 -- 172.21.15.97:0/39195980 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 msgr2=0x5557a407f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f224aa00640 1 --2- 172.21.15.97:0/39195980 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a407f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f224aa00640 1 -- 172.21.15.97:0/39195980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22380a5e70 con 0x5557a406f150 2024-03-11T19:58:22.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f224be00640 1 --2- 172.21.15.97:0/39195980 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a407f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.025+0000 7f224b400640 1 --2- 172.21.15.97:0/39195980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406e760 0x5557a407bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.026+0000 7f224aa00640 1 --2- 172.21.15.97:0/39195980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406f150 0x5557a407c160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x5557a3ffad80 tx=0x7f223802c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=546e63d3e4ffb47c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.026+0000 7f2235600640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.026+0000 7f2235600640 1 -- 172.21.15.97:0/39195980 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22380bc6a0 con 0x5557a406f150 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.026+0000 7f2255255740 1 -- 172.21.15.97:0/39195980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5557a4085a20 con 0x5557a406f150 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.026+0000 7f2235600640 1 -- 172.21.15.97:0/39195980 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f22380c4e40 con 0x5557a406f150 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.026+0000 7f2235600640 1 -- 172.21.15.97:0/39195980 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22380c07f0 con 0x5557a406f150 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.026+0000 7f2255255740 10 client.25192 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.026+0000 7f2255255740 1 -- 172.21.15.97:0/39195980 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5557a3f97510 con 0x5557a406f150 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.026+0000 7f2235600640 1 -- 172.21.15.97:0/39195980 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5557a4085a20 con 0x5557a406f150 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.026+0000 7f2235600640 1 -- 172.21.15.97:0/39195980 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f223803c070 con 0x5557a406f150 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.026+0000 7f2255255740 10 client.25192 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.027+0000 7f2255255740 10 client.25192 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-11T19:58:22.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.027+0000 7f2255255740 20 client.25192 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.027+0000 7f2255255740 1 --2- 172.21.15.97:0/39195980 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5557a40df220 0x5557a40db710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.027+0000 7f2255255740 4 client.25192 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.027+0000 7f2255255740 1 -- 172.21.15.97:0/39195980 --> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f223c015070 con 0x5557a40df220 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.027+0000 7f224b400640 1 --2- 172.21.15.97:0/39195980 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5557a40df220 0x5557a40db710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.027+0000 7f224b400640 1 --2- 172.21.15.97:0/39195980 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5557a40df220 0x5557a40db710 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=39ad4a0358f7edf9 server_cookie=2b52a18b4e8d82ea in_seq=0 out_seq=0 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.027+0000 7f2235600640 10 client.25192 ms_handle_connect on v2:172.21.15.114:6832/2550912009 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2235600640 1 -- 172.21.15.97:0/39195980 <== mds.4 v2:172.21.15.114:6832/2550912009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x7f223c015070 con 0x5557a40df220 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2235600640 10 client.25192 handle_command_reply: tid=0 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2255255740 1 client.25192 shutdown 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2255255740 1 -- 172.21.15.97:0/39195980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406f150 msgr2=0x5557a407c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2255255740 1 --2- 172.21.15.97:0/39195980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406f150 0x5557a407c160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x5557a3ffad80 tx=0x7f223802c470 comp rx=0 tx=0).stop 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2255255740 1 -- 172.21.15.97:0/39195980 shutdown_connections 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2255255740 1 --2- 172.21.15.97:0/39195980 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5557a40df220 0x5557a40db710 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2255255740 1 --2- 172.21.15.97:0/39195980 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5557a40742c0 0x5557a407f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2255255740 1 --2- 172.21.15.97:0/39195980 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5557a406f150 0x5557a407c160 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2255255740 1 --2- 172.21.15.97:0/39195980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557a406e760 0x5557a407bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2235600640 0 client.25192 ms_handle_reset on v2:172.21.15.114:6832/2550912009 2024-03-11T19:58:22.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2255255740 1 -- 172.21.15.97:0/39195980 >> 172.21.15.97:0/39195980 conn(0x5557a4069cd0 msgr2=0x5557a4068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2255255740 1 -- 172.21.15.97:0/39195980 shutdown_connections 2024-03-11T19:58:22.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2255255740 1 -- 172.21.15.97:0/39195980 wait complete. 2024-03-11T19:58:22.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.028+0000 7f2255255740 20 client.25192 trim_cache size 0 max 16384 2024-03-11T19:58:22.031 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T19:58:22.031 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:58:22.031 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 4, 2024-03-11T19:58:22.031 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24464, 2024-03-11T19:58:22.032 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-11T19:58:22.032 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-11T19:58:22.032 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-11T19:58:22.032 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 123.40203395, 2024-03-11T19:58:22.032 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-03-11T19:58:22.032 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 93, 2024-03-11T19:58:22.032 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 89, 2024-03-11T19:58:22.032 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 196.70091328399999 2024-03-11T19:58:22.032 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T19:58:22.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.029+0000 7f2251000640 1 -- 172.21.15.97:0/1476334856 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22300a1a30 msgr2=0x7f22300a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.029+0000 7f2251000640 1 --2- 172.21.15.97:0/1476334856 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22300a1a30 0x7f22300a3ef0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x5557a406f9b0 tx=0x7f2244009500 comp rx=0 tx=0).stop 2024-03-11T19:58:22.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.029+0000 7f2251000640 1 -- 172.21.15.97:0/1476334856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c0751c0 msgr2=0x7f224c199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.029+0000 7f2251000640 1 --2- 172.21.15.97:0/1476334856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c0751c0 0x7f224c199830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2238006010 tx=0x7f22380040c0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.029+0000 7f2251000640 1 -- 172.21.15.97:0/1476334856 shutdown_connections 2024-03-11T19:58:22.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.029+0000 7f2251000640 1 --2- 172.21.15.97:0/1476334856 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f22300a1a30 0x7f22300a3ef0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.029+0000 7f2251000640 1 --2- 172.21.15.97:0/1476334856 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f224c108f40 0x7f224c1a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.029+0000 7f2251000640 1 --2- 172.21.15.97:0/1476334856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f224c0751c0 0x7f224c199830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.029+0000 7f2251000640 1 --2- 172.21.15.97:0/1476334856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f224c073be0 0x7f224c1992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.029+0000 7f2251000640 1 -- 172.21.15.97:0/1476334856 >> 172.21.15.97:0/1476334856 conn(0x7f224c0fc5f0 msgr2=0x7f224c104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.029+0000 7f2251000640 1 -- 172.21.15.97:0/1476334856 shutdown_connections 2024-03-11T19:58:22.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.029+0000 7f2251000640 1 -- 172.21.15.97:0/1476334856 wait complete. 2024-03-11T19:58:22.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:21 smithi161 ceph-mon[36332]: pgmap v401: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:22.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:21 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3535730502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:22.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:21 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1541563311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:22.040 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:21 smithi097 ceph-mon[31898]: pgmap v401: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:22.041 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:21 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3535730502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:22.041 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:21 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1541563311' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:22.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-11T19:58:22.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.239+0000 7f793d800640 1 -- 172.21.15.97:0/2464178269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79381045b0 msgr2=0x7f7938104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.239+0000 7f793d800640 1 --2- 172.21.15.97:0/2464178269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79381045b0 0x7f7938104a30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7928009860 tx=0x7f792802bb70 comp rx=0 tx=0).stop 2024-03-11T19:58:22.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.239+0000 7f793d800640 1 -- 172.21.15.97:0/2464178269 shutdown_connections 2024-03-11T19:58:22.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.239+0000 7f793d800640 1 --2- 172.21.15.97:0/2464178269 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7938104f70 0x7f793810b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.239+0000 7f793d800640 1 --2- 172.21.15.97:0/2464178269 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79381045b0 0x7f7938104a30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.239+0000 7f793d800640 1 --2- 172.21.15.97:0/2464178269 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79381033c0 0x7f79381037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.239+0000 7f793d800640 1 -- 172.21.15.97:0/2464178269 >> 172.21.15.97:0/2464178269 conn(0x7f79380fe930 msgr2=0x7f7938100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.239+0000 7f793d800640 1 -- 172.21.15.97:0/2464178269 shutdown_connections 2024-03-11T19:58:22.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.239+0000 7f793d800640 1 -- 172.21.15.97:0/2464178269 wait complete. 2024-03-11T19:58:22.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.241+0000 7f793d800640 1 Processor -- start 2024-03-11T19:58:22.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.241+0000 7f793d800640 1 -- start start 2024-03-11T19:58:22.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.241+0000 7f793d800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79381033c0 0x7f7938199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.241+0000 7f793d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79381045b0 0x7f7938199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.241+0000 7f793d800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7938104f70 0x7f79381a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.241+0000 7f793d800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79381a0f30 con 0x7f7938104f70 2024-03-11T19:58:22.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.241+0000 7f793d800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79381a10a0 con 0x7f79381045b0 2024-03-11T19:58:22.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.241+0000 7f793d800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79381a1260 con 0x7f79381033c0 2024-03-11T19:58:22.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.241+0000 7f7937400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7938104f70 0x7f79381a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.242+0000 7f7936a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79381033c0 0x7f7938199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.242+0000 7f7937400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7938104f70 0x7f79381a09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38884/0 (socket says 172.21.15.97:38884) 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.242+0000 7f7937400640 1 -- 172.21.15.97:0/773335343 learned_addr learned my addr 172.21.15.97:0/773335343 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.242+0000 7f7936000640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79381045b0 0x7f7938199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.242+0000 7f7936a00640 1 -- 172.21.15.97:0/773335343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79381045b0 msgr2=0x7f7938199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.242+0000 7f7936a00640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79381045b0 0x7f7938199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.242+0000 7f7936a00640 1 -- 172.21.15.97:0/773335343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7938104f70 msgr2=0x7f79381a09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.242+0000 7f7936a00640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7938104f70 0x7f79381a09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.242+0000 7f7936a00640 1 -- 172.21.15.97:0/773335343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7928009490 con 0x7f79381033c0 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.242+0000 7f7937400640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7938104f70 0x7f79381a09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.242+0000 7f7936a00640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79381033c0 0x7f7938199440 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f792000e9c0 tx=0x7f792000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.242+0000 7f792f400640 1 -- 172.21.15.97:0/773335343 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792000ccc0 con 0x7f79381033c0 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.243+0000 7f792f400640 1 -- 172.21.15.97:0/773335343 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f792000ce20 con 0x7f79381033c0 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.243+0000 7f793d800640 1 -- 172.21.15.97:0/773335343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79381a1540 con 0x7f79381033c0 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.243+0000 7f792f400640 1 -- 172.21.15.97:0/773335343 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920010430 con 0x7f79381033c0 2024-03-11T19:58:22.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.243+0000 7f793d800640 1 -- 172.21.15.97:0/773335343 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79381a19d0 con 0x7f79381033c0 2024-03-11T19:58:22.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.247+0000 7f794058c740 1 Processor -- start 2024-03-11T19:58:22.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.247+0000 7f794058c740 1 -- start start 2024-03-11T19:58:22.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.247+0000 7f794058c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b04580 0x562976b049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.247+0000 7f794058c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04f70 0x562976b09a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.247+0000 7f794058c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b0a0e0 0x562976b0c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.247+0000 7f794058c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x562976a73b30 con 0x562976b0a0e0 2024-03-11T19:58:22.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.247+0000 7f794058c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x562976a71060 con 0x562976b04f70 2024-03-11T19:58:22.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.247+0000 7f794058c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x562976a6c840 con 0x562976b04580 2024-03-11T19:58:22.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.247+0000 7f792f400640 1 -- 172.21.15.97:0/773335343 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7920004000 con 0x7f79381033c0 2024-03-11T19:58:22.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.247+0000 7f7936000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b0a0e0 0x562976b0c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.248+0000 7f7937400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b04580 0x562976b049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.248+0000 7f7936000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b0a0e0 0x562976b0c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38892/0 (socket says 172.21.15.97:38892) 2024-03-11T19:58:22.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.248+0000 7f7936000640 1 -- 172.21.15.97:0/2383853981 learned_addr learned my addr 172.21.15.97:0/2383853981 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.248+0000 7f7936a00640 1 --2- 172.21.15.97:0/2383853981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04f70 0x562976b09a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.248+0000 7f7937400640 1 -- 172.21.15.97:0/2383853981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04f70 msgr2=0x562976b09a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.248+0000 7f7937400640 1 --2- 172.21.15.97:0/2383853981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04f70 0x562976b09a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.248+0000 7f7937400640 1 -- 172.21.15.97:0/2383853981 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b0a0e0 msgr2=0x562976b0c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.248+0000 7f7937400640 1 --2- 172.21.15.97:0/2383853981 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b0a0e0 0x562976b0c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.248+0000 7f7937400640 1 -- 172.21.15.97:0/2383853981 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562976a344c0 con 0x562976b04580 2024-03-11T19:58:22.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.248+0000 7f7936a00640 1 --2- 172.21.15.97:0/2383853981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04f70 0x562976b09a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:22.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.248+0000 7f7937400640 1 --2- 172.21.15.97:0/2383853981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b04580 0x562976b049a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f793000e930 tx=0x7f793000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ef9de7c111611b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.249+0000 7f7927e00640 1 -- 172.21.15.97:0/2383853981 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793001c070 con 0x562976b04580 2024-03-11T19:58:22.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.249+0000 7f7927e00640 1 -- 172.21.15.97:0/2383853981 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7930014040 con 0x562976b04580 2024-03-11T19:58:22.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.249+0000 7f7927e00640 1 -- 172.21.15.97:0/2383853981 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930004460 con 0x562976b04580 2024-03-11T19:58:22.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.249+0000 7f792f400640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f791c0a1a30 0x7f791c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.249+0000 7f792f400640 1 -- 172.21.15.97:0/773335343 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f7920014070 con 0x7f79381033c0 2024-03-11T19:58:22.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.249+0000 7f7936a00640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f791c0a1a30 0x7f791c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.249+0000 7f794058c740 1 -- 172.21.15.97:0/2383853981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b04580 msgr2=0x562976b049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.249+0000 7f794058c740 1 --2- 172.21.15.97:0/2383853981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b04580 0x562976b049a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f793000e930 tx=0x7f793000edf0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.249+0000 7f794058c740 1 -- 172.21.15.97:0/2383853981 shutdown_connections 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.249+0000 7f794058c740 1 --2- 172.21.15.97:0/2383853981 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b0a0e0 0x562976b0c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.249+0000 7f794058c740 1 --2- 172.21.15.97:0/2383853981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04f70 0x562976b09a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.249+0000 7f794058c740 1 --2- 172.21.15.97:0/2383853981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b04580 0x562976b049a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.249+0000 7f794058c740 1 -- 172.21.15.97:0/2383853981 >> 172.21.15.97:0/2383853981 conn(0x562976affaf0 msgr2=0x562976affef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.250+0000 7f794058c740 1 -- 172.21.15.97:0/2383853981 shutdown_connections 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.250+0000 7f794058c740 1 -- 172.21.15.97:0/2383853981 wait complete. 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.250+0000 7f7936a00640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f791c0a1a30 0x7f791c0a3ef0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f79200cb5c0 tx=0x7f792000cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.250+0000 7f794058c740 1 -- start start 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.250+0000 7f794058c740 1 Processor -- start 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f794058c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04580 0x562976b0e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f794058c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b04f70 0x562976b0e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f794058c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b0a0e0 0x562976b15920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f794058c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x562976a11ea0 con 0x562976b0a0e0 2024-03-11T19:58:22.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f794058c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x562976a11b40 con 0x562976b04580 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f794058c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629767b4f20 con 0x562976b04f70 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f794058c740 10 client.0 _ll_register_callbacks cb 0x562976a25950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f7937400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b0a0e0 0x562976b15920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f7936000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b04f70 0x562976b0e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f7936000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b04f70 0x562976b0e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:39310/0 (socket says 172.21.15.97:39310) 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f7936000640 1 -- 172.21.15.97:0/2341144510 learned_addr learned my addr 172.21.15.97:0/2341144510 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f7937400640 1 --2- 172.21.15.97:0/2341144510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b0a0e0 0x562976b15920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38908/0 (socket says 172.21.15.97:38908) 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f7937400640 1 --2- 172.21.15.97:0/2341144510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04580 0x562976b0e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.251+0000 7f7936000640 1 -- 172.21.15.97:0/2341144510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04580 msgr2=0x562976b0e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.252+0000 7f7936000640 1 --2- 172.21.15.97:0/2341144510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04580 0x562976b0e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.252+0000 7f7936000640 1 -- 172.21.15.97:0/2341144510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b0a0e0 msgr2=0x562976b15920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.252+0000 7f7936000640 1 --2- 172.21.15.97:0/2341144510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b0a0e0 0x562976b15920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.252+0000 7f7936000640 1 -- 172.21.15.97:0/2341144510 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7930009df0 con 0x562976b04f70 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.252+0000 7f7936000640 1 --2- 172.21.15.97:0/2341144510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b04f70 0x562976b0e8e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7928034b10 tx=0x7f79280045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=860178497cfe40b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.252+0000 7f7924c00640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.252+0000 7f7924c00640 1 -- 172.21.15.97:0/2341144510 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7928034910 con 0x562976b04f70 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.252+0000 7f7924c00640 1 -- 172.21.15.97:0/2341144510 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f792802c510 con 0x562976b04f70 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.252+0000 7f794058c740 1 -- 172.21.15.97:0/2341144510 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562976a90c40 con 0x562976b04f70 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.253+0000 7f7924c00640 1 -- 172.21.15.97:0/2341144510 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f792802c6d0 con 0x562976b04f70 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.253+0000 7f794058c740 10 client.25204 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.253+0000 7f794058c740 1 -- 172.21.15.97:0/2341144510 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562976a6aec0 con 0x562976b04f70 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.253+0000 7f7924c00640 1 -- 172.21.15.97:0/2341144510 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562976a90c40 con 0x562976b04f70 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.253+0000 7f7924c00640 1 -- 172.21.15.97:0/2341144510 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7928045020 con 0x562976b04f70 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.253+0000 7f794058c740 10 client.25204 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.253+0000 7f794058c740 10 client.25204 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.253+0000 7f794058c740 20 client.25204 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.253+0000 7f794058c740 1 --2- 172.21.15.97:0/2341144510 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x562976b1a200 0x562976b19b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.253+0000 7f794058c740 4 client.25204 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.253+0000 7f794058c740 1 -- 172.21.15.97:0/2341144510 --> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56297676c6c0 con 0x562976b1a200 2024-03-11T19:58:22.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.254+0000 7f7937400640 1 --2- 172.21.15.97:0/2341144510 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x562976b1a200 0x562976b19b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.254+0000 7f7937400640 1 --2- 172.21.15.97:0/2341144510 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x562976b1a200 0x562976b19b30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d779ee0c2e12a2a1 server_cookie=372cb30ad28b1e13 in_seq=0 out_seq=0 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.254+0000 7f7924c00640 10 client.25204 ms_handle_connect on v2:172.21.15.114:6836/1112816294 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.255+0000 7f7924c00640 1 -- 172.21.15.97:0/2341144510 <== mds.3 v2:172.21.15.114:6836/1112816294 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56297676c6c0 con 0x562976b1a200 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.255+0000 7f7924c00640 10 client.25204 handle_command_reply: tid=0 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.255+0000 7f794058c740 1 client.25204 shutdown 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.255+0000 7f794058c740 1 -- 172.21.15.97:0/2341144510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b04f70 msgr2=0x562976b0e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.255+0000 7f794058c740 1 --2- 172.21.15.97:0/2341144510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b04f70 0x562976b0e8e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7928034b10 tx=0x7f79280045b0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.255+0000 7f794058c740 1 -- 172.21.15.97:0/2341144510 shutdown_connections 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.255+0000 7f794058c740 1 --2- 172.21.15.97:0/2341144510 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x562976b1a200 0x562976b19b30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.255+0000 7f794058c740 1 --2- 172.21.15.97:0/2341144510 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b0a0e0 0x562976b15920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.255+0000 7f7924c00640 0 client.25204 ms_handle_reset on v2:172.21.15.114:6836/1112816294 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.255+0000 7f794058c740 1 --2- 172.21.15.97:0/2341144510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b04f70 0x562976b0e8e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.255+0000 7f794058c740 1 --2- 172.21.15.97:0/2341144510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04580 0x562976b0e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.255+0000 7f794058c740 1 -- 172.21.15.97:0/2341144510 >> 172.21.15.97:0/2341144510 conn(0x562976affaf0 msgr2=0x562976afee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.255+0000 7f794058c740 1 -- 172.21.15.97:0/2341144510 shutdown_connections 2024-03-11T19:58:22.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.256+0000 7f794058c740 1 -- 172.21.15.97:0/2341144510 wait complete. 2024-03-11T19:58:22.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.256+0000 7f794058c740 20 client.25204 trim_cache size 0 max 16384 2024-03-11T19:58:22.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.272+0000 7f794058c740 1 Processor -- start 2024-03-11T19:58:22.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.273+0000 7f794058c740 1 -- start start 2024-03-11T19:58:22.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.273+0000 7f794058c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04580 0x562976b03460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.273+0000 7f794058c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04f70 0x562976b158e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.273+0000 7f794058c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b0a0e0 0x562976b05a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.273+0000 7f794058c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x562976b0c700 con 0x562976b04f70 2024-03-11T19:58:22.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.273+0000 7f794058c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x562976b0efb0 con 0x562976b04580 2024-03-11T19:58:22.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.273+0000 7f794058c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x562976a90c40 con 0x562976b0a0e0 2024-03-11T19:58:22.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.273+0000 7f7936000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04f70 0x562976b158e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.273+0000 7f7936000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04f70 0x562976b158e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38918/0 (socket says 172.21.15.97:38918) 2024-03-11T19:58:22.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.273+0000 7f7936000640 1 -- 172.21.15.97:0/3161148868 learned_addr learned my addr 172.21.15.97:0/3161148868 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.273+0000 7f7937400640 1 --2- 172.21.15.97:0/3161148868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04580 0x562976b03460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.274+0000 7f7937400640 1 --2- 172.21.15.97:0/3161148868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b0a0e0 0x562976b05a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.274+0000 7f7936000640 1 -- 172.21.15.97:0/3161148868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b0a0e0 msgr2=0x562976b05a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.274+0000 7f7936000640 1 --2- 172.21.15.97:0/3161148868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b0a0e0 0x562976b05a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.274+0000 7f7936000640 1 -- 172.21.15.97:0/3161148868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04580 msgr2=0x562976b03460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.274+0000 7f7936000640 1 --2- 172.21.15.97:0/3161148868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04580 0x562976b03460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.274+0000 7f7936000640 1 -- 172.21.15.97:0/3161148868 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562976a078b0 con 0x562976b04f70 2024-03-11T19:58:22.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.274+0000 7f7937400640 1 --2- 172.21.15.97:0/3161148868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04580 0x562976b03460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:22.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.274+0000 7f7936000640 1 --2- 172.21.15.97:0/3161148868 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04f70 0x562976b158e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f79280044c0 tx=0x7f7928005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a0d19dd7c06915f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.274+0000 7f7927400640 1 -- 172.21.15.97:0/3161148868 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79280096b0 con 0x562976b04f70 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.274+0000 7f7927400640 1 -- 172.21.15.97:0/3161148868 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7928005510 con 0x562976b04f70 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.274+0000 7f7927400640 1 -- 172.21.15.97:0/3161148868 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792802c510 con 0x562976b04f70 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.275+0000 7f794058c740 1 -- 172.21.15.97:0/3161148868 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04f70 msgr2=0x562976b158e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.275+0000 7f794058c740 1 --2- 172.21.15.97:0/3161148868 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04f70 0x562976b158e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f79280044c0 tx=0x7f7928005d50 comp rx=0 tx=0).stop 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.275+0000 7f794058c740 1 -- 172.21.15.97:0/3161148868 shutdown_connections 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.275+0000 7f794058c740 1 --2- 172.21.15.97:0/3161148868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b0a0e0 0x562976b05a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.275+0000 7f794058c740 1 --2- 172.21.15.97:0/3161148868 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04f70 0x562976b158e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.275+0000 7f794058c740 1 --2- 172.21.15.97:0/3161148868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04580 0x562976b03460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.275+0000 7f794058c740 1 -- 172.21.15.97:0/3161148868 >> 172.21.15.97:0/3161148868 conn(0x562976affaf0 msgr2=0x562976b02d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.275+0000 7f794058c740 1 -- 172.21.15.97:0/3161148868 shutdown_connections 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.275+0000 7f794058c740 1 -- 172.21.15.97:0/3161148868 wait complete. 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.275+0000 7f794058c740 1 -- start start 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.276+0000 7f794058c740 1 Processor -- start 2024-03-11T19:58:22.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.276+0000 7f794058c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04580 0x562976b12e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.276+0000 7f794058c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04f70 0x562976b133b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.276+0000 7f794058c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b0a0e0 0x562976b14350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.276+0000 7f794058c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x562976b14890 con 0x562976b04580 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.276+0000 7f794058c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x562976b14a00 con 0x562976b04f70 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.276+0000 7f794058c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x562976b14bc0 con 0x562976b0a0e0 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.276+0000 7f794058c740 10 client.0 _ll_register_callbacks cb 0x7f7920005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.276+0000 7f7937400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04580 0x562976b12e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.276+0000 7f7937400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04580 0x562976b12e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38920/0 (socket says 172.21.15.97:38920) 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.276+0000 7f7937400640 1 -- 172.21.15.97:0/3138455921 learned_addr learned my addr 172.21.15.97:0/3138455921 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.276+0000 7f7936000640 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04f70 0x562976b133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.277+0000 7f7937400640 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b0a0e0 0x562976b14350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.277+0000 7f7937400640 1 -- 172.21.15.97:0/3138455921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b0a0e0 msgr2=0x562976b14350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.277+0000 7f7937400640 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b0a0e0 0x562976b14350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.277+0000 7f7937400640 1 -- 172.21.15.97:0/3138455921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04f70 msgr2=0x562976b133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.277+0000 7f7937400640 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04f70 0x562976b133b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.277+0000 7f7937400640 1 -- 172.21.15.97:0/3138455921 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562976a6aec0 con 0x562976b04580 2024-03-11T19:58:22.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.277+0000 7f7936000640 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04f70 0x562976b133b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:22.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.277+0000 7f7937400640 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04580 0x562976b12e70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f79300097f0 tx=0x7f7930009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d47cfc676156509 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.277+0000 7f7924c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:22.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.277+0000 7f7924c00640 1 -- 172.21.15.97:0/3138455921 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793001c040 con 0x562976b04580 2024-03-11T19:58:22.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.277+0000 7f794058c740 1 -- 172.21.15.97:0/3138455921 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562976b1b210 con 0x562976b04580 2024-03-11T19:58:22.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.277+0000 7f7924c00640 1 -- 172.21.15.97:0/3138455921 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f793002dd80 con 0x562976b04580 2024-03-11T19:58:22.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.278+0000 7f7924c00640 1 -- 172.21.15.97:0/3138455921 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f793002fea0 con 0x562976b04580 2024-03-11T19:58:22.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.278+0000 7f794058c740 10 client.15102 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:22.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.278+0000 7f794058c740 1 -- 172.21.15.97:0/3138455921 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562976a2d510 con 0x562976b04580 2024-03-11T19:58:22.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.278+0000 7f7924c00640 1 -- 172.21.15.97:0/3138455921 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562976b1b210 con 0x562976b04580 2024-03-11T19:58:22.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.278+0000 7f7924c00640 1 -- 172.21.15.97:0/3138455921 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7930038070 con 0x562976b04580 2024-03-11T19:58:22.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.278+0000 7f794058c740 10 client.15102 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:22.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.278+0000 7f794058c740 10 client.15102 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-11T19:58:22.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.278+0000 7f794058c740 20 client.15102 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:22.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.278+0000 7f794058c740 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x562976b76420 0x562976b728a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.278+0000 7f794058c740 4 client.15102 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:58:22.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.278+0000 7f794058c740 1 -- 172.21.15.97:0/3138455921 --> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f7930014040 con 0x562976b76420 2024-03-11T19:58:22.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.278+0000 7f7936000640 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x562976b76420 0x562976b728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f7936000640 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x562976b76420 0x562976b728a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=30a852f4332131 server_cookie=347487dd4b3657a2 in_seq=0 out_seq=0 2024-03-11T19:58:22.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f7924c00640 10 client.15102 ms_handle_connect on v2:172.21.15.114:6836/1112816294 2024-03-11T19:58:22.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f7924c00640 1 -- 172.21.15.97:0/3138455921 <== mds.3 v2:172.21.15.114:6836/1112816294 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f7930014040 con 0x562976b76420 2024-03-11T19:58:22.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f7924c00640 10 client.15102 handle_command_reply: tid=0 2024-03-11T19:58:22.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f794058c740 1 client.15102 shutdown 2024-03-11T19:58:22.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f794058c740 1 -- 172.21.15.97:0/3138455921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04580 msgr2=0x562976b12e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f794058c740 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04580 0x562976b12e70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f79300097f0 tx=0x7f7930009780 comp rx=0 tx=0).stop 2024-03-11T19:58:22.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f794058c740 1 -- 172.21.15.97:0/3138455921 shutdown_connections 2024-03-11T19:58:22.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f794058c740 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x562976b76420 0x562976b728a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f794058c740 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562976b0a0e0 0x562976b14350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f794058c740 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562976b04f70 0x562976b133b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f794058c740 1 --2- 172.21.15.97:0/3138455921 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x562976b04580 0x562976b12e70 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f7924c00640 0 client.15102 ms_handle_reset on v2:172.21.15.114:6836/1112816294 2024-03-11T19:58:22.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.279+0000 7f794058c740 1 -- 172.21.15.97:0/3138455921 >> 172.21.15.97:0/3138455921 conn(0x562976affaf0 msgr2=0x562976b035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.280+0000 7f794058c740 1 -- 172.21.15.97:0/3138455921 shutdown_connections 2024-03-11T19:58:22.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.280+0000 7f794058c740 1 -- 172.21.15.97:0/3138455921 wait complete. 2024-03-11T19:58:22.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.280+0000 7f794058c740 20 client.15102 trim_cache size 0 max 16384 2024-03-11T19:58:22.285 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T19:58:22.285 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:58:22.285 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 3, 2024-03-11T19:58:22.285 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24545, 2024-03-11T19:58:22.286 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-11T19:58:22.286 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-11T19:58:22.286 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-11T19:58:22.286 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 124.62729299199999, 2024-03-11T19:58:22.286 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-03-11T19:58:22.286 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 93, 2024-03-11T19:58:22.287 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 89, 2024-03-11T19:58:22.287 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 177.332921693 2024-03-11T19:58:22.287 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T19:58:22.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.280+0000 7f793d800640 1 -- 172.21.15.97:0/773335343 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f791c0a1a30 msgr2=0x7f791c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.280+0000 7f793d800640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f791c0a1a30 0x7f791c0a3ef0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f79200cb5c0 tx=0x7f792000cf90 comp rx=0 tx=0).stop 2024-03-11T19:58:22.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.280+0000 7f793d800640 1 -- 172.21.15.97:0/773335343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79381033c0 msgr2=0x7f7938199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.280+0000 7f793d800640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79381033c0 0x7f7938199440 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f792000e9c0 tx=0x7f792000ee80 comp rx=0 tx=0).stop 2024-03-11T19:58:22.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.281+0000 7f793d800640 1 -- 172.21.15.97:0/773335343 shutdown_connections 2024-03-11T19:58:22.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.281+0000 7f793d800640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f791c0a1a30 0x7f791c0a3ef0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.281+0000 7f793d800640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7938104f70 0x7f79381a09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.281+0000 7f793d800640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79381045b0 0x7f7938199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.281+0000 7f793d800640 1 --2- 172.21.15.97:0/773335343 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79381033c0 0x7f7938199440 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.281+0000 7f793d800640 1 -- 172.21.15.97:0/773335343 >> 172.21.15.97:0/773335343 conn(0x7f79380fe930 msgr2=0x7f7938107900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.281+0000 7f793d800640 1 -- 172.21.15.97:0/773335343 shutdown_connections 2024-03-11T19:58:22.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.281+0000 7f793d800640 1 -- 172.21.15.97:0/773335343 wait complete. 2024-03-11T19:58:22.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-11T19:58:22.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.498+0000 7f578e800640 1 -- 172.21.15.97:0/1967775443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5788109350 msgr2=0x7f578810fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.499+0000 7f578e800640 1 --2- 172.21.15.97:0/1967775443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5788109350 0x7f578810fbe0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5780009ca0 tx=0x7f578002c100 comp rx=0 tx=0).stop 2024-03-11T19:58:22.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.499+0000 7f578e800640 1 -- 172.21.15.97:0/1967775443 shutdown_connections 2024-03-11T19:58:22.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.499+0000 7f578e800640 1 --2- 172.21.15.97:0/1967775443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5788109350 0x7f578810fbe0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.499+0000 7f578e800640 1 --2- 172.21.15.97:0/1967775443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5788108990 0x7f5788108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.499+0000 7f578e800640 1 --2- 172.21.15.97:0/1967775443 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57881077a0 0x7f5788107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.499+0000 7f578e800640 1 -- 172.21.15.97:0/1967775443 >> 172.21.15.97:0/1967775443 conn(0x7f5788076740 msgr2=0x7f5788078b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.499+0000 7f578e800640 1 -- 172.21.15.97:0/1967775443 shutdown_connections 2024-03-11T19:58:22.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.499+0000 7f578e800640 1 -- 172.21.15.97:0/1967775443 wait complete. 2024-03-11T19:58:22.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.500+0000 7f578e800640 1 Processor -- start 2024-03-11T19:58:22.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.500+0000 7f578e800640 1 -- start start 2024-03-11T19:58:22.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.500+0000 7f578e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57881077a0 0x7f578819d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.500+0000 7f578e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5788108990 0x7f578819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.500+0000 7f578e800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5788109350 0x7f57881a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.500+0000 7f578e800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57881a52e0 con 0x7f5788108990 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.500+0000 7f578e800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57881a5450 con 0x7f57881077a0 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.500+0000 7f578e800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57881a5610 con 0x7f5788109350 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.500+0000 7f5786a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5788108990 0x7f578819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.500+0000 7f5786a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5788108990 0x7f578819dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38946/0 (socket says 172.21.15.97:38946) 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.500+0000 7f5786a00640 1 -- 172.21.15.97:0/748770169 learned_addr learned my addr 172.21.15.97:0/748770169 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.500+0000 7f5787e00640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5788109350 0x7f57881a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.500+0000 7f5787400640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57881077a0 0x7f578819d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.501+0000 7f5786a00640 1 -- 172.21.15.97:0/748770169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5788109350 msgr2=0x7f57881a4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.501+0000 7f5786a00640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5788109350 0x7f57881a4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.501+0000 7f5786a00640 1 -- 172.21.15.97:0/748770169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57881077a0 msgr2=0x7f578819d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.501+0000 7f5786a00640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57881077a0 0x7f578819d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.501+0000 7f5786a00640 1 -- 172.21.15.97:0/748770169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5780009900 con 0x7f5788108990 2024-03-11T19:58:22.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.501+0000 7f5787e00640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5788109350 0x7f57881a4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:22.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.501+0000 7f5787400640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57881077a0 0x7f578819d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:22.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.501+0000 7f5786a00640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5788108990 0x7f578819dd30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f577400e9f0 tx=0x7f577400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.501+0000 7f577fe00640 1 -- 172.21.15.97:0/748770169 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5774009700 con 0x7f5788108990 2024-03-11T19:58:22.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.501+0000 7f577fe00640 1 -- 172.21.15.97:0/748770169 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f577400cd80 con 0x7f5788108990 2024-03-11T19:58:22.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.501+0000 7f578e800640 1 -- 172.21.15.97:0/748770169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57881a58f0 con 0x7f5788108990 2024-03-11T19:58:22.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.501+0000 7f577fe00640 1 -- 172.21.15.97:0/748770169 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5774018920 con 0x7f5788108990 2024-03-11T19:58:22.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.502+0000 7f578e800640 1 -- 172.21.15.97:0/748770169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57881a5e00 con 0x7f5788108990 2024-03-11T19:58:22.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.505+0000 7f579144a740 1 Processor -- start 2024-03-11T19:58:22.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.505+0000 7f579144a740 1 -- start start 2024-03-11T19:58:22.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.505+0000 7f577fe00640 1 -- 172.21.15.97:0/748770169 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5774018a80 con 0x7f5788108990 2024-03-11T19:58:22.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.505+0000 7f579144a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed44f580 0x561bed44f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f579144a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44ff70 0x561bed454a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f579144a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed4550e0 0x561bed457570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f579144a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed3beb30 con 0x561bed4550e0 2024-03-11T19:58:22.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f579144a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed3bc060 con 0x561bed44ff70 2024-03-11T19:58:22.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f579144a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed3b7840 con 0x561bed44f580 2024-03-11T19:58:22.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f5786a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed4550e0 0x561bed457570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f5787e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed44f580 0x561bed44f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f5786a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed4550e0 0x561bed457570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38952/0 (socket says 172.21.15.97:38952) 2024-03-11T19:58:22.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f5786a00640 1 -- 172.21.15.97:0/186198234 learned_addr learned my addr 172.21.15.97:0/186198234 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f5787400640 1 --2- 172.21.15.97:0/186198234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44ff70 0x561bed454a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f5787e00640 1 -- 172.21.15.97:0/186198234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44ff70 msgr2=0x561bed454a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f5787e00640 1 --2- 172.21.15.97:0/186198234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44ff70 0x561bed454a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f5787e00640 1 -- 172.21.15.97:0/186198234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed4550e0 msgr2=0x561bed457570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.512 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f5787e00640 1 --2- 172.21.15.97:0/186198234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed4550e0 0x561bed457570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.506+0000 7f5787e00640 1 -- 172.21.15.97:0/186198234 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561bed37f4c0 con 0x561bed44f580 2024-03-11T19:58:22.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.507+0000 7f5787e00640 1 --2- 172.21.15.97:0/186198234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed44f580 0x561bed44f9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f578002c640 tx=0x7f57800093f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b43c3db0ed2191c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.507+0000 7f577fe00640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f576c0a1a30 0x7f576c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.507+0000 7f577cc00640 1 -- 172.21.15.97:0/186198234 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5780032d90 con 0x561bed44f580 2024-03-11T19:58:22.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.507+0000 7f577fe00640 1 -- 172.21.15.97:0/748770169 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f5774014070 con 0x7f5788108990 2024-03-11T19:58:22.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.507+0000 7f577cc00640 1 -- 172.21.15.97:0/186198234 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5780003970 con 0x561bed44f580 2024-03-11T19:58:22.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.507+0000 7f577cc00640 1 -- 172.21.15.97:0/186198234 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5780004460 con 0x561bed44f580 2024-03-11T19:58:22.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.507+0000 7f5787400640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f576c0a1a30 0x7f576c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.508+0000 7f579144a740 1 -- 172.21.15.97:0/186198234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed44f580 msgr2=0x561bed44f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.508+0000 7f5787400640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f576c0a1a30 0x7f576c0a3ef0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x561bed450370 tx=0x7f577800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.508+0000 7f579144a740 1 --2- 172.21.15.97:0/186198234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed44f580 0x561bed44f9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f578002c640 tx=0x7f57800093f0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.508+0000 7f579144a740 1 -- 172.21.15.97:0/186198234 shutdown_connections 2024-03-11T19:58:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.508+0000 7f579144a740 1 --2- 172.21.15.97:0/186198234 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed4550e0 0x561bed457570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.508+0000 7f579144a740 1 --2- 172.21.15.97:0/186198234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44ff70 0x561bed454a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.508+0000 7f579144a740 1 --2- 172.21.15.97:0/186198234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed44f580 0x561bed44f9a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.508+0000 7f579144a740 1 -- 172.21.15.97:0/186198234 >> 172.21.15.97:0/186198234 conn(0x561bed44aaf0 msgr2=0x561bed44aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.508+0000 7f579144a740 1 -- 172.21.15.97:0/186198234 shutdown_connections 2024-03-11T19:58:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.508+0000 7f579144a740 1 -- 172.21.15.97:0/186198234 wait complete. 2024-03-11T19:58:22.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.509+0000 7f579144a740 1 -- start start 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.509+0000 7f579144a740 1 Processor -- start 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.509+0000 7f579144a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 0x561bed4593d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.509+0000 7f579144a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed44ff70 0x561bed459910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.509+0000 7f579144a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed4550e0 0x561bed460950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.509+0000 7f579144a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed35cea0 con 0x561bed44f580 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.509+0000 7f579144a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed35cb40 con 0x561bed4550e0 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.509+0000 7f579144a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed0fff20 con 0x561bed44ff70 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.509+0000 7f579144a740 10 client.0 _ll_register_callbacks cb 0x561bed370950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.509+0000 7f5787400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 0x561bed4593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.509+0000 7f5787400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 0x561bed4593d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38966/0 (socket says 172.21.15.97:38966) 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f5787400640 1 -- 172.21.15.97:0/177179325 learned_addr learned my addr 172.21.15.97:0/177179325 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f5787e00640 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed4550e0 0x561bed460950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f5786a00640 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed44ff70 0x561bed459910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.515 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f5787400640 1 -- 172.21.15.97:0/177179325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed44ff70 msgr2=0x561bed459910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f5787400640 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed44ff70 0x561bed459910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f5787400640 1 -- 172.21.15.97:0/177179325 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed4550e0 msgr2=0x561bed460950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f5787400640 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed4550e0 0x561bed460950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f5787400640 1 -- 172.21.15.97:0/177179325 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57881104e0 con 0x561bed44f580 2024-03-11T19:58:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f5787e00640 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed4550e0 0x561bed460950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f5787400640 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 0x561bed4593d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f577800bf40 tx=0x7f5778004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a4e9a2073d39db server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f5771600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f5771600640 1 -- 172.21.15.97:0/177179325 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5778016070 con 0x561bed44f580 2024-03-11T19:58:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f5771600640 1 -- 172.21.15.97:0/177179325 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f5778010e70 con 0x561bed44f580 2024-03-11T19:58:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.510+0000 7f579144a740 1 -- 172.21.15.97:0/177179325 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561bed3dbc40 con 0x561bed44f580 2024-03-11T19:58:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.511+0000 7f5771600640 1 -- 172.21.15.97:0/177179325 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f577801fac0 con 0x561bed44f580 2024-03-11T19:58:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.511+0000 7f579144a740 10 client.15114 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:22.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.511+0000 7f579144a740 1 -- 172.21.15.97:0/177179325 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561bed3b5ec0 con 0x561bed44f580 2024-03-11T19:58:22.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.511+0000 7f5771600640 1 -- 172.21.15.97:0/177179325 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561bed3dbc40 con 0x561bed44f580 2024-03-11T19:58:22.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.511+0000 7f5771600640 1 -- 172.21.15.97:0/177179325 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f577801b390 con 0x561bed44f580 2024-03-11T19:58:22.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.511+0000 7f579144a740 10 client.15114 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:22.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.511+0000 7f579144a740 10 client.15114 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-11T19:58:22.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.511+0000 7f579144a740 20 client.15114 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:22.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.511+0000 7f579144a740 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x561bed465af0 0x561bed464d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.511+0000 7f579144a740 4 client.15114 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:58:22.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.511+0000 7f579144a740 1 -- 172.21.15.97:0/177179325 --> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561bed0b76c0 con 0x561bed465af0 2024-03-11T19:58:22.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.512+0000 7f5786a00640 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x561bed465af0 0x561bed464d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.512+0000 7f5786a00640 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x561bed465af0 0x561bed464d40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e5006fb16b4c5e3 server_cookie=6d363bb7523de1d0 in_seq=0 out_seq=0 2024-03-11T19:58:22.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.512+0000 7f5771600640 10 client.15114 ms_handle_connect on v2:172.21.15.161:6838/565994162 2024-03-11T19:58:22.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.513+0000 7f5771600640 1 -- 172.21.15.97:0/177179325 <== mds.2 v2:172.21.15.161:6838/565994162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561bed0b76c0 con 0x561bed465af0 2024-03-11T19:58:22.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.513+0000 7f5771600640 10 client.15114 handle_command_reply: tid=0 2024-03-11T19:58:22.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.513+0000 7f579144a740 1 client.15114 shutdown 2024-03-11T19:58:22.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.513+0000 7f579144a740 1 -- 172.21.15.97:0/177179325 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 msgr2=0x561bed4593d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.513+0000 7f579144a740 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 0x561bed4593d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f577800bf40 tx=0x7f5778004730 comp rx=0 tx=0).stop 2024-03-11T19:58:22.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.513+0000 7f579144a740 1 -- 172.21.15.97:0/177179325 shutdown_connections 2024-03-11T19:58:22.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.513+0000 7f579144a740 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x561bed465af0 0x561bed464d40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.513+0000 7f579144a740 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed4550e0 0x561bed460950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.513+0000 7f579144a740 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed44ff70 0x561bed459910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.513+0000 7f579144a740 1 --2- 172.21.15.97:0/177179325 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 0x561bed4593d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.513+0000 7f579144a740 1 -- 172.21.15.97:0/177179325 >> 172.21.15.97:0/177179325 conn(0x561bed44aaf0 msgr2=0x561bed449e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.513+0000 7f5771600640 0 client.15114 ms_handle_reset on v2:172.21.15.161:6838/565994162 2024-03-11T19:58:22.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.513+0000 7f579144a740 1 -- 172.21.15.97:0/177179325 shutdown_connections 2024-03-11T19:58:22.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.514+0000 7f579144a740 1 -- 172.21.15.97:0/177179325 wait complete. 2024-03-11T19:58:22.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.514+0000 7f579144a740 20 client.15114 trim_cache size 0 max 16384 2024-03-11T19:58:22.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.530+0000 7f579144a740 1 Processor -- start 2024-03-11T19:58:22.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.530+0000 7f579144a740 1 -- start start 2024-03-11T19:58:22.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.531+0000 7f579144a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44f580 0x561bed464d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.531+0000 7f579144a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44ff70 0x561bed460910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.531+0000 7f579144a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed4550e0 0x561bed449580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.531+0000 7f579144a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed461210 con 0x561bed44ff70 2024-03-11T19:58:22.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.531+0000 7f579144a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed44b5e0 con 0x561bed44f580 2024-03-11T19:58:22.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.531+0000 7f579144a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed0b76c0 con 0x561bed4550e0 2024-03-11T19:58:22.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.531+0000 7f5786a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44ff70 0x561bed460910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.531+0000 7f5786a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44ff70 0x561bed460910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38974/0 (socket says 172.21.15.97:38974) 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.531+0000 7f5786a00640 1 -- 172.21.15.97:0/2951606743 learned_addr learned my addr 172.21.15.97:0/2951606743 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.531+0000 7f5787e00640 1 --2- 172.21.15.97:0/2951606743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed4550e0 0x561bed449580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.531+0000 7f5787400640 1 --2- 172.21.15.97:0/2951606743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44f580 0x561bed464d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.531+0000 7f5786a00640 1 -- 172.21.15.97:0/2951606743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed4550e0 msgr2=0x561bed449580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.531+0000 7f5786a00640 1 --2- 172.21.15.97:0/2951606743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed4550e0 0x561bed449580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.532+0000 7f5786a00640 1 -- 172.21.15.97:0/2951606743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44f580 msgr2=0x561bed464d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.532+0000 7f5786a00640 1 --2- 172.21.15.97:0/2951606743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44f580 0x561bed464d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.532+0000 7f5786a00640 1 -- 172.21.15.97:0/2951606743 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561bed3528b0 con 0x561bed44ff70 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.532+0000 7f5786a00640 1 --2- 172.21.15.97:0/2951606743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44ff70 0x561bed460910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f57740cb2c0 tx=0x7f57740cb140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837ee535f1faf463 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.532+0000 7f5773e00640 1 -- 172.21.15.97:0/2951606743 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577409a0f0 con 0x561bed44ff70 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.532+0000 7f5773e00640 1 -- 172.21.15.97:0/2951606743 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f577409a790 con 0x561bed44ff70 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.532+0000 7f5773e00640 1 -- 172.21.15.97:0/2951606743 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5774010b80 con 0x561bed44ff70 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.533+0000 7f579144a740 1 -- 172.21.15.97:0/2951606743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44ff70 msgr2=0x561bed460910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.533+0000 7f579144a740 1 --2- 172.21.15.97:0/2951606743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44ff70 0x561bed460910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f57740cb2c0 tx=0x7f57740cb140 comp rx=0 tx=0).stop 2024-03-11T19:58:22.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.533+0000 7f579144a740 1 -- 172.21.15.97:0/2951606743 shutdown_connections 2024-03-11T19:58:22.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.533+0000 7f579144a740 1 --2- 172.21.15.97:0/2951606743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed4550e0 0x561bed449580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.533+0000 7f579144a740 1 --2- 172.21.15.97:0/2951606743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44ff70 0x561bed460910 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.533+0000 7f579144a740 1 --2- 172.21.15.97:0/2951606743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44f580 0x561bed464d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.533+0000 7f579144a740 1 -- 172.21.15.97:0/2951606743 >> 172.21.15.97:0/2951606743 conn(0x561bed44aaf0 msgr2=0x561bed44aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.533+0000 7f579144a740 1 -- 172.21.15.97:0/2951606743 shutdown_connections 2024-03-11T19:58:22.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.533+0000 7f579144a740 1 -- 172.21.15.97:0/2951606743 wait complete. 2024-03-11T19:58:22.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.533+0000 7f579144a740 1 -- start start 2024-03-11T19:58:22.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.533+0000 7f579144a740 1 Processor -- start 2024-03-11T19:58:22.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f579144a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 0x561bed4604d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f579144a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44ff70 0x561bed45cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f579144a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed4550e0 0x561bed45d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f579144a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed45dbb0 con 0x561bed44f580 2024-03-11T19:58:22.536 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f579144a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed460b00 con 0x561bed44ff70 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f579144a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed460cc0 con 0x561bed4550e0 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f579144a740 10 client.0 _ll_register_callbacks cb 0x7f577800be90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f5787400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 0x561bed4604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f5787400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 0x561bed4604d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38988/0 (socket says 172.21.15.97:38988) 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f5787400640 1 -- 172.21.15.97:0/1655133659 learned_addr learned my addr 172.21.15.97:0/1655133659 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f5786a00640 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44ff70 0x561bed45cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f5787e00640 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed4550e0 0x561bed45d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f5787400640 1 -- 172.21.15.97:0/1655133659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed4550e0 msgr2=0x561bed45d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f5787400640 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed4550e0 0x561bed45d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f5787400640 1 -- 172.21.15.97:0/1655133659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44ff70 msgr2=0x561bed45cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.534+0000 7f5787400640 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44ff70 0x561bed45cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.535+0000 7f5787400640 1 -- 172.21.15.97:0/1655133659 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57881a5e00 con 0x561bed44f580 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.535+0000 7f5786a00640 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44ff70 0x561bed45cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.535+0000 7f5787400640 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 0x561bed4604d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5778026000 tx=0x7f5778005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4fd03d16a1a67f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.535+0000 7f5771600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.535+0000 7f5771600640 1 -- 172.21.15.97:0/1655133659 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5778016040 con 0x561bed44f580 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.535+0000 7f579144a740 1 -- 172.21.15.97:0/1655133659 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561bed4664c0 con 0x561bed44f580 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.535+0000 7f5771600640 1 -- 172.21.15.97:0/1655133659 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f5778010e70 con 0x561bed44f580 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.535+0000 7f5771600640 1 -- 172.21.15.97:0/1655133659 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f577801bb70 con 0x561bed44f580 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.535+0000 7f579144a740 10 client.15120 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.535+0000 7f579144a740 1 -- 172.21.15.97:0/1655133659 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561bed378510 con 0x561bed44f580 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.536+0000 7f5771600640 1 -- 172.21.15.97:0/1655133659 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561bed4664c0 con 0x561bed44f580 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.536+0000 7f5771600640 1 -- 172.21.15.97:0/1655133659 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5778037ad0 con 0x561bed44f580 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.536+0000 7f579144a740 10 client.15120 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.536+0000 7f579144a740 10 client.15120 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.536+0000 7f579144a740 20 client.15120 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.536+0000 7f579144a740 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x561bed4c0960 0x561bed4bcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.536+0000 7f579144a740 4 client.15120 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:58:22.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.536+0000 7f579144a740 1 -- 172.21.15.97:0/1655133659 --> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f5780003970 con 0x561bed4c0960 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.536+0000 7f5786a00640 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x561bed4c0960 0x561bed4bcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f5786a00640 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x561bed4c0960 0x561bed4bcd80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2766b60cf2daf92 server_cookie=b05858402eb711d in_seq=0 out_seq=0 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f5771600640 10 client.15120 ms_handle_connect on v2:172.21.15.161:6838/565994162 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f5771600640 1 -- 172.21.15.97:0/1655133659 <== mds.2 v2:172.21.15.161:6838/565994162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f5780003970 con 0x561bed4c0960 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f5771600640 10 client.15120 handle_command_reply: tid=0 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f579144a740 1 client.15120 shutdown 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f579144a740 1 -- 172.21.15.97:0/1655133659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 msgr2=0x561bed4604d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f579144a740 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 0x561bed4604d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5778026000 tx=0x7f5778005b20 comp rx=0 tx=0).stop 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f579144a740 1 -- 172.21.15.97:0/1655133659 shutdown_connections 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f579144a740 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x561bed4c0960 0x561bed4bcd80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f579144a740 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x561bed4550e0 0x561bed45d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f579144a740 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bed44ff70 0x561bed45cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f5771600640 0 client.15120 ms_handle_reset on v2:172.21.15.161:6838/565994162 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f579144a740 1 --2- 172.21.15.97:0/1655133659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x561bed44f580 0x561bed4604d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f579144a740 1 -- 172.21.15.97:0/1655133659 >> 172.21.15.97:0/1655133659 conn(0x561bed44aaf0 msgr2=0x561bed44dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f579144a740 1 -- 172.21.15.97:0/1655133659 shutdown_connections 2024-03-11T19:58:22.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f579144a740 1 -- 172.21.15.97:0/1655133659 wait complete. 2024-03-11T19:58:22.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.537+0000 7f579144a740 20 client.15120 trim_cache size 0 max 16384 2024-03-11T19:58:22.540 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T19:58:22.540 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:58:22.540 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 2, 2024-03-11T19:58:22.540 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24547, 2024-03-11T19:58:22.540 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-11T19:58:22.540 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-11T19:58:22.540 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-11T19:58:22.540 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 125.892355028, 2024-03-11T19:58:22.541 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-03-11T19:58:22.541 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 93, 2024-03-11T19:58:22.541 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 89, 2024-03-11T19:58:22.541 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 174.66068103800001 2024-03-11T19:58:22.541 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T19:58:22.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.538+0000 7f578e800640 1 -- 172.21.15.97:0/748770169 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f576c0a1a30 msgr2=0x7f576c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.538+0000 7f578e800640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f576c0a1a30 0x7f576c0a3ef0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x561bed450370 tx=0x7f577800b390 comp rx=0 tx=0).stop 2024-03-11T19:58:22.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.538+0000 7f578e800640 1 -- 172.21.15.97:0/748770169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5788108990 msgr2=0x7f578819dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.538+0000 7f578e800640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5788108990 0x7f578819dd30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f577400e9f0 tx=0x7f577400eeb0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.538+0000 7f578e800640 1 -- 172.21.15.97:0/748770169 shutdown_connections 2024-03-11T19:58:22.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.538+0000 7f578e800640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5788109350 0x7f57881a4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.538+0000 7f578e800640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f576c0a1a30 0x7f576c0a3ef0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.538+0000 7f578e800640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5788108990 0x7f578819dd30 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.538+0000 7f578e800640 1 --2- 172.21.15.97:0/748770169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57881077a0 0x7f578819d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.538+0000 7f578e800640 1 -- 172.21.15.97:0/748770169 >> 172.21.15.97:0/748770169 conn(0x7f5788076740 msgr2=0x7f578810bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.538+0000 7f578e800640 1 -- 172.21.15.97:0/748770169 shutdown_connections 2024-03-11T19:58:22.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.538+0000 7f578e800640 1 -- 172.21.15.97:0/748770169 wait complete. 2024-03-11T19:58:22.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-11T19:58:22.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.757+0000 7f15ac000640 1 -- 172.21.15.97:0/3741551212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15a41033c0 msgr2=0x7f15a41037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.757+0000 7f15ac000640 1 --2- 172.21.15.97:0/3741551212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15a41033c0 0x7f15a41037e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1594009ca0 tx=0x7f159402c0f0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.757+0000 7f15ac000640 1 -- 172.21.15.97:0/3741551212 shutdown_connections 2024-03-11T19:58:22.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.757+0000 7f15ac000640 1 --2- 172.21.15.97:0/3741551212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f15a4104f70 0x7f15a410b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.757+0000 7f15ac000640 1 --2- 172.21.15.97:0/3741551212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15a41045b0 0x7f15a4104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.757+0000 7f15ac000640 1 --2- 172.21.15.97:0/3741551212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15a41033c0 0x7f15a41037e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.757+0000 7f15ac000640 1 -- 172.21.15.97:0/3741551212 >> 172.21.15.97:0/3741551212 conn(0x7f15a40fe930 msgr2=0x7f15a4100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.757+0000 7f15ac000640 1 -- 172.21.15.97:0/3741551212 shutdown_connections 2024-03-11T19:58:22.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.757+0000 7f15ac000640 1 -- 172.21.15.97:0/3741551212 wait complete. 2024-03-11T19:58:22.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.758+0000 7f15ac000640 1 Processor -- start 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.758+0000 7f15ac000640 1 -- start start 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15ac000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15a41033c0 0x7f15a4199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15ac000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15a41045b0 0x7f15a41999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15ac000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f15a4104f70 0x7f15a41a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15ac000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a41a0f60 con 0x7f15a4104f70 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15ac000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a41a10d0 con 0x7f15a41033c0 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15ac000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a41a1290 con 0x7f15a41045b0 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15aa000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f15a4104f70 0x7f15a41a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15a8c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15a41045b0 0x7f15a41999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15aa000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f15a4104f70 0x7f15a41a0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:38996/0 (socket says 172.21.15.97:38996) 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15aa000640 1 -- 172.21.15.97:0/988243298 learned_addr learned my addr 172.21.15.97:0/988243298 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15a9600640 1 --2- 172.21.15.97:0/988243298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15a41033c0 0x7f15a4199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15a8c00640 1 -- 172.21.15.97:0/988243298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15a41033c0 msgr2=0x7f15a4199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15a8c00640 1 --2- 172.21.15.97:0/988243298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15a41033c0 0x7f15a4199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15a8c00640 1 -- 172.21.15.97:0/988243298 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f15a4104f70 msgr2=0x7f15a41a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15a8c00640 1 --2- 172.21.15.97:0/988243298 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f15a4104f70 0x7f15a41a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.759+0000 7f15a8c00640 1 -- 172.21.15.97:0/988243298 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1590009490 con 0x7f15a41045b0 2024-03-11T19:58:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.760+0000 7f15a8c00640 1 --2- 172.21.15.97:0/988243298 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15a41045b0 0x7f15a41999b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f159000e9f0 tx=0x7f159000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.760+0000 7f15a2000640 1 -- 172.21.15.97:0/988243298 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159000cd80 con 0x7f15a41045b0 2024-03-11T19:58:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.760+0000 7f15a2000640 1 -- 172.21.15.97:0/988243298 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15900035f0 con 0x7f15a41045b0 2024-03-11T19:58:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.760+0000 7f15ac000640 1 -- 172.21.15.97:0/988243298 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1594009900 con 0x7f15a41045b0 2024-03-11T19:58:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.760+0000 7f15a2000640 1 -- 172.21.15.97:0/988243298 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590010690 con 0x7f15a41045b0 2024-03-11T19:58:22.760 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.760+0000 7f15ac000640 1 -- 172.21.15.97:0/988243298 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a41a18f0 con 0x7f15a41045b0 2024-03-11T19:58:22.764 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.763+0000 7f15aee2d740 1 Processor -- start 2024-03-11T19:58:22.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.763+0000 7f15aee2d740 1 -- start start 2024-03-11T19:58:22.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15aee2d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e9e580 0x5568b9e9e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15aee2d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9ea3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15aee2d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9ea40e0 0x5568b9ea6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15aee2d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b9e0db30 con 0x5568b9e9ef70 2024-03-11T19:58:22.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15aee2d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b9e0b060 con 0x5568b9e9e580 2024-03-11T19:58:22.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15aee2d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b9e06840 con 0x5568b9ea40e0 2024-03-11T19:58:22.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15a2000640 1 -- 172.21.15.97:0/988243298 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1590003af0 con 0x7f15a41045b0 2024-03-11T19:58:22.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15a9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9ea3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15a9600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9ea3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39010/0 (socket says 172.21.15.97:39010) 2024-03-11T19:58:22.765 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15a9600640 1 -- 172.21.15.97:0/2845708169 learned_addr learned my addr 172.21.15.97:0/2845708169 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15aa000640 1 --2- 172.21.15.97:0/2845708169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e9e580 0x5568b9e9e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15a8c00640 1 --2- 172.21.15.97:0/2845708169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9ea40e0 0x5568b9ea6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15a9600640 1 -- 172.21.15.97:0/2845708169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9ea40e0 msgr2=0x5568b9ea6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15a9600640 1 --2- 172.21.15.97:0/2845708169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9ea40e0 0x5568b9ea6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15a9600640 1 -- 172.21.15.97:0/2845708169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e9e580 msgr2=0x5568b9e9e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15a9600640 1 --2- 172.21.15.97:0/2845708169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e9e580 0x5568b9e9e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.764+0000 7f15a9600640 1 -- 172.21.15.97:0/2845708169 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568b9dce4c0 con 0x5568b9e9ef70 2024-03-11T19:58:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.765+0000 7f15aa000640 1 --2- 172.21.15.97:0/2845708169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e9e580 0x5568b9e9e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.765+0000 7f15a9600640 1 --2- 172.21.15.97:0/2845708169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9ea3a70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f159402c700 tx=0x7f1594005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55a8869a7f99569f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.765+0000 7f159a600640 1 -- 172.21.15.97:0/2845708169 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594034440 con 0x5568b9e9ef70 2024-03-11T19:58:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.765+0000 7f159a600640 1 -- 172.21.15.97:0/2845708169 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15940345a0 con 0x5568b9e9ef70 2024-03-11T19:58:22.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.765+0000 7f159a600640 1 -- 172.21.15.97:0/2845708169 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1594003bd0 con 0x5568b9e9ef70 2024-03-11T19:58:22.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.765+0000 7f15a2000640 1 --2- 172.21.15.97:0/988243298 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f15880a1a30 0x7f15880a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.766+0000 7f15a2000640 1 -- 172.21.15.97:0/988243298 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f1590014070 con 0x7f15a41045b0 2024-03-11T19:58:22.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.766+0000 7f15aa000640 1 --2- 172.21.15.97:0/988243298 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f15880a1a30 0x7f15880a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.766+0000 7f15aee2d740 1 -- 172.21.15.97:0/2845708169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 msgr2=0x5568b9ea3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.766+0000 7f15aee2d740 1 --2- 172.21.15.97:0/2845708169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9ea3a70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f159402c700 tx=0x7f1594005ef0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.766+0000 7f15aee2d740 1 -- 172.21.15.97:0/2845708169 shutdown_connections 2024-03-11T19:58:22.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.766+0000 7f15aee2d740 1 --2- 172.21.15.97:0/2845708169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9ea40e0 0x5568b9ea6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.766+0000 7f15aee2d740 1 --2- 172.21.15.97:0/2845708169 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9ea3a70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.766+0000 7f15aee2d740 1 --2- 172.21.15.97:0/2845708169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e9e580 0x5568b9e9e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.766+0000 7f15aee2d740 1 -- 172.21.15.97:0/2845708169 >> 172.21.15.97:0/2845708169 conn(0x5568b9e99af0 msgr2=0x5568b9e99ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.766+0000 7f15aee2d740 1 -- 172.21.15.97:0/2845708169 shutdown_connections 2024-03-11T19:58:22.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.766+0000 7f15aee2d740 1 -- 172.21.15.97:0/2845708169 wait complete. 2024-03-11T19:58:22.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.767+0000 7f15aa000640 1 --2- 172.21.15.97:0/988243298 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f15880a1a30 0x7f15880a3ef0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f159c009640 tx=0x7f159c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.767+0000 7f15aee2d740 1 -- start start 2024-03-11T19:58:22.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.767+0000 7f15aee2d740 1 Processor -- start 2024-03-11T19:58:22.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15aee2d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e9e580 0x5568b9ea8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15aee2d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9ea8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15aee2d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9ea40e0 0x5568b9eaf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15aee2d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b9dabea0 con 0x5568b9e9ef70 2024-03-11T19:58:22.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15aee2d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b9dabb40 con 0x5568b9ea40e0 2024-03-11T19:58:22.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15aee2d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b9b4ef20 con 0x5568b9e9e580 2024-03-11T19:58:22.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15aee2d740 10 client.0 _ll_register_callbacks cb 0x5568b9dbf950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:22.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15a8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9ea8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15a8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9ea8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39016/0 (socket says 172.21.15.97:39016) 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15a8c00640 1 -- 172.21.15.97:0/712553456 learned_addr learned my addr 172.21.15.97:0/712553456 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15a9600640 1 --2- 172.21.15.97:0/712553456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e9e580 0x5568b9ea8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15aa000640 1 --2- 172.21.15.97:0/712553456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9ea40e0 0x5568b9eaf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15a8c00640 1 -- 172.21.15.97:0/712553456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e9e580 msgr2=0x5568b9ea8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15a8c00640 1 --2- 172.21.15.97:0/712553456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e9e580 0x5568b9ea8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15a8c00640 1 -- 172.21.15.97:0/712553456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9ea40e0 msgr2=0x5568b9eaf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15a8c00640 1 --2- 172.21.15.97:0/712553456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9ea40e0 0x5568b9eaf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.768+0000 7f15a8c00640 1 -- 172.21.15.97:0/712553456 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a410c100 con 0x5568b9e9ef70 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.769+0000 7f15a8c00640 1 --2- 172.21.15.97:0/712553456 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9ea8880 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1590096b80 tx=0x7f159009a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2452fc64f7f16d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.769+0000 7f158ea00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.769+0000 7f158ea00640 1 -- 172.21.15.97:0/712553456 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159009a720 con 0x5568b9e9ef70 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.769+0000 7f158ea00640 1 -- 172.21.15.97:0/712553456 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f159009e870 con 0x5568b9e9ef70 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.769+0000 7f15aee2d740 1 -- 172.21.15.97:0/712553456 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5568b9e2ac40 con 0x5568b9e9ef70 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.769+0000 7f158ea00640 1 -- 172.21.15.97:0/712553456 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15900a2ec0 con 0x5568b9e9ef70 2024-03-11T19:58:22.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.769+0000 7f15aee2d740 10 client.15132 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.769+0000 7f15aee2d740 1 -- 172.21.15.97:0/712553456 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5568b9e04ec0 con 0x5568b9e9ef70 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.769+0000 7f158ea00640 1 -- 172.21.15.97:0/712553456 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5568b9e2ac40 con 0x5568b9e9ef70 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.769+0000 7f158ea00640 1 -- 172.21.15.97:0/712553456 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f15900ba070 con 0x5568b9e9ef70 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.770+0000 7f15aee2d740 10 client.15132 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.770+0000 7f15aee2d740 10 client.15132 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.770+0000 7f15aee2d740 20 client.15132 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.770+0000 7f15aee2d740 1 --2- 172.21.15.97:0/712553456 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x5568b9eb4940 0x5568b9eb3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.770+0000 7f15aee2d740 4 client.15132 mds_command: new command op to 24583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.770+0000 7f15aee2d740 1 -- 172.21.15.97:0/712553456 --> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5568b9b066c0 con 0x5568b9eb4940 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.770+0000 7f15a9600640 1 --2- 172.21.15.97:0/712553456 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x5568b9eb4940 0x5568b9eb3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.771+0000 7f158ea00640 10 client.15132 ms_handle_connect on v2:172.21.15.97:6840/2531110572 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.771+0000 7f15a9600640 1 --2- 172.21.15.97:0/712553456 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x5568b9eb4940 0x5568b9eb3c00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=611610665ac69b50 server_cookie=bee8ae59c07d4508 in_seq=0 out_seq=0 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f158ea00640 1 -- 172.21.15.97:0/712553456 <== mds.1 v2:172.21.15.97:6840/2531110572 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5568b9b066c0 con 0x5568b9eb4940 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f158ea00640 10 client.15132 handle_command_reply: tid=0 2024-03-11T19:58:22.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f15aee2d740 1 client.15132 shutdown 2024-03-11T19:58:22.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f15aee2d740 1 -- 172.21.15.97:0/712553456 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 msgr2=0x5568b9ea8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f15aee2d740 1 --2- 172.21.15.97:0/712553456 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9ea8880 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1590096b80 tx=0x7f159009a2e0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f15aee2d740 1 -- 172.21.15.97:0/712553456 shutdown_connections 2024-03-11T19:58:22.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f15aee2d740 1 --2- 172.21.15.97:0/712553456 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x5568b9eb4940 0x5568b9eb3c00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f15aee2d740 1 --2- 172.21.15.97:0/712553456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9ea40e0 0x5568b9eaf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f15aee2d740 1 --2- 172.21.15.97:0/712553456 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9ea8880 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f15aee2d740 1 --2- 172.21.15.97:0/712553456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e9e580 0x5568b9ea8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f15aee2d740 1 -- 172.21.15.97:0/712553456 >> 172.21.15.97:0/712553456 conn(0x5568b9e99af0 msgr2=0x5568b9e98e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f158ea00640 0 client.15132 ms_handle_reset on v2:172.21.15.97:6840/2531110572 2024-03-11T19:58:22.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f15aee2d740 1 -- 172.21.15.97:0/712553456 shutdown_connections 2024-03-11T19:58:22.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f15aee2d740 1 -- 172.21.15.97:0/712553456 wait complete. 2024-03-11T19:58:22.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.772+0000 7f15aee2d740 20 client.15132 trim_cache size 0 max 16384 2024-03-11T19:58:22.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.789+0000 7f15aee2d740 1 Processor -- start 2024-03-11T19:58:22.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.789+0000 7f15aee2d740 1 -- start start 2024-03-11T19:58:22.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.789+0000 7f15aee2d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e99af0 0x5568b9ebcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.789+0000 7f15aee2d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e9e580 0x5568b9eaf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.789+0000 7f15aee2d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9e98580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.789+0000 7f15aee2d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b9eb6920 con 0x5568b9e9ef70 2024-03-11T19:58:22.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.789+0000 7f15aee2d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b9e9aae0 con 0x5568b9e99af0 2024-03-11T19:58:22.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.789+0000 7f15aee2d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b9e2ac40 con 0x5568b9e9e580 2024-03-11T19:58:22.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.790+0000 7f15aa000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9e98580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.790+0000 7f15aa000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9e98580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39028/0 (socket says 172.21.15.97:39028) 2024-03-11T19:58:22.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.790+0000 7f15aa000640 1 -- 172.21.15.97:0/313556688 learned_addr learned my addr 172.21.15.97:0/313556688 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.790+0000 7f15a9600640 1 --2- 172.21.15.97:0/313556688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e99af0 0x5568b9ebcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.790+0000 7f15a8c00640 1 --2- 172.21.15.97:0/313556688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e9e580 0x5568b9eaf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.790+0000 7f15aa000640 1 -- 172.21.15.97:0/313556688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e9e580 msgr2=0x5568b9eaf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.790+0000 7f15aa000640 1 --2- 172.21.15.97:0/313556688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e9e580 0x5568b9eaf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.790+0000 7f15aa000640 1 -- 172.21.15.97:0/313556688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e99af0 msgr2=0x5568b9ebcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.790+0000 7f15aa000640 1 --2- 172.21.15.97:0/313556688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e99af0 0x5568b9ebcde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.790+0000 7f15aa000640 1 -- 172.21.15.97:0/313556688 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568b9da18b0 con 0x5568b9e9ef70 2024-03-11T19:58:22.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.790+0000 7f15a9600640 1 --2- 172.21.15.97:0/313556688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e99af0 0x5568b9ebcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:22.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.790+0000 7f15a8c00640 1 --2- 172.21.15.97:0/313556688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e9e580 0x5568b9eaf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:22.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.790+0000 7f15aa000640 1 --2- 172.21.15.97:0/313556688 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9e98580 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f159c01d760 tx=0x7f159c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf3f938a111ba0c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.791+0000 7f1599c00640 1 -- 172.21.15.97:0/313556688 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c023050 con 0x5568b9e9ef70 2024-03-11T19:58:22.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.791+0000 7f1599c00640 1 -- 172.21.15.97:0/313556688 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f159c024040 con 0x5568b9e9ef70 2024-03-11T19:58:22.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.791+0000 7f1599c00640 1 -- 172.21.15.97:0/313556688 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c017070 con 0x5568b9e9ef70 2024-03-11T19:58:22.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.791+0000 7f15aee2d740 1 -- 172.21.15.97:0/313556688 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 msgr2=0x5568b9e98580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.791+0000 7f15aee2d740 1 --2- 172.21.15.97:0/313556688 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9e98580 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f159c01d760 tx=0x7f159c009f90 comp rx=0 tx=0).stop 2024-03-11T19:58:22.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.792+0000 7f15aee2d740 1 -- 172.21.15.97:0/313556688 shutdown_connections 2024-03-11T19:58:22.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.792+0000 7f15aee2d740 1 --2- 172.21.15.97:0/313556688 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9ef70 0x5568b9e98580 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.792+0000 7f15aee2d740 1 --2- 172.21.15.97:0/313556688 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e9e580 0x5568b9eaf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.792+0000 7f15aee2d740 1 --2- 172.21.15.97:0/313556688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e99af0 0x5568b9ebcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.792+0000 7f15aee2d740 1 -- 172.21.15.97:0/313556688 >> 172.21.15.97:0/313556688 conn(0x5568b9e9a240 msgr2=0x5568b9ea3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.792+0000 7f15aee2d740 1 -- 172.21.15.97:0/313556688 shutdown_connections 2024-03-11T19:58:22.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.792+0000 7f15aee2d740 1 -- 172.21.15.97:0/313556688 wait complete. 2024-03-11T19:58:22.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.792+0000 7f15aee2d740 1 -- start start 2024-03-11T19:58:22.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.792+0000 7f15aee2d740 1 Processor -- start 2024-03-11T19:58:22.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15aee2d740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e99af0 0x5568b9eac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15aee2d740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9e580 0x5568b9eacee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15aee2d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e9ef70 0x5568b9eadef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15aee2d740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b9eae430 con 0x5568b9e9e580 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15aee2d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b9eae5a0 con 0x5568b9e9ef70 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15aee2d740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568b9eae760 con 0x5568b9e99af0 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15aee2d740 10 client.0 _ll_register_callbacks cb 0x7f159000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15a8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9e580 0x5568b9eacee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15a8c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9e580 0x5568b9eacee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39042/0 (socket says 172.21.15.97:39042) 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15a8c00640 1 -- 172.21.15.97:0/3761182029 learned_addr learned my addr 172.21.15.97:0/3761182029 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15a9600640 1 --2- 172.21.15.97:0/3761182029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e99af0 0x5568b9eac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15aa000640 1 --2- 172.21.15.97:0/3761182029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e9ef70 0x5568b9eadef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15a8c00640 1 -- 172.21.15.97:0/3761182029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e99af0 msgr2=0x5568b9eac9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15a8c00640 1 --2- 172.21.15.97:0/3761182029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e99af0 0x5568b9eac9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15a8c00640 1 -- 172.21.15.97:0/3761182029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e9ef70 msgr2=0x5568b9eadef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15a8c00640 1 --2- 172.21.15.97:0/3761182029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e9ef70 0x5568b9eadef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.793+0000 7f15a8c00640 1 -- 172.21.15.97:0/3761182029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159c01d410 con 0x5568b9e9e580 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.794+0000 7f15a8c00640 1 --2- 172.21.15.97:0/3761182029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9e580 0x5568b9eacee0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1590009460 tx=0x7f159000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d80218ad8359c28 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.794+0000 7f158ea00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.794+0000 7f158ea00640 1 -- 172.21.15.97:0/3761182029 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15900d30d0 con 0x5568b9e9e580 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.794+0000 7f158ea00640 1 -- 172.21.15.97:0/3761182029 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f159009e870 con 0x5568b9e9e580 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.794+0000 7f15aee2d740 1 -- 172.21.15.97:0/3761182029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5568b9eb5970 con 0x5568b9e9e580 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.794+0000 7f158ea00640 1 -- 172.21.15.97:0/3761182029 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f159009a080 con 0x5568b9e9e580 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.794+0000 7f15aee2d740 10 client.15138 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.794+0000 7f15aee2d740 1 -- 172.21.15.97:0/3761182029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5568b9dc7510 con 0x5568b9e9e580 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.794+0000 7f158ea00640 1 -- 172.21.15.97:0/3761182029 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5568b9eb5970 con 0x5568b9e9e580 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.794+0000 7f158ea00640 1 -- 172.21.15.97:0/3761182029 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1590099b70 con 0x5568b9e9e580 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.795+0000 7f15aee2d740 10 client.15138 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.795+0000 7f15aee2d740 10 client.15138 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.795+0000 7f15aee2d740 20 client.15138 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.795+0000 7f15aee2d740 1 --2- 172.21.15.97:0/3761182029 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x5568b9ea40e0 0x5568b9f0c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.795+0000 7f15aee2d740 4 client.15138 mds_command: new command op to 24583 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.795+0000 7f15aee2d740 1 -- 172.21.15.97:0/3761182029 --> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f15940345a0 con 0x5568b9ea40e0 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.795+0000 7f15a9600640 1 --2- 172.21.15.97:0/3761182029 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x5568b9ea40e0 0x5568b9f0c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.795+0000 7f15a9600640 1 --2- 172.21.15.97:0/3761182029 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x5568b9ea40e0 0x5568b9f0c550 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c8ead52ebefbc985 server_cookie=ddd56ae455674796 in_seq=0 out_seq=0 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.795+0000 7f158ea00640 10 client.15138 ms_handle_connect on v2:172.21.15.97:6840/2531110572 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f158ea00640 1 -- 172.21.15.97:0/3761182029 <== mds.1 v2:172.21.15.97:6840/2531110572 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x7f15940345a0 con 0x5568b9ea40e0 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f158ea00640 10 client.15138 handle_command_reply: tid=0 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f15aee2d740 1 client.15138 shutdown 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f15aee2d740 1 -- 172.21.15.97:0/3761182029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9e580 msgr2=0x5568b9eacee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f15aee2d740 1 --2- 172.21.15.97:0/3761182029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9e580 0x5568b9eacee0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1590009460 tx=0x7f159000c840 comp rx=0 tx=0).stop 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f15aee2d740 1 -- 172.21.15.97:0/3761182029 shutdown_connections 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f15aee2d740 1 --2- 172.21.15.97:0/3761182029 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x5568b9ea40e0 0x5568b9f0c550 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f15aee2d740 1 --2- 172.21.15.97:0/3761182029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5568b9e9ef70 0x5568b9eadef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f15aee2d740 1 --2- 172.21.15.97:0/3761182029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5568b9e9e580 0x5568b9eacee0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f15aee2d740 1 --2- 172.21.15.97:0/3761182029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568b9e99af0 0x5568b9eac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f158ea00640 0 client.15138 ms_handle_reset on v2:172.21.15.97:6840/2531110572 2024-03-11T19:58:22.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f15aee2d740 1 -- 172.21.15.97:0/3761182029 >> 172.21.15.97:0/3761182029 conn(0x5568b9e9a240 msgr2=0x5568b9ea3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f15aee2d740 1 -- 172.21.15.97:0/3761182029 shutdown_connections 2024-03-11T19:58:22.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f15aee2d740 1 -- 172.21.15.97:0/3761182029 wait complete. 2024-03-11T19:58:22.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.796+0000 7f15aee2d740 20 client.15138 trim_cache size 0 max 16384 2024-03-11T19:58:22.799 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T19:58:22.799 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:58:22.799 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 1, 2024-03-11T19:58:22.799 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24583, 2024-03-11T19:58:22.799 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-11T19:58:22.800 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-11T19:58:22.800 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-11T19:58:22.800 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 127.16375517900001, 2024-03-11T19:58:22.800 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-03-11T19:58:22.800 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 93, 2024-03-11T19:58:22.800 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 89, 2024-03-11T19:58:22.800 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 166.7169878 2024-03-11T19:58:22.800 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T19:58:22.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.797+0000 7f15ac000640 1 -- 172.21.15.97:0/988243298 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f15880a1a30 msgr2=0x7f15880a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.797+0000 7f15ac000640 1 --2- 172.21.15.97:0/988243298 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f15880a1a30 0x7f15880a3ef0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f159c009640 tx=0x7f159c005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:22.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.797+0000 7f15ac000640 1 -- 172.21.15.97:0/988243298 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15a41045b0 msgr2=0x7f15a41999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:22.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.797+0000 7f15ac000640 1 --2- 172.21.15.97:0/988243298 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15a41045b0 0x7f15a41999b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f159000e9f0 tx=0x7f159000eeb0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.797+0000 7f15ac000640 1 -- 172.21.15.97:0/988243298 shutdown_connections 2024-03-11T19:58:22.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.797+0000 7f15ac000640 1 --2- 172.21.15.97:0/988243298 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f15880a1a30 0x7f15880a3ef0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.797+0000 7f15ac000640 1 --2- 172.21.15.97:0/988243298 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f15a4104f70 0x7f15a41a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.797+0000 7f15ac000640 1 --2- 172.21.15.97:0/988243298 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f15a41045b0 0x7f15a41999b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.797+0000 7f15ac000640 1 --2- 172.21.15.97:0/988243298 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15a41033c0 0x7f15a4199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:22.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.797+0000 7f15ac000640 1 -- 172.21.15.97:0/988243298 >> 172.21.15.97:0/988243298 conn(0x7f15a40fe930 msgr2=0x7f15a41078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:22.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.797+0000 7f15ac000640 1 -- 172.21.15.97:0/988243298 shutdown_connections 2024-03-11T19:58:22.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:22.797+0000 7f15ac000640 1 -- 172.21.15.97:0/988243298 wait complete. 2024-03-11T19:58:22.810 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:22 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/4008055119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:22.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-11T19:58:22.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:22 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/4008055119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:23.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.016+0000 7fd9d4000640 1 -- 172.21.15.97:0/2667123848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9cc104d10 msgr2=0x7fd9cc105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:23.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.016+0000 7fd9d4000640 1 --2- 172.21.15.97:0/2667123848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9cc104d10 0x7fd9cc105190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc009860 tx=0x7fd9bc02bb70 comp rx=0 tx=0).stop 2024-03-11T19:58:23.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.016+0000 7fd9d4000640 1 -- 172.21.15.97:0/2667123848 shutdown_connections 2024-03-11T19:58:23.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.016+0000 7fd9d4000640 1 --2- 172.21.15.97:0/2667123848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9cc104d10 0x7fd9cc105190 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.016+0000 7fd9d4000640 1 --2- 172.21.15.97:0/2667123848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9cc102360 0x7fd9cc1047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.016+0000 7fd9d4000640 1 --2- 172.21.15.97:0/2667123848 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9cc0ff9f0 0x7fd9cc101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.016+0000 7fd9d4000640 1 -- 172.21.15.97:0/2667123848 >> 172.21.15.97:0/2667123848 conn(0x7fd9cc0fb610 msgr2=0x7fd9cc0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:23.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.016+0000 7fd9d4000640 1 -- 172.21.15.97:0/2667123848 shutdown_connections 2024-03-11T19:58:23.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.016+0000 7fd9d4000640 1 -- 172.21.15.97:0/2667123848 wait complete. 2024-03-11T19:58:23.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.017+0000 7fd9d4000640 1 Processor -- start 2024-03-11T19:58:23.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.017+0000 7fd9d4000640 1 -- start start 2024-03-11T19:58:23.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d4000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9cc0ff9f0 0x7fd9cc19d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d4000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9cc102360 0x7fd9cc19dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d4000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9cc104d10 0x7fd9cc1a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d4000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc1a52b0 con 0x7fd9cc102360 2024-03-11T19:58:23.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d4000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc1a5420 con 0x7fd9cc104d10 2024-03-11T19:58:23.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d4000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc1a55e0 con 0x7fd9cc0ff9f0 2024-03-11T19:58:23.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9cc102360 0x7fd9cc19dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9cc102360 0x7fd9cc19dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39060/0 (socket says 172.21.15.97:39060) 2024-03-11T19:58:23.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d0c00640 1 -- 172.21.15.97:0/2652745450 learned_addr learned my addr 172.21.15.97:0/2652745450 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:23.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d1600640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9cc0ff9f0 0x7fd9cc19d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d2000640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9cc104d10 0x7fd9cc1a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d0c00640 1 -- 172.21.15.97:0/2652745450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9cc0ff9f0 msgr2=0x7fd9cc19d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:23.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d0c00640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9cc0ff9f0 0x7fd9cc19d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d0c00640 1 -- 172.21.15.97:0/2652745450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9cc104d10 msgr2=0x7fd9cc1a4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:23.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d0c00640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9cc104d10 0x7fd9cc1a4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.018+0000 7fd9d0c00640 1 -- 172.21.15.97:0/2652745450 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9bc009490 con 0x7fd9cc102360 2024-03-11T19:58:23.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.019+0000 7fd9d2000640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9cc104d10 0x7fd9cc1a4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:23.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.019+0000 7fd9d0c00640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9cc102360 0x7fd9cc19dd00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800efb0 tx=0x7fd9b800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:23.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.019+0000 7fd9ca000640 1 -- 172.21.15.97:0/2652745450 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b800b570 con 0x7fd9cc102360 2024-03-11T19:58:23.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.019+0000 7fd9ca000640 1 -- 172.21.15.97:0/2652745450 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9b800bbb0 con 0x7fd9cc102360 2024-03-11T19:58:23.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.019+0000 7fd9d4000640 1 -- 172.21.15.97:0/2652745450 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9cc1a58c0 con 0x7fd9cc102360 2024-03-11T19:58:23.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.019+0000 7fd9ca000640 1 -- 172.21.15.97:0/2652745450 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8016d20 con 0x7fd9cc102360 2024-03-11T19:58:23.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.019+0000 7fd9d4000640 1 -- 172.21.15.97:0/2652745450 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9cc1a5dd0 con 0x7fd9cc102360 2024-03-11T19:58:23.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.022+0000 7fd9d6c32740 1 Processor -- start 2024-03-11T19:58:23.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.022+0000 7fd9d6c32740 1 -- start start 2024-03-11T19:58:23.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.023+0000 7fd9ca000640 1 -- 172.21.15.97:0/2652745450 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd9b800a040 con 0x7fd9cc102360 2024-03-11T19:58:23.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.023+0000 7fd9d6c32740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa3596580 0x55caa35969a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.023+0000 7fd9d6c32740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa359ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.023+0000 7fd9d6c32740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa359c0e0 0x55caa359e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.023+0000 7fd9d6c32740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa3505b30 con 0x55caa359c0e0 2024-03-11T19:58:23.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.023+0000 7fd9d6c32740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa3503060 con 0x55caa3596f70 2024-03-11T19:58:23.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.023+0000 7fd9d6c32740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa34fe840 con 0x55caa3596580 2024-03-11T19:58:23.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.023+0000 7fd9d0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa359c0e0 0x55caa359e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.023+0000 7fd9d1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa359ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.023+0000 7fd9d0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa359c0e0 0x55caa359e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39066/0 (socket says 172.21.15.97:39066) 2024-03-11T19:58:23.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.023+0000 7fd9d2000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa3596580 0x55caa35969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.023+0000 7fd9d0c00640 1 -- 172.21.15.97:0/262475735 learned_addr learned my addr 172.21.15.97:0/262475735 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.024+0000 7fd9d1600640 1 -- 172.21.15.97:0/262475735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa3596580 msgr2=0x55caa35969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.024+0000 7fd9d1600640 1 --2- 172.21.15.97:0/262475735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa3596580 0x55caa35969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.024+0000 7fd9d1600640 1 -- 172.21.15.97:0/262475735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa359c0e0 msgr2=0x55caa359e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.024+0000 7fd9d1600640 1 --2- 172.21.15.97:0/262475735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa359c0e0 0x55caa359e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.024+0000 7fd9d1600640 1 -- 172.21.15.97:0/262475735 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55caa34c64c0 con 0x55caa3596f70 2024-03-11T19:58:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.024+0000 7fd9d1600640 1 --2- 172.21.15.97:0/262475735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa359ba70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4009c20 tx=0x7fd9c4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa1d082f07972645 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.024+0000 7fd9c2600640 1 -- 172.21.15.97:0/262475735 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4004050 con 0x55caa3596f70 2024-03-11T19:58:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.024+0000 7fd9ca000640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd9b00a1a30 0x7fd9b00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.024+0000 7fd9c2600640 1 -- 172.21.15.97:0/262475735 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9c4005500 con 0x55caa3596f70 2024-03-11T19:58:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.024+0000 7fd9c2600640 1 -- 172.21.15.97:0/262475735 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4004d00 con 0x55caa3596f70 2024-03-11T19:58:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.024+0000 7fd9ca000640 1 -- 172.21.15.97:0/2652745450 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fd9b80cd810 con 0x7fd9cc102360 2024-03-11T19:58:23.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.025+0000 7fd9d2000640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd9b00a1a30 0x7fd9b00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.025+0000 7fd9d6c32740 1 -- 172.21.15.97:0/262475735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 msgr2=0x55caa359ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.025+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/262475735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa359ba70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4009c20 tx=0x7fd9c4011810 comp rx=0 tx=0).stop 2024-03-11T19:58:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.025+0000 7fd9d6c32740 1 -- 172.21.15.97:0/262475735 shutdown_connections 2024-03-11T19:58:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.025+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/262475735 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa359c0e0 0x55caa359e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.025+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/262475735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa359ba70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.025+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/262475735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa3596580 0x55caa35969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.025+0000 7fd9d6c32740 1 -- 172.21.15.97:0/262475735 >> 172.21.15.97:0/262475735 conn(0x55caa3591af0 msgr2=0x55caa3591ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.025+0000 7fd9d2000640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd9b00a1a30 0x7fd9b00a3ef0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc02c180 tx=0x7fd9bc005ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.025+0000 7fd9d6c32740 1 -- 172.21.15.97:0/262475735 shutdown_connections 2024-03-11T19:58:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.025+0000 7fd9d6c32740 1 -- 172.21.15.97:0/262475735 wait complete. 2024-03-11T19:58:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.026+0000 7fd9d6c32740 1 -- start start 2024-03-11T19:58:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.026+0000 7fd9d6c32740 1 Processor -- start 2024-03-11T19:58:23.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.026+0000 7fd9d6c32740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596580 0x55caa35a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.026+0000 7fd9d6c32740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa35a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.026+0000 7fd9d6c32740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa35a7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.026+0000 7fd9d6c32740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa34a3ea0 con 0x55caa3596580 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.026+0000 7fd9d6c32740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa34a3b40 con 0x55caa3596f70 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.026+0000 7fd9d6c32740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa3246f20 con 0x55caa359c0e0 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.026+0000 7fd9d6c32740 10 client.0 _ll_register_callbacks cb 0x55caa34b7950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.027+0000 7fd9d1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596580 0x55caa35a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.027+0000 7fd9d1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596580 0x55caa35a03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39068/0 (socket says 172.21.15.97:39068) 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.027+0000 7fd9d1600640 1 -- 172.21.15.97:0/51144270 learned_addr learned my addr 172.21.15.97:0/51144270 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.027+0000 7fd9d2000640 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa35a7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.027+0000 7fd9d0c00640 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa35a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.027+0000 7fd9d1600640 1 -- 172.21.15.97:0/51144270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 msgr2=0x55caa35a7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.027+0000 7fd9d1600640 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa35a7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.027+0000 7fd9d1600640 1 -- 172.21.15.97:0/51144270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 msgr2=0x55caa35a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:23.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.027+0000 7fd9d1600640 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa35a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.027+0000 7fd9d1600640 1 -- 172.21.15.97:0/51144270 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c4009850 con 0x55caa3596580 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.027+0000 7fd9d0c00640 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa35a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.027+0000 7fd9d1600640 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596580 0x55caa35a03a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4000c00 tx=0x7fd9c4005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1b4bbe63a8b9c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.027+0000 7fd9b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.028+0000 7fd9b6a00640 1 -- 172.21.15.97:0/51144270 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4003a00 con 0x55caa3596580 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.028+0000 7fd9b6a00640 1 -- 172.21.15.97:0/51144270 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fd9c4005c50 con 0x55caa3596580 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.028+0000 7fd9d6c32740 1 -- 172.21.15.97:0/51144270 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55caa3522c40 con 0x55caa3596580 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.028+0000 7fd9b6a00640 1 -- 172.21.15.97:0/51144270 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9c4003b60 con 0x55caa3596580 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.028+0000 7fd9d6c32740 10 client.15150 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.028+0000 7fd9d6c32740 1 -- 172.21.15.97:0/51144270 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55caa34fcec0 con 0x55caa3596580 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.028+0000 7fd9b6a00640 1 -- 172.21.15.97:0/51144270 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55caa3522c40 con 0x55caa3596580 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.029+0000 7fd9b6a00640 1 -- 172.21.15.97:0/51144270 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd9c4021040 con 0x55caa3596580 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.029+0000 7fd9d6c32740 10 client.15150 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:23.032 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.029+0000 7fd9d6c32740 10 client.15150 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T19:58:23.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.029+0000 7fd9d6c32740 20 client.15150 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:23.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.029+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55caa35ac890 0x55caa35abb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.029+0000 7fd9d6c32740 4 client.15150 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:58:23.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.029+0000 7fd9d6c32740 1 -- 172.21.15.97:0/51144270 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55caa31fe6c0 con 0x55caa35ac890 2024-03-11T19:58:23.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.029+0000 7fd9d0c00640 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55caa35ac890 0x55caa35abb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.030+0000 7fd9d0c00640 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55caa35ac890 0x55caa35abb40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0c8e95ca98130e3 server_cookie=928bf7b47c39598a in_seq=0 out_seq=0 2024-03-11T19:58:23.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.030+0000 7fd9b6a00640 10 client.15150 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T19:58:23.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.030+0000 7fd9b6a00640 1 -- 172.21.15.97:0/51144270 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55caa31fe6c0 con 0x55caa35ac890 2024-03-11T19:58:23.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.030+0000 7fd9b6a00640 10 client.15150 handle_command_reply: tid=0 2024-03-11T19:58:23.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.031+0000 7fd9d6c32740 1 client.15150 shutdown 2024-03-11T19:58:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.031+0000 7fd9d6c32740 1 -- 172.21.15.97:0/51144270 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596580 msgr2=0x55caa35a03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.031+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596580 0x55caa35a03a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4000c00 tx=0x7fd9c4005730 comp rx=0 tx=0).stop 2024-03-11T19:58:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.031+0000 7fd9d6c32740 1 -- 172.21.15.97:0/51144270 shutdown_connections 2024-03-11T19:58:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.031+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa35a7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.031+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa35a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.031+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55caa35ac890 0x55caa35abb40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.031+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/51144270 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596580 0x55caa35a03a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.031+0000 7fd9b6a00640 0 client.15150 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T19:58:23.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.031+0000 7fd9d6c32740 1 -- 172.21.15.97:0/51144270 >> 172.21.15.97:0/51144270 conn(0x55caa3591af0 msgr2=0x55caa3590e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:23.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.031+0000 7fd9d6c32740 1 -- 172.21.15.97:0/51144270 shutdown_connections 2024-03-11T19:58:23.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.031+0000 7fd9d6c32740 1 -- 172.21.15.97:0/51144270 wait complete. 2024-03-11T19:58:23.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.031+0000 7fd9d6c32740 20 client.15150 trim_cache size 0 max 16384 2024-03-11T19:58:23.036 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:22 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/4008055119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.048+0000 7fd9d6c32740 1 Processor -- start 2024-03-11T19:58:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.048+0000 7fd9d6c32740 1 -- start start 2024-03-11T19:58:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d6c32740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596580 0x55caa35b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d6c32740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa35a78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d6c32740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa3590580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d6c32740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa359f850 con 0x55caa3596580 2024-03-11T19:58:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d6c32740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa35aa7c0 con 0x55caa3596f70 2024-03-11T19:58:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d6c32740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa35abd70 con 0x55caa359c0e0 2024-03-11T19:58:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596580 0x55caa35b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d2000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa3590580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d2000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa3590580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:39500/0 (socket says 172.21.15.97:39500) 2024-03-11T19:58:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d2000640 1 -- 172.21.15.97:0/3808617876 learned_addr learned my addr 172.21.15.97:0/3808617876 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d0c00640 1 --2- 172.21.15.97:0/3808617876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa35a78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d2000640 1 -- 172.21.15.97:0/3808617876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 msgr2=0x55caa35a78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d2000640 1 --2- 172.21.15.97:0/3808617876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa35a78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d2000640 1 -- 172.21.15.97:0/3808617876 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596580 msgr2=0x55caa35b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d2000640 1 --2- 172.21.15.97:0/3808617876 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596580 0x55caa35b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.049+0000 7fd9d2000640 1 -- 172.21.15.97:0/3808617876 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55caa34998b0 con 0x55caa359c0e0 2024-03-11T19:58:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.050+0000 7fd9d0c00640 1 --2- 172.21.15.97:0/3808617876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa35a78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.050+0000 7fd9d1600640 1 --2- 172.21.15.97:0/3808617876 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596580 0x55caa35b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.050+0000 7fd9d2000640 1 --2- 172.21.15.97:0/3808617876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa3590580 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc034b10 tx=0x7fd9bc03f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f411447363082d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.050+0000 7fd9c1c00640 1 -- 172.21.15.97:0/3808617876 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc046040 con 0x55caa359c0e0 2024-03-11T19:58:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.050+0000 7fd9c1c00640 1 -- 172.21.15.97:0/3808617876 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9bc0348c0 con 0x55caa359c0e0 2024-03-11T19:58:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.050+0000 7fd9c1c00640 1 -- 172.21.15.97:0/3808617876 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc0046c0 con 0x55caa359c0e0 2024-03-11T19:58:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.051+0000 7fd9d6c32740 1 -- 172.21.15.97:0/3808617876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 msgr2=0x55caa3590580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.051+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/3808617876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa3590580 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc034b10 tx=0x7fd9bc03f000 comp rx=0 tx=0).stop 2024-03-11T19:58:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.051+0000 7fd9d6c32740 1 -- 172.21.15.97:0/3808617876 shutdown_connections 2024-03-11T19:58:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.051+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/3808617876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa3590580 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.051+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/3808617876 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596f70 0x55caa35a78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.051+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/3808617876 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596580 0x55caa35b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.051+0000 7fd9d6c32740 1 -- 172.21.15.97:0/3808617876 >> 172.21.15.97:0/3808617876 conn(0x55caa3591af0 msgr2=0x55caa3591ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.051+0000 7fd9d6c32740 1 -- 172.21.15.97:0/3808617876 shutdown_connections 2024-03-11T19:58:23.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.051+0000 7fd9d6c32740 1 -- 172.21.15.97:0/3808617876 wait complete. 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.051+0000 7fd9d6c32740 1 -- start start 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.052+0000 7fd9d6c32740 1 Processor -- start 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.052+0000 7fd9d6c32740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596580 0x55caa35a7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.052+0000 7fd9d6c32740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596f70 0x55caa35a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.052+0000 7fd9d6c32740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa35a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.052+0000 7fd9d6c32740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa3592a90 con 0x55caa3596f70 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.052+0000 7fd9d6c32740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa35a4ce0 con 0x55caa3596580 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.052+0000 7fd9d6c32740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55caa3602490 con 0x55caa359c0e0 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.052+0000 7fd9d6c32740 10 client.0 _ll_register_callbacks cb 0x7fd9b80d3400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.052+0000 7fd9d0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596f70 0x55caa35a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.052+0000 7fd9d0c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596f70 0x55caa35a41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:39088/0 (socket says 172.21.15.97:39088) 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.052+0000 7fd9d0c00640 1 -- 172.21.15.97:0/4249147029 learned_addr learned my addr 172.21.15.97:0/4249147029 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.052+0000 7fd9d2000640 1 --2- 172.21.15.97:0/4249147029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa35a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.052+0000 7fd9d1600640 1 --2- 172.21.15.97:0/4249147029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596580 0x55caa35a7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.053+0000 7fd9d0c00640 1 -- 172.21.15.97:0/4249147029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 msgr2=0x55caa35a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.053+0000 7fd9d0c00640 1 --2- 172.21.15.97:0/4249147029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa35a46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.053+0000 7fd9d0c00640 1 -- 172.21.15.97:0/4249147029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596580 msgr2=0x55caa35a7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.053+0000 7fd9d0c00640 1 --2- 172.21.15.97:0/4249147029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596580 0x55caa35a7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.053+0000 7fd9d0c00640 1 -- 172.21.15.97:0/4249147029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55caa34fcec0 con 0x55caa3596f70 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.053+0000 7fd9d0c00640 1 --2- 172.21.15.97:0/4249147029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596f70 0x55caa35a41b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800ef80 tx=0x7fd9b8096040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b37c7cea98041f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.053+0000 7fd9b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.053+0000 7fd9b6a00640 1 -- 172.21.15.97:0/4249147029 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b809e4b0 con 0x55caa3596f70 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.053+0000 7fd9b6a00640 1 -- 172.21.15.97:0/4249147029 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fd9b80a2b00 con 0x55caa3596f70 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.053+0000 7fd9d6c32740 1 -- 172.21.15.97:0/4249147029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9c4005500 con 0x55caa3596f70 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.053+0000 7fd9b6a00640 1 -- 172.21.15.97:0/4249147029 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9b80b5020 con 0x55caa3596f70 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.053+0000 7fd9d6c32740 10 client.15156 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.053+0000 7fd9d6c32740 1 -- 172.21.15.97:0/4249147029 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fd9cc105c80 con 0x55caa3596f70 2024-03-11T19:58:23.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.054+0000 7fd9b6a00640 1 -- 172.21.15.97:0/4249147029 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9c4005500 con 0x55caa3596f70 2024-03-11T19:58:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.054+0000 7fd9b6a00640 1 -- 172.21.15.97:0/4249147029 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd9b80a2670 con 0x55caa3596f70 2024-03-11T19:58:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.054+0000 7fd9d6c32740 10 client.15156 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.054+0000 7fd9d6c32740 10 client.15156 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T19:58:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.054+0000 7fd9d6c32740 20 client.15156 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.054+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/4249147029 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55caa3607730 0x55caa3603bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.054+0000 7fd9d6c32740 4 client.15156 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:58:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.054+0000 7fd9d6c32740 1 -- 172.21.15.97:0/4249147029 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55caa324f270 con 0x55caa3607730 2024-03-11T19:58:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.054+0000 7fd9d1600640 1 --2- 172.21.15.97:0/4249147029 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55caa3607730 0x55caa3603bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9d1600640 1 --2- 172.21.15.97:0/4249147029 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55caa3607730 0x55caa3603bd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f22c789cc29ad7b server_cookie=4657dec14c64758a in_seq=0 out_seq=0 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9b6a00640 10 client.15156 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9b6a00640 1 -- 172.21.15.97:0/4249147029 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55caa324f270 con 0x55caa3607730 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9b6a00640 10 client.15156 handle_command_reply: tid=0 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9d6c32740 1 client.15156 shutdown 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9d6c32740 1 -- 172.21.15.97:0/4249147029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596f70 msgr2=0x55caa35a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/4249147029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596f70 0x55caa35a41b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800ef80 tx=0x7fd9b8096040 comp rx=0 tx=0).stop 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9d6c32740 1 -- 172.21.15.97:0/4249147029 shutdown_connections 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/4249147029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55caa359c0e0 0x55caa35a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/4249147029 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55caa3607730 0x55caa3603bd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/4249147029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55caa3596f70 0x55caa35a41b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9d6c32740 1 --2- 172.21.15.97:0/4249147029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55caa3596580 0x55caa35a7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9b6a00640 0 client.15156 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9d6c32740 1 -- 172.21.15.97:0/4249147029 >> 172.21.15.97:0/4249147029 conn(0x55caa3591af0 msgr2=0x55caa35b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:23.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9d6c32740 1 -- 172.21.15.97:0/4249147029 shutdown_connections 2024-03-11T19:58:23.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9d6c32740 1 -- 172.21.15.97:0/4249147029 wait complete. 2024-03-11T19:58:23.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.055+0000 7fd9d6c32740 20 client.15156 trim_cache size 0 max 16384 2024-03-11T19:58:23.061 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T19:58:23.061 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:58:23.061 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-11T19:58:23.061 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24599, 2024-03-11T19:58:23.061 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-11T19:58:23.061 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-11T19:58:23.061 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-11T19:58:23.061 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 157.01571522099999, 2024-03-11T19:58:23.062 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-03-11T19:58:23.062 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 93, 2024-03-11T19:58:23.062 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 89, 2024-03-11T19:58:23.062 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 164.02896206899999 2024-03-11T19:58:23.062 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T19:58:23.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.056+0000 7fd9d4000640 1 -- 172.21.15.97:0/2652745450 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd9b00a1a30 msgr2=0x7fd9b00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:23.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.056+0000 7fd9d4000640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd9b00a1a30 0x7fd9b00a3ef0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc02c180 tx=0x7fd9bc005ab0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.056+0000 7fd9d4000640 1 -- 172.21.15.97:0/2652745450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9cc102360 msgr2=0x7fd9cc19dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:23.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.056+0000 7fd9d4000640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9cc102360 0x7fd9cc19dd00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800efb0 tx=0x7fd9b800c800 comp rx=0 tx=0).stop 2024-03-11T19:58:23.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.056+0000 7fd9d4000640 1 -- 172.21.15.97:0/2652745450 shutdown_connections 2024-03-11T19:58:23.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.056+0000 7fd9d4000640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9cc104d10 0x7fd9cc1a4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.056+0000 7fd9d4000640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd9b00a1a30 0x7fd9b00a3ef0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.056+0000 7fd9d4000640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd9cc102360 0x7fd9cc19dd00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.056+0000 7fd9d4000640 1 --2- 172.21.15.97:0/2652745450 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9cc0ff9f0 0x7fd9cc19d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:23.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.056+0000 7fd9d4000640 1 -- 172.21.15.97:0/2652745450 >> 172.21.15.97:0/2652745450 conn(0x7fd9cc0fb610 msgr2=0x7fd9cc100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:23.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.056+0000 7fd9d4000640 1 -- 172.21.15.97:0/2652745450 shutdown_connections 2024-03-11T19:58:23.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:23.056+0000 7fd9d4000640 1 -- 172.21.15.97:0/2652745450 wait complete. 2024-03-11T19:58:23.071 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T19:58:23.072 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T19:58:23.082 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-03-11T19:58:23.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-11T19:58:23.528 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:58:23.978 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:23 smithi097 ceph-mon[31898]: pgmap v402: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:23.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:23 smithi114 ceph-mon[36376]: pgmap v402: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:24.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:23 smithi161 ceph-mon[36332]: pgmap v402: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:25.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.876+0000 7f84d8400700 1 -- 172.21.15.97:0/4111931768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d0105060 msgr2=0x7f84d0107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:25.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.876+0000 7f84d8400700 1 --2- 172.21.15.97:0/4111931768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d0105060 0x7f84d0107450 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f84bc0098f0 tx=0x7f84bc009c00 comp rx=0 tx=0).stop 2024-03-11T19:58:25.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.876+0000 7f84d8400700 1 -- 172.21.15.97:0/4111931768 shutdown_connections 2024-03-11T19:58:25.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.876+0000 7f84d8400700 1 --2- 172.21.15.97:0/4111931768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84d0102730 0x7f84d0104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:25.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.876+0000 7f84d8400700 1 --2- 172.21.15.97:0/4111931768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84d0107990 0x7f84d0109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:25.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.876+0000 7f84d8400700 1 --2- 172.21.15.97:0/4111931768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d0105060 0x7f84d0107450 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:25.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.876+0000 7f84d8400700 1 -- 172.21.15.97:0/4111931768 >> 172.21.15.97:0/4111931768 conn(0x7f84d00fc140 msgr2=0x7f84d00fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:25.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.876+0000 7f84d8400700 1 -- 172.21.15.97:0/4111931768 shutdown_connections 2024-03-11T19:58:25.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.876+0000 7f84d8400700 1 -- 172.21.15.97:0/4111931768 wait complete. 2024-03-11T19:58:25.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.877+0000 7f84d8400700 1 Processor -- start 2024-03-11T19:58:25.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.877+0000 7f84d8400700 1 -- start start 2024-03-11T19:58:25.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.877+0000 7f84d8400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84d0102730 0x7f84d0197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:25.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.877+0000 7f84d8400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84d0105060 0x7f84d0198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:25.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d8400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d0107990 0x7f84d019f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:25.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d8400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d019f870 con 0x7f84d0105060 2024-03-11T19:58:25.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d8400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d019f9b0 con 0x7f84d0107990 2024-03-11T19:58:25.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d8400700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d019fb70 con 0x7f84d0102730 2024-03-11T19:58:25.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d5000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84d0105060 0x7f84d0198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:25.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d5000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84d0105060 0x7f84d0198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44152/0 (socket says 172.21.15.97:44152) 2024-03-11T19:58:25.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d5000700 1 -- 172.21.15.97:0/2393145825 learned_addr learned my addr 172.21.15.97:0/2393145825 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:25.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d5a00700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84d0102730 0x7f84d0197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:25.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d6400700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d0107990 0x7f84d019f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:25.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d5000700 1 -- 172.21.15.97:0/2393145825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84d0102730 msgr2=0x7f84d0197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:25.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d5000700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84d0102730 0x7f84d0197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:25.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d5000700 1 -- 172.21.15.97:0/2393145825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d0107990 msgr2=0x7f84d019f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:25.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d5000700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d0107990 0x7f84d019f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:25.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d5000700 1 -- 172.21.15.97:0/2393145825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc009520 con 0x7f84d0105060 2024-03-11T19:58:25.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.878+0000 7f84d5a00700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84d0102730 0x7f84d0197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:25.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.879+0000 7f84d5000700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84d0105060 0x7f84d0198260 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f84bc005250 tx=0x7f84bc005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:25.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.879+0000 7f84ce000700 1 -- 172.21.15.97:0/2393145825 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc01c070 con 0x7f84d0105060 2024-03-11T19:58:25.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.879+0000 7f84d8400700 1 -- 172.21.15.97:0/2393145825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d019fdf0 con 0x7f84d0105060 2024-03-11T19:58:25.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.879+0000 7f84ce000700 1 -- 172.21.15.97:0/2393145825 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84bc00f930 con 0x7f84d0105060 2024-03-11T19:58:25.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.879+0000 7f84ce000700 1 -- 172.21.15.97:0/2393145825 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc004c30 con 0x7f84d0105060 2024-03-11T19:58:25.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.879+0000 7f84d8400700 1 -- 172.21.15.97:0/2393145825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d01a01b0 con 0x7f84d0105060 2024-03-11T19:58:25.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.880+0000 7f84d8400700 1 -- 172.21.15.97:0/2393145825 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d00fdcc0 con 0x7f84d0105060 2024-03-11T19:58:25.880 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.883+0000 7f84ce000700 1 -- 172.21.15.97:0/2393145825 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f84bc00b560 con 0x7f84d0105060 2024-03-11T19:58:25.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.884+0000 7f84ce000700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f84b80a1b40 0x7f84b80a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:25.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.884+0000 7f84ce000700 1 -- 172.21.15.97:0/2393145825 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f84bc0ce920 con 0x7f84d0105060 2024-03-11T19:58:25.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.884+0000 7f84d5a00700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f84b80a1b40 0x7f84b80a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:25.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.885+0000 7f84d5a00700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f84b80a1b40 0x7f84b80a4000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f84d0199200 tx=0x7f84c4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:25.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:25.889+0000 7f84ce000700 1 -- 172.21.15.97:0/2393145825 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f84bc0977a0 con 0x7f84d0105060 2024-03-11T19:58:25.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:25 smithi114 ceph-mon[36376]: pgmap v403: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:26.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:25 smithi161 ceph-mon[36332]: pgmap v403: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:26.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.126+0000 7f84d8400700 1 -- 172.21.15.97:0/2393145825 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f84d002d990 con 0x7f84b80a1b40 2024-03-11T19:58:26.125 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:25 smithi097 ceph-mon[31898]: pgmap v403: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:26.127 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.129+0000 7f84ce000700 1 -- 172.21.15.97:0/2393145825 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f84d002d990 con 0x7f84b80a1b40 2024-03-11T19:58:26.127 INFO:teuthology.orchestra.run.smithi097.stdout:Backend: cephadm 2024-03-11T19:58:26.127 INFO:teuthology.orchestra.run.smithi097.stdout:Available: Yes 2024-03-11T19:58:26.127 INFO:teuthology.orchestra.run.smithi097.stdout:Paused: No 2024-03-11T19:58:26.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.132+0000 7f84d8400700 1 -- 172.21.15.97:0/2393145825 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f84b80a1b40 msgr2=0x7f84b80a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:26.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.132+0000 7f84d8400700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f84b80a1b40 0x7f84b80a4000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f84d0199200 tx=0x7f84c4009310 comp rx=0 tx=0).stop 2024-03-11T19:58:26.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.132+0000 7f84d8400700 1 -- 172.21.15.97:0/2393145825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84d0105060 msgr2=0x7f84d0198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:26.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.132+0000 7f84d8400700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84d0105060 0x7f84d0198260 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f84bc005250 tx=0x7f84bc005280 comp rx=0 tx=0).stop 2024-03-11T19:58:26.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.132+0000 7f84d8400700 1 -- 172.21.15.97:0/2393145825 shutdown_connections 2024-03-11T19:58:26.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.132+0000 7f84d8400700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f84b80a1b40 0x7f84b80a4000 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:26.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.132+0000 7f84d8400700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d0107990 0x7f84d019f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:26.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.132+0000 7f84d8400700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f84d0102730 0x7f84d0197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:26.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.132+0000 7f84d8400700 1 --2- 172.21.15.97:0/2393145825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f84d0105060 0x7f84d0198260 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:26.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.132+0000 7f84d8400700 1 -- 172.21.15.97:0/2393145825 >> 172.21.15.97:0/2393145825 conn(0x7f84d00fc140 msgr2=0x7f84d01045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:26.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.132+0000 7f84d8400700 1 -- 172.21.15.97:0/2393145825 shutdown_connections 2024-03-11T19:58:26.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:26.132+0000 7f84d8400700 1 -- 172.21.15.97:0/2393145825 wait complete. 2024-03-11T19:58:27.037 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-11T19:58:27.496 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:58:27.833 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:27 smithi097 ceph-mon[31898]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:27.833 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:27 smithi097 ceph-mon[31898]: pgmap v404: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:27.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:27 smithi114 ceph-mon[36376]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:27.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:27 smithi114 ceph-mon[36376]: pgmap v404: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:28.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:27 smithi161 ceph-mon[36332]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:28.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:27 smithi161 ceph-mon[36332]: pgmap v404: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:29.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.705+0000 7facb0600700 1 -- 172.21.15.97:0/596344768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faca8074670 msgr2=0x7faca8074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:29.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.705+0000 7facb0600700 1 --2- 172.21.15.97:0/596344768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faca8074670 0x7faca8074af0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fac940098f0 tx=0x7fac94009c00 comp rx=0 tx=0).stop 2024-03-11T19:58:29.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.706+0000 7facb0600700 1 -- 172.21.15.97:0/596344768 shutdown_connections 2024-03-11T19:58:29.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.706+0000 7facb0600700 1 --2- 172.21.15.97:0/596344768 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faca8075c40 0x7faca80740a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:29.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.706+0000 7facb0600700 1 --2- 172.21.15.97:0/596344768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca8109bd0 0x7faca810c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:29.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.706+0000 7facb0600700 1 --2- 172.21.15.97:0/596344768 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faca8074670 0x7faca8074af0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:29.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.706+0000 7facb0600700 1 -- 172.21.15.97:0/596344768 >> 172.21.15.97:0/596344768 conn(0x7faca80fd290 msgr2=0x7faca80ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:29.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.706+0000 7facb0600700 1 -- 172.21.15.97:0/596344768 shutdown_connections 2024-03-11T19:58:29.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.706+0000 7facb0600700 1 -- 172.21.15.97:0/596344768 wait complete. 2024-03-11T19:58:29.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.707+0000 7facb0600700 1 Processor -- start 2024-03-11T19:58:29.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.707+0000 7facb0600700 1 -- start start 2024-03-11T19:58:29.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.707+0000 7facb0600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca8074670 0x7faca8199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.707+0000 7facb0600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faca8075c40 0x7faca819a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.707+0000 7facb0600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faca8109bd0 0x7faca81a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.708+0000 7facb0600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca81a1a10 con 0x7faca8109bd0 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.708+0000 7facb0600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca81a1b50 con 0x7faca8074670 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.708+0000 7facb0600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca81a1ce0 con 0x7faca8075c40 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.708+0000 7facadc00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca8074670 0x7faca8199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.708+0000 7facadc00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca8074670 0x7faca8199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:43024/0 (socket says 172.21.15.97:43024) 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.708+0000 7facadc00700 1 -- 172.21.15.97:0/3883417136 learned_addr learned my addr 172.21.15.97:0/3883417136 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.708+0000 7facad200700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faca8075c40 0x7faca819a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.708+0000 7facae600700 1 --2- 172.21.15.97:0/3883417136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faca8109bd0 0x7faca81a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.708+0000 7facadc00700 1 -- 172.21.15.97:0/3883417136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faca8075c40 msgr2=0x7faca819a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.708+0000 7facadc00700 1 --2- 172.21.15.97:0/3883417136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faca8075c40 0x7faca819a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.708+0000 7facadc00700 1 -- 172.21.15.97:0/3883417136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faca8109bd0 msgr2=0x7faca81a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.708+0000 7facadc00700 1 --2- 172.21.15.97:0/3883417136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faca8109bd0 0x7faca81a1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:29.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.708+0000 7facadc00700 1 -- 172.21.15.97:0/3883417136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac94009520 con 0x7faca8074670 2024-03-11T19:58:29.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.709+0000 7facadc00700 1 --2- 172.21.15.97:0/3883417136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca8074670 0x7faca8199e80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fac9c00e9c0 tx=0x7fac9c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:29.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.709+0000 7faca6000700 1 -- 172.21.15.97:0/3883417136 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c00cbd0 con 0x7faca8074670 2024-03-11T19:58:29.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.709+0000 7faca6000700 1 -- 172.21.15.97:0/3883417136 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac9c00eeb0 con 0x7faca8074670 2024-03-11T19:58:29.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.709+0000 7facb0600700 1 -- 172.21.15.97:0/3883417136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca81a1fc0 con 0x7faca8074670 2024-03-11T19:58:29.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.709+0000 7faca6000700 1 -- 172.21.15.97:0/3883417136 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c005680 con 0x7faca8074670 2024-03-11T19:58:29.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.709+0000 7facb0600700 1 -- 172.21.15.97:0/3883417136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca81a23d0 con 0x7faca8074670 2024-03-11T19:58:29.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.710+0000 7facb0600700 1 -- 172.21.15.97:0/3883417136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca81089e0 con 0x7faca8074670 2024-03-11T19:58:29.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.712+0000 7faca6000700 1 -- 172.21.15.97:0/3883417136 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fac9c004750 con 0x7faca8074670 2024-03-11T19:58:29.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.713+0000 7faca6000700 1 --2- 172.21.15.97:0/3883417136 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fac900a1ac0 0x7fac900a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:29.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.713+0000 7faca6000700 1 -- 172.21.15.97:0/3883417136 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fac9c014070 con 0x7faca8074670 2024-03-11T19:58:29.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.714+0000 7facad200700 1 --2- 172.21.15.97:0/3883417136 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fac900a1ac0 0x7fac900a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:29.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.714+0000 7facad200700 1 --2- 172.21.15.97:0/3883417136 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fac900a1ac0 0x7fac900a3f80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fac940098c0 tx=0x7fac94019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:29.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.718+0000 7faca6000700 1 -- 172.21.15.97:0/3883417136 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fac9c096750 con 0x7faca8074670 2024-03-11T19:58:29.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:29 smithi097 ceph-mon[31898]: pgmap v405: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:29.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.945+0000 7facb0600700 1 -- 172.21.15.97:0/3883417136 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faca802da40 con 0x7fac900a1ac0 2024-03-11T19:58:29.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.960+0000 7faca6000700 1 -- 172.21.15.97:0/3883417136 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4123 (secure 0 0 0) 0x7faca802da40 con 0x7fac900a1ac0 2024-03-11T19:58:29.959 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T19:58:29.959 INFO:teuthology.orchestra.run.smithi097.stdout:mds.a smithi097 running (2m) 51s ago 2m 16.0M - 19.0.0-1607-g94a53970 a6477a474d71 f4167ce20b3a 2024-03-11T19:58:29.959 INFO:teuthology.orchestra.run.smithi097.stdout:mds.b smithi114 running (2m) 102s ago 2m 25.8M - 19.0.0-1607-g94a53970 a6477a474d71 1d053fe741d3 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mds.c smithi161 running (3m) 102s ago 3m 19.0M - 19.0.0-1607-g94a53970 a6477a474d71 1a128a11cdb9 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mds.d smithi097 running (3m) 51s ago 3m 15.2M - 19.0.0-1607-g94a53970 a6477a474d71 8295e80e4b3a 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mds.e smithi114 running (3m) 102s ago 3m 16.0M - 19.0.0-1607-g94a53970 a6477a474d71 8ef026329e7f 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mds.f smithi161 running (3m) 102s ago 3m 18.2M - 19.0.0-1607-g94a53970 a6477a474d71 7764197c160f 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mds.g smithi097 running (2m) 51s ago 2m 12.3M - 19.0.0-1607-g94a53970 a6477a474d71 7d64ddc1dbc3 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mds.h smithi114 running (3m) 102s ago 3m 19.3M - 19.0.0-1607-g94a53970 a6477a474d71 cbd981c5cca1 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mds.i smithi161 running (3m) 102s ago 3m 18.9M - 19.0.0-1607-g94a53970 a6477a474d71 2e4833f4daa6 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mds.j smithi097 running (3m) 51s ago 3m 12.5M - 19.0.0-1607-g94a53970 a6477a474d71 80123fcb5e22 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mds.k smithi114 running (3m) 102s ago 3m 16.5M - 19.0.0-1607-g94a53970 a6477a474d71 80803eebc596 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mds.l smithi161 running (3m) 102s ago 3m 15.0M - 19.0.0-1607-g94a53970 a6477a474d71 4691c5ad38ed 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi097 *:9283,8765 running (11m) 51s ago 11m 513M - 19.0.0-1607-g94a53970 a6477a474d71 69fd8238897f 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi114 *:8443,8765 running (9m) 102s ago 9m 451M - 19.0.0-1607-g94a53970 a6477a474d71 45ae7a66dab3 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.z smithi161 *:8443,8765 running (9m) 102s ago 9m 452M - 19.0.0-1607-g94a53970 a6477a474d71 b71e0c963bb2 2024-03-11T19:58:29.960 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (11m) 51s ago 11m 47.3M 2048M 19.0.0-1607-g94a53970 a6477a474d71 3c2b428eeaee 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (9m) 102s ago 9m 40.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 02bc52469f3a 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi161 running (9m) 102s ago 9m 40.8M 2048M 19.0.0-1607-g94a53970 a6477a474d71 767d2c35ced1 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (8m) 51s ago 8m 52.9M 4272M 19.0.0-1607-g94a53970 a6477a474d71 9091d43dab05 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi114 running (8m) 102s ago 8m 50.0M 4270M 19.0.0-1607-g94a53970 a6477a474d71 cb023e21741b 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi161 running (8m) 102s ago 8m 57.9M 4270M 19.0.0-1607-g94a53970 a6477a474d71 54cba555377a 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (7m) 51s ago 7m 43.8M 4272M 19.0.0-1607-g94a53970 a6477a474d71 8fa5f084638c 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (7m) 102s ago 7m 45.6M 4270M 19.0.0-1607-g94a53970 a6477a474d71 8d193d081841 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi161 running (6m) 102s ago 6m 55.5M 4270M 19.0.0-1607-g94a53970 a6477a474d71 83084f0fbfd2 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi097 running (6m) 51s ago 6m 41.0M 4272M 19.0.0-1607-g94a53970 a6477a474d71 5f774f361093 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (5m) 102s ago 5m 53.0M 4270M 19.0.0-1607-g94a53970 a6477a474d71 47e62e2f4d5d 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8 smithi161 running (5m) 102s ago 5m 53.1M 4270M 19.0.0-1607-g94a53970 a6477a474d71 99b754ca2fc4 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9 smithi097 running (4m) 51s ago 4m 48.8M 4272M 19.0.0-1607-g94a53970 a6477a474d71 e0b34c2d0e31 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10 smithi114 running (4m) 102s ago 4m 51.5M 4270M 19.0.0-1607-g94a53970 a6477a474d71 cf5216ee0706 2024-03-11T19:58:29.961 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11 smithi161 running (3m) 102s ago 3m 52.1M 4270M 19.0.0-1607-g94a53970 a6477a474d71 9151dbf7de92 2024-03-11T19:58:29.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.964+0000 7facb0600700 1 -- 172.21.15.97:0/3883417136 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fac900a1ac0 msgr2=0x7fac900a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:29.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.964+0000 7facb0600700 1 --2- 172.21.15.97:0/3883417136 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fac900a1ac0 0x7fac900a3f80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fac940098c0 tx=0x7fac94019040 comp rx=0 tx=0).stop 2024-03-11T19:58:29.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.964+0000 7facb0600700 1 -- 172.21.15.97:0/3883417136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca8074670 msgr2=0x7faca8199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:29.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.964+0000 7facb0600700 1 --2- 172.21.15.97:0/3883417136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca8074670 0x7faca8199e80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fac9c00e9c0 tx=0x7fac9c00ecd0 comp rx=0 tx=0).stop 2024-03-11T19:58:29.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.964+0000 7facb0600700 1 -- 172.21.15.97:0/3883417136 shutdown_connections 2024-03-11T19:58:29.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.964+0000 7facb0600700 1 --2- 172.21.15.97:0/3883417136 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fac900a1ac0 0x7fac900a3f80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:29.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.964+0000 7facb0600700 1 --2- 172.21.15.97:0/3883417136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faca8109bd0 0x7faca81a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:29.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.964+0000 7facb0600700 1 --2- 172.21.15.97:0/3883417136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faca8074670 0x7faca8199e80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:29.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.964+0000 7facb0600700 1 --2- 172.21.15.97:0/3883417136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faca8075c40 0x7faca819a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:29.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.964+0000 7facb0600700 1 -- 172.21.15.97:0/3883417136 >> 172.21.15.97:0/3883417136 conn(0x7faca80fd290 msgr2=0x7faca8105b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:29.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.964+0000 7facb0600700 1 -- 172.21.15.97:0/3883417136 shutdown_connections 2024-03-11T19:58:29.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:29.964+0000 7facb0600700 1 -- 172.21.15.97:0/3883417136 wait complete. 2024-03-11T19:58:29.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:29 smithi114 ceph-mon[36376]: pgmap v405: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:30.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:29 smithi161 ceph-mon[36332]: pgmap v405: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:30.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-11T19:58:31.429 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:58:31.939 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:31 smithi097 ceph-mon[31898]: from='client.25280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:31.939 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:31 smithi097 ceph-mon[31898]: pgmap v406: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:31.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:31 smithi114 ceph-mon[36376]: from='client.25280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:31.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:31 smithi114 ceph-mon[36376]: pgmap v406: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:32.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:31 smithi161 ceph-mon[36332]: from='client.25280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:32.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:31 smithi161 ceph-mon[36332]: pgmap v406: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:33.856 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:33 smithi097 ceph-mon[31898]: pgmap v407: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:33.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.892+0000 7f37b5800700 1 -- 172.21.15.97:0/1988087960 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b0109ba0 msgr2=0x7f37b010c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:33.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.892+0000 7f37b5800700 1 --2- 172.21.15.97:0/1988087960 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b0109ba0 0x7f37b010c010 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f37a800b0a0 tx=0x7f37a800b3b0 comp rx=0 tx=0).stop 2024-03-11T19:58:33.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.892+0000 7f37b5800700 1 -- 172.21.15.97:0/1988087960 shutdown_connections 2024-03-11T19:58:33.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.892+0000 7f37b5800700 1 --2- 172.21.15.97:0/1988087960 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b0109ba0 0x7f37b010c010 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:33.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.892+0000 7f37b5800700 1 --2- 172.21.15.97:0/1988087960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b0075c20 0x7f37b0074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:33.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.892+0000 7f37b5800700 1 --2- 172.21.15.97:0/1988087960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37b0074650 0x7f37b0074ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:33.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.892+0000 7f37b5800700 1 -- 172.21.15.97:0/1988087960 >> 172.21.15.97:0/1988087960 conn(0x7f37b00fd280 msgr2=0x7f37b00ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:33.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.892+0000 7f37b5800700 1 -- 172.21.15.97:0/1988087960 shutdown_connections 2024-03-11T19:58:33.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.893+0000 7f37b5800700 1 -- 172.21.15.97:0/1988087960 wait complete. 2024-03-11T19:58:33.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.893+0000 7f37b5800700 1 Processor -- start 2024-03-11T19:58:33.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.894+0000 7f37b5800700 1 -- start start 2024-03-11T19:58:33.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.894+0000 7f37b5800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b0074650 0x7f37b0199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:33.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.894+0000 7f37b5800700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37b0075c20 0x7f37b019a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:33.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.894+0000 7f37b5800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b0109ba0 0x7f37b01a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:33.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.894+0000 7f37b5800700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b01a1ae0 con 0x7f37b0074650 2024-03-11T19:58:33.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.894+0000 7f37b5800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b01a1c20 con 0x7f37b0109ba0 2024-03-11T19:58:33.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.894+0000 7f37b5800700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b01a1db0 con 0x7f37b0075c20 2024-03-11T19:58:33.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.894+0000 7f37aea00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b0074650 0x7f37b0199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:33.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.894+0000 7f37ae000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37b0075c20 0x7f37b019a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.894+0000 7f37aea00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b0074650 0x7f37b0199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58194/0 (socket says 172.21.15.97:58194) 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.894+0000 7f37aea00700 1 -- 172.21.15.97:0/4272186404 learned_addr learned my addr 172.21.15.97:0/4272186404 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.894+0000 7f37af400700 1 --2- 172.21.15.97:0/4272186404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b0109ba0 0x7f37b01a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.895+0000 7f37ae000700 1 -- 172.21.15.97:0/4272186404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b0109ba0 msgr2=0x7f37b01a1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.895+0000 7f37ae000700 1 --2- 172.21.15.97:0/4272186404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b0109ba0 0x7f37b01a1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.895+0000 7f37ae000700 1 -- 172.21.15.97:0/4272186404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b0074650 msgr2=0x7f37b0199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.895+0000 7f37ae000700 1 --2- 172.21.15.97:0/4272186404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b0074650 0x7f37b0199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.895+0000 7f37ae000700 1 -- 172.21.15.97:0/4272186404 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a8009d00 con 0x7f37b0075c20 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.895+0000 7f37ae000700 1 --2- 172.21.15.97:0/4272186404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37b0075c20 0x7f37b019a490 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f379c00ea20 tx=0x7f379c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.895+0000 7f37a7400700 1 -- 172.21.15.97:0/4272186404 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c00cbd0 con 0x7f37b0075c20 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.896+0000 7f37a7400700 1 -- 172.21.15.97:0/4272186404 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f379c00ee60 con 0x7f37b0075c20 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.896+0000 7f37b5800700 1 -- 172.21.15.97:0/4272186404 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b01a2090 con 0x7f37b0075c20 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.896+0000 7f37a7400700 1 -- 172.21.15.97:0/4272186404 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c018680 con 0x7f37b0075c20 2024-03-11T19:58:33.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.896+0000 7f37b5800700 1 -- 172.21.15.97:0/4272186404 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b01a2590 con 0x7f37b0075c20 2024-03-11T19:58:33.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.897+0000 7f37b5800700 1 -- 172.21.15.97:0/4272186404 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b01089b0 con 0x7f37b0075c20 2024-03-11T19:58:33.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.899+0000 7f37a7400700 1 -- 172.21.15.97:0/4272186404 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f379c010430 con 0x7f37b0075c20 2024-03-11T19:58:33.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.900+0000 7f37a7400700 1 --2- 172.21.15.97:0/4272186404 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f37940a1ac0 0x7f37940a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:33.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.900+0000 7f37a7400700 1 -- 172.21.15.97:0/4272186404 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f379c014070 con 0x7f37b0075c20 2024-03-11T19:58:33.899 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.900+0000 7f37aea00700 1 --2- 172.21.15.97:0/4272186404 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f37940a1ac0 0x7f37940a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:33.900 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.901+0000 7f37aea00700 1 --2- 172.21.15.97:0/4272186404 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f37940a1ac0 0x7f37940a3f80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009d20 tx=0x7f37a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:33.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:33.904+0000 7f37a7400700 1 -- 172.21.15.97:0/4272186404 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f379c052d20 con 0x7f37b0075c20 2024-03-11T19:58:33.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:33 smithi114 ceph-mon[36376]: pgmap v407: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:34.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:33 smithi161 ceph-mon[36332]: pgmap v407: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:34.132 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.133+0000 7f37b5800700 1 -- 172.21.15.97:0/4272186404 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f37b002d870 con 0x7f37940a1ac0 2024-03-11T19:58:34.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.141+0000 7f37a7400700 1 -- 172.21.15.97:0/4272186404 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f37b002d870 con 0x7f37940a1ac0 2024-03-11T19:58:34.141 INFO:teuthology.orchestra.run.smithi097.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-11T19:58:34.141 INFO:teuthology.orchestra.run.smithi097.stdout:mds.all 12/12 107s ago 3m smithi097=a;smithi097=d;smithi097=g;smithi097=j;smithi114=b;smithi114=e;smithi114=h;smithi114=k;smithi161=c;smithi161=f;smithi161=i;smithi161=l;count:12 2024-03-11T19:58:34.141 INFO:teuthology.orchestra.run.smithi097.stdout:mgr 3/3 107s ago 9m smithi097=x;smithi114=y;smithi161=z;count:3 2024-03-11T19:58:34.141 INFO:teuthology.orchestra.run.smithi097.stdout:mon 3/3 107s ago 9m smithi097:172.21.15.97=a;smithi114:172.21.15.114=b;smithi161:172.21.15.161=c;count:3 2024-03-11T19:58:34.141 INFO:teuthology.orchestra.run.smithi097.stdout:osd 12 107s ago - 2024-03-11T19:58:34.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.145+0000 7f37b5800700 1 -- 172.21.15.97:0/4272186404 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f37940a1ac0 msgr2=0x7f37940a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:34.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.145+0000 7f37b5800700 1 --2- 172.21.15.97:0/4272186404 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f37940a1ac0 0x7f37940a3f80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009d20 tx=0x7f37a0009260 comp rx=0 tx=0).stop 2024-03-11T19:58:34.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.145+0000 7f37b5800700 1 -- 172.21.15.97:0/4272186404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37b0075c20 msgr2=0x7f37b019a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:34.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.145+0000 7f37b5800700 1 --2- 172.21.15.97:0/4272186404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37b0075c20 0x7f37b019a490 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f379c00ea20 tx=0x7f379c00ed30 comp rx=0 tx=0).stop 2024-03-11T19:58:34.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.145+0000 7f37b5800700 1 -- 172.21.15.97:0/4272186404 shutdown_connections 2024-03-11T19:58:34.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.145+0000 7f37b5800700 1 --2- 172.21.15.97:0/4272186404 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f37940a1ac0 0x7f37940a3f80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:34.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.145+0000 7f37b5800700 1 --2- 172.21.15.97:0/4272186404 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f37b0074650 0x7f37b0199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:34.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.145+0000 7f37b5800700 1 --2- 172.21.15.97:0/4272186404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b0109ba0 0x7f37b01a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:34.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.145+0000 7f37b5800700 1 --2- 172.21.15.97:0/4272186404 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37b0075c20 0x7f37b019a490 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:34.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.145+0000 7f37b5800700 1 -- 172.21.15.97:0/4272186404 >> 172.21.15.97:0/4272186404 conn(0x7f37b00fd280 msgr2=0x7f37b0105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:34.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.145+0000 7f37b5800700 1 -- 172.21.15.97:0/4272186404 shutdown_connections 2024-03-11T19:58:34.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:34.145+0000 7f37b5800700 1 -- 172.21.15.97:0/4272186404 wait complete. 2024-03-11T19:58:35.144 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-11T19:58:35.596 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:58:35.966 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:35 smithi097 ceph-mon[31898]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:35.967 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:35 smithi097 ceph-mon[31898]: pgmap v408: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:35.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:35 smithi114 ceph-mon[36376]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:35.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:35 smithi114 ceph-mon[36376]: pgmap v408: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:36.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:35 smithi161 ceph-mon[36332]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:36.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:35 smithi161 ceph-mon[36332]: pgmap v408: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:37.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:37 smithi114 ceph-mon[36376]: pgmap v409: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:38.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.014+0000 7f42a0600700 1 -- 172.21.15.97:0/3951321440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42981082f0 msgr2=0x7f4298108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:38.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.014+0000 7f42a0600700 1 --2- 172.21.15.97:0/3951321440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42981082f0 0x7f4298108710 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f428c009990 tx=0x7f428c009ca0 comp rx=0 tx=0).stop 2024-03-11T19:58:38.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.014+0000 7f42a0600700 1 -- 172.21.15.97:0/3951321440 shutdown_connections 2024-03-11T19:58:38.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.014+0000 7f42a0600700 1 --2- 172.21.15.97:0/3951321440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4298109e90 0x7f4298110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:38.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.014+0000 7f42a0600700 1 --2- 172.21.15.97:0/3951321440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42981082f0 0x7f4298108710 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:38.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.014+0000 7f42a0600700 1 --2- 172.21.15.97:0/3951321440 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f42981094f0 0x7f4298109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:38.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.014+0000 7f42a0600700 1 -- 172.21.15.97:0/3951321440 >> 172.21.15.97:0/3951321440 conn(0x7f42980772c0 msgr2=0x7f4298079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:38.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.014+0000 7f42a0600700 1 -- 172.21.15.97:0/3951321440 shutdown_connections 2024-03-11T19:58:38.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.014+0000 7f42a0600700 1 -- 172.21.15.97:0/3951321440 wait complete. 2024-03-11T19:58:38.015 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:37 smithi097 ceph-mon[31898]: pgmap v409: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:38.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.016+0000 7f42a0600700 1 Processor -- start 2024-03-11T19:58:38.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.016+0000 7f42a0600700 1 -- start start 2024-03-11T19:58:38.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.017+0000 7f42a0600700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f42981082f0 0x7f429819e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:38.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.017+0000 7f42a0600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42981094f0 0x7f429819e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:38.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.017+0000 7f42a0600700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4298109e90 0x7f42981a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:38.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.017+0000 7f42a0600700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42981a5e20 con 0x7f42981082f0 2024-03-11T19:58:38.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.017+0000 7f42a0600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42981a5f60 con 0x7f42981094f0 2024-03-11T19:58:38.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.017+0000 7f42a0600700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42981a60f0 con 0x7f4298109e90 2024-03-11T19:58:38.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.017+0000 7f429dc00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f42981082f0 0x7f429819e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:38.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.017+0000 7f429dc00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f42981082f0 0x7f429819e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58216/0 (socket says 172.21.15.97:58216) 2024-03-11T19:58:38.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.017+0000 7f429dc00700 1 -- 172.21.15.97:0/3126057832 learned_addr learned my addr 172.21.15.97:0/3126057832 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:38.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.017+0000 7f429d200700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42981094f0 0x7f429819e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:38.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.017+0000 7f429e600700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4298109e90 0x7f42981a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:38.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.018+0000 7f429dc00700 1 -- 172.21.15.97:0/3126057832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4298109e90 msgr2=0x7f42981a5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:38.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.018+0000 7f429dc00700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4298109e90 0x7f42981a5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:38.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.018+0000 7f429dc00700 1 -- 172.21.15.97:0/3126057832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42981094f0 msgr2=0x7f429819e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:38.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.018+0000 7f429dc00700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42981094f0 0x7f429819e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:38.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.018+0000 7f429dc00700 1 -- 172.21.15.97:0/3126057832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4284009520 con 0x7f42981082f0 2024-03-11T19:58:38.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.018+0000 7f429dc00700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f42981082f0 0x7f429819e290 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f428c000c00 tx=0x7f428c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:38.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.018+0000 7f4296000700 1 -- 172.21.15.97:0/3126057832 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c01d070 con 0x7f42981082f0 2024-03-11T19:58:38.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.018+0000 7f4296000700 1 -- 172.21.15.97:0/3126057832 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f428c005030 con 0x7f42981082f0 2024-03-11T19:58:38.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.019+0000 7f42a0600700 1 -- 172.21.15.97:0/3126057832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428c0095f0 con 0x7f42981082f0 2024-03-11T19:58:38.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.019+0000 7f4296000700 1 -- 172.21.15.97:0/3126057832 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c004c40 con 0x7f42981082f0 2024-03-11T19:58:38.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.019+0000 7f42a0600700 1 -- 172.21.15.97:0/3126057832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42981a6590 con 0x7f42981082f0 2024-03-11T19:58:38.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.020+0000 7f42a0600700 1 -- 172.21.15.97:0/3126057832 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429810f6b0 con 0x7f42981082f0 2024-03-11T19:58:38.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.022+0000 7f4296000700 1 -- 172.21.15.97:0/3126057832 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f428c005530 con 0x7f42981082f0 2024-03-11T19:58:38.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.024+0000 7f4296000700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f42800a1ac0 0x7f42800a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:38.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.024+0000 7f4296000700 1 -- 172.21.15.97:0/3126057832 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f428c0cedb0 con 0x7f42981082f0 2024-03-11T19:58:38.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.024+0000 7f429d200700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f42800a1ac0 0x7f42800a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:38.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.025+0000 7f429d200700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f42800a1ac0 0x7f42800a3f80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4284009df0 tx=0x7f4284009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:38.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.026+0000 7f4296000700 1 -- 172.21.15.97:0/3126057832 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f428c097c30 con 0x7f42981082f0 2024-03-11T19:58:38.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:37 smithi161 ceph-mon[36332]: pgmap v409: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:38.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.256+0000 7f42a0600700 1 -- 172.21.15.97:0/3126057832 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f429802d8d0 con 0x7f42800a1ac0 2024-03-11T19:58:38.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.258+0000 7f4296000700 1 -- 172.21.15.97:0/3126057832 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f429802d8d0 con 0x7f42800a1ac0 2024-03-11T19:58:38.258 INFO:teuthology.orchestra.run.smithi097.stdout:HOST ADDR LABELS STATUS 2024-03-11T19:58:38.258 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 172.21.15.97 2024-03-11T19:58:38.258 INFO:teuthology.orchestra.run.smithi097.stdout:smithi114 172.21.15.114 2024-03-11T19:58:38.258 INFO:teuthology.orchestra.run.smithi097.stdout:smithi161 172.21.15.161 2024-03-11T19:58:38.258 INFO:teuthology.orchestra.run.smithi097.stdout:3 hosts in cluster 2024-03-11T19:58:38.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.261+0000 7f42a0600700 1 -- 172.21.15.97:0/3126057832 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f42800a1ac0 msgr2=0x7f42800a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:38.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.261+0000 7f42a0600700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f42800a1ac0 0x7f42800a3f80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4284009df0 tx=0x7f4284009260 comp rx=0 tx=0).stop 2024-03-11T19:58:38.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.261+0000 7f42a0600700 1 -- 172.21.15.97:0/3126057832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f42981082f0 msgr2=0x7f429819e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:38.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.261+0000 7f42a0600700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f42981082f0 0x7f429819e290 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f428c000c00 tx=0x7f428c004730 comp rx=0 tx=0).stop 2024-03-11T19:58:38.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.262+0000 7f42a0600700 1 -- 172.21.15.97:0/3126057832 shutdown_connections 2024-03-11T19:58:38.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.262+0000 7f42a0600700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f42800a1ac0 0x7f42800a3f80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:38.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.262+0000 7f42a0600700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f42981082f0 0x7f429819e290 secure :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f428c000c00 tx=0x7f428c004730 comp rx=0 tx=0).stop 2024-03-11T19:58:38.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.262+0000 7f42a0600700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4298109e90 0x7f42981a5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:38.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.262+0000 7f42a0600700 1 --2- 172.21.15.97:0/3126057832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42981094f0 0x7f429819e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:38.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.262+0000 7f42a0600700 1 -- 172.21.15.97:0/3126057832 >> 172.21.15.97:0/3126057832 conn(0x7f42980772c0 msgr2=0x7f429810c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:38.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.262+0000 7f42a0600700 1 -- 172.21.15.97:0/3126057832 shutdown_connections 2024-03-11T19:58:38.262 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:38.262+0000 7f42a0600700 1 -- 172.21.15.97:0/3126057832 wait complete. 2024-03-11T19:58:39.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-11T19:58:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:58:39.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:39 smithi114 ceph-mon[36376]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:39.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:39 smithi114 ceph-mon[36376]: pgmap v410: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:40.016 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:39 smithi097 ceph-mon[31898]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:40.016 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:39 smithi097 ceph-mon[31898]: pgmap v410: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:40.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:39 smithi161 ceph-mon[36332]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:40.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:39 smithi161 ceph-mon[36332]: pgmap v410: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:41.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:40 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:58:41.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:40 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:40 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:40 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:40 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.098 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:40 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:58:41.098 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:40 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.098 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:40 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.098 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:40 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.098 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:40 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:40 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:58:41.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:40 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:40 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:40 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:40 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:41 smithi161 ceph-mon[36332]: pgmap v411: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:41.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:41 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:41 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.901 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:41 smithi097 ceph-mon[31898]: pgmap v411: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:41.901 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:41 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.901 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:41 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.960+0000 7fbc81000700 1 -- 172.21.15.97:0/3708857132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc7c075ca0 msgr2=0x7fbc7c074100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:41.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.960+0000 7fbc81000700 1 --2- 172.21.15.97:0/3708857132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc7c075ca0 0x7fbc7c074100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0098f0 tx=0x7fbc6c009c00 comp rx=0 tx=0).stop 2024-03-11T19:58:41.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.960+0000 7fbc81000700 1 -- 172.21.15.97:0/3708857132 shutdown_connections 2024-03-11T19:58:41.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.960+0000 7fbc81000700 1 --2- 172.21.15.97:0/3708857132 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc7c10e1c0 0x7fbc7c1105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:41.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.960+0000 7fbc81000700 1 --2- 172.21.15.97:0/3708857132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc7c075ca0 0x7fbc7c074100 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:41.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.960+0000 7fbc81000700 1 --2- 172.21.15.97:0/3708857132 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc7c0746d0 0x7fbc7c074b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:41.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.960+0000 7fbc81000700 1 -- 172.21.15.97:0/3708857132 >> 172.21.15.97:0/3708857132 conn(0x7fbc7c0fd180 msgr2=0x7fbc7c0ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:41.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.960+0000 7fbc81000700 1 -- 172.21.15.97:0/3708857132 shutdown_connections 2024-03-11T19:58:41.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.960+0000 7fbc81000700 1 -- 172.21.15.97:0/3708857132 wait complete. 2024-03-11T19:58:41.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.962+0000 7fbc81000700 1 Processor -- start 2024-03-11T19:58:41.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.962+0000 7fbc81000700 1 -- start start 2024-03-11T19:58:41.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.962+0000 7fbc81000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc7c0746d0 0x7fbc7c19e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:41.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.962+0000 7fbc81000700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc7c075ca0 0x7fbc7c19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:41.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.962+0000 7fbc81000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc7c10e1c0 0x7fbc7c1a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:41.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.962+0000 7fbc81000700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c1a5d80 con 0x7fbc7c075ca0 2024-03-11T19:58:41.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.962+0000 7fbc81000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c1a5ec0 con 0x7fbc7c10e1c0 2024-03-11T19:58:41.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.962+0000 7fbc81000700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c1a6050 con 0x7fbc7c0746d0 2024-03-11T19:58:41.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.963+0000 7fbc7be00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc7c10e1c0 0x7fbc7c1a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:41.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.963+0000 7fbc7aa00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc7c075ca0 0x7fbc7c19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:41.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.963+0000 7fbc7b400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc7c0746d0 0x7fbc7c19e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:41.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.963+0000 7fbc7be00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc7c10e1c0 0x7fbc7c1a57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:50732/0 (socket says 172.21.15.97:50732) 2024-03-11T19:58:41.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.963+0000 7fbc7be00700 1 -- 172.21.15.97:0/237343606 learned_addr learned my addr 172.21.15.97:0/237343606 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:41.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.963+0000 7fbc7b400700 1 -- 172.21.15.97:0/237343606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc7c10e1c0 msgr2=0x7fbc7c1a57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:41.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.963+0000 7fbc7b400700 1 --2- 172.21.15.97:0/237343606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc7c10e1c0 0x7fbc7c1a57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:41.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.963+0000 7fbc7b400700 1 -- 172.21.15.97:0/237343606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc7c075ca0 msgr2=0x7fbc7c19e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:41.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.963+0000 7fbc7b400700 1 --2- 172.21.15.97:0/237343606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc7c075ca0 0x7fbc7c19e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:41.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.963+0000 7fbc7b400700 1 -- 172.21.15.97:0/237343606 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc74009d00 con 0x7fbc7c0746d0 2024-03-11T19:58:41.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.963+0000 7fbc7be00700 1 --2- 172.21.15.97:0/237343606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc7c10e1c0 0x7fbc7c1a57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:41.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.964+0000 7fbc7b400700 1 --2- 172.21.15.97:0/237343606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc7c0746d0 0x7fbc7c19e1f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009230 tx=0x7fbc6c005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:41.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.964+0000 7fbc73e00700 1 -- 172.21.15.97:0/237343606 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c01c070 con 0x7fbc7c0746d0 2024-03-11T19:58:41.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.964+0000 7fbc73e00700 1 -- 172.21.15.97:0/237343606 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc6c005650 con 0x7fbc7c0746d0 2024-03-11T19:58:41.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.964+0000 7fbc73e00700 1 -- 172.21.15.97:0/237343606 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c003d50 con 0x7fbc7c0746d0 2024-03-11T19:58:41.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.964+0000 7fbc81000700 1 -- 172.21.15.97:0/237343606 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc6c009520 con 0x7fbc7c0746d0 2024-03-11T19:58:41.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.964+0000 7fbc81000700 1 -- 172.21.15.97:0/237343606 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7c1a6500 con 0x7fbc7c0746d0 2024-03-11T19:58:41.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.965+0000 7fbc81000700 1 -- 172.21.15.97:0/237343606 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc7c104420 con 0x7fbc7c0746d0 2024-03-11T19:58:41.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.967+0000 7fbc73e00700 1 -- 172.21.15.97:0/237343606 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbc6c00b560 con 0x7fbc7c0746d0 2024-03-11T19:58:41.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.969+0000 7fbc73e00700 1 --2- 172.21.15.97:0/237343606 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fbc600a1b40 0x7fbc600a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:41.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.969+0000 7fbc73e00700 1 -- 172.21.15.97:0/237343606 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fbc6c0ce3d0 con 0x7fbc7c0746d0 2024-03-11T19:58:41.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.969+0000 7fbc7aa00700 1 --2- 172.21.15.97:0/237343606 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fbc600a1b40 0x7fbc600a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:41.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.970+0000 7fbc7aa00700 1 --2- 172.21.15.97:0/237343606 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fbc600a1b40 0x7fbc600a4000 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c0759d0 tx=0x7fbc68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:41.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:41.973+0000 7fbc73e00700 1 -- 172.21.15.97:0/237343606 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbc6c097250 con 0x7fbc7c0746d0 2024-03-11T19:58:41.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:41 smithi114 ceph-mon[36376]: pgmap v411: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:41.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:41 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:41.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:41 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:42.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.229+0000 7fbc81000700 1 -- 172.21.15.97:0/237343606 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc7c02d8d0 con 0x7fbc600a1b40 2024-03-11T19:58:42.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.231+0000 7fbc73e00700 1 -- 172.21.15.97:0/237343606 <== mgr.14152 v2:172.21.15.97:6800/3389151509 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7fbc7c02d8d0 con 0x7fbc600a1b40 2024-03-11T19:58:42.230 INFO:teuthology.orchestra.run.smithi097.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-11T19:58:42.230 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204013T400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-03-11T19:58:42.230 INFO:teuthology.orchestra.run.smithi097.stdout:smithi114 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400WB400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-03-11T19:58:42.230 INFO:teuthology.orchestra.run.smithi097.stdout:smithi161 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233001X400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-11T19:58:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.234+0000 7fbc81000700 1 -- 172.21.15.97:0/237343606 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fbc600a1b40 msgr2=0x7fbc600a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.234+0000 7fbc81000700 1 --2- 172.21.15.97:0/237343606 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fbc600a1b40 0x7fbc600a4000 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c0759d0 tx=0x7fbc68009260 comp rx=0 tx=0).stop 2024-03-11T19:58:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.234+0000 7fbc81000700 1 -- 172.21.15.97:0/237343606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc7c0746d0 msgr2=0x7fbc7c19e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.234+0000 7fbc81000700 1 --2- 172.21.15.97:0/237343606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc7c0746d0 0x7fbc7c19e1f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009230 tx=0x7fbc6c005250 comp rx=0 tx=0).stop 2024-03-11T19:58:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.234+0000 7fbc81000700 1 -- 172.21.15.97:0/237343606 shutdown_connections 2024-03-11T19:58:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.234+0000 7fbc81000700 1 --2- 172.21.15.97:0/237343606 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fbc600a1b40 0x7fbc600a4000 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.234+0000 7fbc81000700 1 --2- 172.21.15.97:0/237343606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc7c10e1c0 0x7fbc7c1a57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.234+0000 7fbc81000700 1 --2- 172.21.15.97:0/237343606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc7c0746d0 0x7fbc7c19e1f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.234+0000 7fbc81000700 1 --2- 172.21.15.97:0/237343606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbc7c075ca0 0x7fbc7c19e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:42.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.234+0000 7fbc81000700 1 -- 172.21.15.97:0/237343606 >> 172.21.15.97:0/237343606 conn(0x7fbc7c0fd180 msgr2=0x7fbc7c101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.234+0000 7fbc81000700 1 -- 172.21.15.97:0/237343606 shutdown_connections 2024-03-11T19:58:42.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:42.234+0000 7fbc81000700 1 -- 172.21.15.97:0/237343606 wait complete. 2024-03-11T19:58:43.587 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T19:58:43.596 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-03-11T19:58:43.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-11T19:58:43.665 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:43 smithi097 ceph-mon[31898]: from='client.25294 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:43.665 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:43 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:43.665 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:43 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:43.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:43 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:43.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:43 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:43.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:43 smithi097 ceph-mon[31898]: pgmap v412: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:43.737 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:43 smithi114 ceph-mon[36376]: from='client.25294 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:43.737 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:43 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:43.737 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:43 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:43.737 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:43 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:43.738 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:43 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:43.738 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:43 smithi114 ceph-mon[36376]: pgmap v412: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:43.783 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:43 smithi161 ceph-mon[36332]: from='client.25294 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:58:43.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:43 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:43.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:43 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:43.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:43 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:43.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:43 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:43.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:43 smithi161 ceph-mon[36332]: pgmap v412: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:44.039 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:58:45.788 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:45 smithi097 ceph-mon[31898]: pgmap v413: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:45.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:45 smithi114 ceph-mon[36376]: pgmap v413: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:46.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:45 smithi161 ceph-mon[36332]: pgmap v413: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:46.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.328+0000 7fb18ce00700 1 -- 172.21.15.97:0/2257740953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb188103850 msgr2=0x7fb18810a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:46.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.328+0000 7fb18ce00700 1 --2- 172.21.15.97:0/2257740953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb188103850 0x7fb18810a100 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb180009f90 tx=0x7fb18000b330 comp rx=0 tx=0).stop 2024-03-11T19:58:46.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.329+0000 7fb18ce00700 1 -- 172.21.15.97:0/2257740953 shutdown_connections 2024-03-11T19:58:46.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.329+0000 7fb18ce00700 1 --2- 172.21.15.97:0/2257740953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb188101cb0 0x7fb1881020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:46.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.329+0000 7fb18ce00700 1 --2- 172.21.15.97:0/2257740953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb188103850 0x7fb18810a100 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:46.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.329+0000 7fb18ce00700 1 --2- 172.21.15.97:0/2257740953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb188102eb0 0x7fb188103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:46.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.329+0000 7fb18ce00700 1 -- 172.21.15.97:0/2257740953 >> 172.21.15.97:0/2257740953 conn(0x7fb1880fd210 msgr2=0x7fb1880ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:46.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.329+0000 7fb18ce00700 1 -- 172.21.15.97:0/2257740953 shutdown_connections 2024-03-11T19:58:46.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.329+0000 7fb18ce00700 1 -- 172.21.15.97:0/2257740953 wait complete. 2024-03-11T19:58:46.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.330+0000 7fb18ce00700 1 Processor -- start 2024-03-11T19:58:46.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.330+0000 7fb18ce00700 1 -- start start 2024-03-11T19:58:46.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.331+0000 7fb18ce00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb188101cb0 0x7fb188199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:46.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.331+0000 7fb18ce00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb188102eb0 0x7fb18819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:46.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.331+0000 7fb18ce00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb188103850 0x7fb1881a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:46.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.331+0000 7fb18ce00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1881a1b00 con 0x7fb188101cb0 2024-03-11T19:58:46.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.331+0000 7fb18ce00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1881a1c40 con 0x7fb188102eb0 2024-03-11T19:58:46.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.331+0000 7fb18ce00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1881a1dd0 con 0x7fb188103850 2024-03-11T19:58:46.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.331+0000 7fb185e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb188101cb0 0x7fb188199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:46.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.331+0000 7fb185e00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb188101cb0 0x7fb188199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58270/0 (socket says 172.21.15.97:58270) 2024-03-11T19:58:46.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.331+0000 7fb185400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb188102eb0 0x7fb18819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:46.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.331+0000 7fb185e00700 1 -- 172.21.15.97:0/4204065185 learned_addr learned my addr 172.21.15.97:0/4204065185 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:46.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.331+0000 7fb186800700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb188103850 0x7fb1881a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:46.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.332+0000 7fb185400700 1 -- 172.21.15.97:0/4204065185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb188103850 msgr2=0x7fb1881a1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:46.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.332+0000 7fb185400700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb188103850 0x7fb1881a1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:46.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.332+0000 7fb185400700 1 -- 172.21.15.97:0/4204065185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb188101cb0 msgr2=0x7fb188199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:46.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.332+0000 7fb185400700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb188101cb0 0x7fb188199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:46.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.332+0000 7fb185400700 1 -- 172.21.15.97:0/4204065185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb180009c40 con 0x7fb188102eb0 2024-03-11T19:58:46.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.332+0000 7fb186800700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb188103850 0x7fb1881a1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:46.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.332+0000 7fb185e00700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb188101cb0 0x7fb188199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:46.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.332+0000 7fb185400700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb188102eb0 0x7fb18819a4b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb17400e9f0 tx=0x7fb17400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:46.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.332+0000 7fb17ea00700 1 -- 172.21.15.97:0/4204065185 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17400c9c0 con 0x7fb188102eb0 2024-03-11T19:58:46.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.332+0000 7fb17ea00700 1 -- 172.21.15.97:0/4204065185 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb174010430 con 0x7fb188102eb0 2024-03-11T19:58:46.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.332+0000 7fb18ce00700 1 -- 172.21.15.97:0/4204065185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1881a20b0 con 0x7fb188102eb0 2024-03-11T19:58:46.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.332+0000 7fb17ea00700 1 -- 172.21.15.97:0/4204065185 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb174018680 con 0x7fb188102eb0 2024-03-11T19:58:46.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.332+0000 7fb18ce00700 1 -- 172.21.15.97:0/4204065185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1881a25b0 con 0x7fb188102eb0 2024-03-11T19:58:46.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.334+0000 7fb18ce00700 1 -- 172.21.15.97:0/4204065185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb188068930 con 0x7fb188102eb0 2024-03-11T19:58:46.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.336+0000 7fb17ea00700 1 -- 172.21.15.97:0/4204065185 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb1740187e0 con 0x7fb188102eb0 2024-03-11T19:58:46.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.337+0000 7fb17ea00700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb16c0a1ac0 0x7fb16c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:46.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.337+0000 7fb17ea00700 1 -- 172.21.15.97:0/4204065185 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fb174014070 con 0x7fb188102eb0 2024-03-11T19:58:46.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.337+0000 7fb185e00700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb16c0a1ac0 0x7fb16c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:46.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.338+0000 7fb185e00700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb16c0a1ac0 0x7fb16c0a3f80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb178009fd0 tx=0x7fb178009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:46.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.341+0000 7fb17ea00700 1 -- 172.21.15.97:0/4204065185 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb174052d20 con 0x7fb188102eb0 2024-03-11T19:58:46.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.627+0000 7fb18ce00700 1 -- 172.21.15.97:0/4204065185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb1881a2990 con 0x7fb188102eb0 2024-03-11T19:58:46.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.629+0000 7fb17ea00700 1 -- 172.21.15.97:0/4204065185 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+2977 (secure 0 0 0) 0x7fb17409a220 con 0x7fb188102eb0 2024-03-11T19:58:46.628 INFO:teuthology.orchestra.run.smithi097.stdout:e28 2024-03-11T19:58:46.628 INFO:teuthology.orchestra.run.smithi097.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T19:58:46.628 INFO:teuthology.orchestra.run.smithi097.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T19:58:46.628 INFO:teuthology.orchestra.run.smithi097.stdout:legacy client fscid: 1 2024-03-11T19:58:46.628 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:Filesystem 'cephfs' (1) 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:fs_name cephfs 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:epoch 28 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:created 2024-03-11T19:55:45.149335+0000 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:modified 2024-03-11T19:56:19.631638+0000 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:tableserver 0 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:root 0 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:session_timeout 300 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:session_autoclose 300 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:max_file_size 1099511627776 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:max_xattr_size 65536 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:required_client_features {} 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:last_failure 0 2024-03-11T19:58:46.629 INFO:teuthology.orchestra.run.smithi097.stdout:last_failure_osd_epoch 0 2024-03-11T19:58:46.630 INFO:teuthology.orchestra.run.smithi097.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T19:58:46.630 INFO:teuthology.orchestra.run.smithi097.stdout:max_mds 5 2024-03-11T19:58:46.630 INFO:teuthology.orchestra.run.smithi097.stdout:in 0,1,2,3,4 2024-03-11T19:58:46.630 INFO:teuthology.orchestra.run.smithi097.stdout:up {0=24599,1=24583,2=24547,3=24545,4=24464} 2024-03-11T19:58:46.630 INFO:teuthology.orchestra.run.smithi097.stdout:failed 2024-03-11T19:58:46.630 INFO:teuthology.orchestra.run.smithi097.stdout:damaged 2024-03-11T19:58:46.630 INFO:teuthology.orchestra.run.smithi097.stdout:stopped 2024-03-11T19:58:46.630 INFO:teuthology.orchestra.run.smithi097.stdout:data_pools [3,4] 2024-03-11T19:58:46.630 INFO:teuthology.orchestra.run.smithi097.stdout:metadata_pool 2 2024-03-11T19:58:46.630 INFO:teuthology.orchestra.run.smithi097.stdout:inline_data disabled 2024-03-11T19:58:46.630 INFO:teuthology.orchestra.run.smithi097.stdout:balancer 2024-03-11T19:58:46.630 INFO:teuthology.orchestra.run.smithi097.stdout:bal_rank_mask -1 2024-03-11T19:58:46.630 INFO:teuthology.orchestra.run.smithi097.stdout:standby_count_wanted 0 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.b{0:24599} state up:active seq 3 addr [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.c{0:24529} state up:standby-replay seq 1 addr [v2:172.21.15.161:6836/1600156948,v1:172.21.15.161:6837/1600156948] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.a{1:24583} state up:active seq 11 addr [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.f{1:24517} state up:standby-replay seq 1 addr [v2:172.21.15.161:6834/3358622196,v1:172.21.15.161:6835/3358622196] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.i{2:24547} state up:active seq 14 addr [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.e{2:24512} state up:standby-replay seq 1 addr [v2:172.21.15.114:6834/2623690462,v1:172.21.15.114:6835/2623690462] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.h{3:24545} state up:active seq 15 addr [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.d{3:24499} state up:standby-replay seq 1 addr [v2:172.21.15.97:6836/267259401,v1:172.21.15.97:6837/267259401] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.k{4:24464} state up:active seq 20 addr [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.l{4:24454} state up:standby-replay seq 1 addr [v2:172.21.15.161:6832/2213740234,v1:172.21.15.161:6833/2213740234] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout:Standby daemons: 2024-03-11T19:58:46.631 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:46.632 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.j{-1:14391} state up:standby seq 1 addr [v2:172.21.15.97:6834/207363921,v1:172.21.15.97:6835/207363921] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:58:46.632 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.g{-1:14454} state up:standby seq 1 addr [v2:172.21.15.97:6838/2355075568,v1:172.21.15.97:6839/2355075568] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T19:58:46.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.632+0000 7fb18ce00700 1 -- 172.21.15.97:0/4204065185 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb16c0a1ac0 msgr2=0x7fb16c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:46.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.632+0000 7fb18ce00700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb16c0a1ac0 0x7fb16c0a3f80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb178009fd0 tx=0x7fb178009310 comp rx=0 tx=0).stop 2024-03-11T19:58:46.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.632+0000 7fb18ce00700 1 -- 172.21.15.97:0/4204065185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb188102eb0 msgr2=0x7fb18819a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:46.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.632+0000 7fb18ce00700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb188102eb0 0x7fb18819a4b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb17400e9f0 tx=0x7fb17400ed00 comp rx=0 tx=0).stop 2024-03-11T19:58:46.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.632+0000 7fb18ce00700 1 -- 172.21.15.97:0/4204065185 shutdown_connections 2024-03-11T19:58:46.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.632+0000 7fb18ce00700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fb16c0a1ac0 0x7fb16c0a3f80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:46.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.632+0000 7fb18ce00700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb188101cb0 0x7fb188199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:46.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.632+0000 7fb18ce00700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb188103850 0x7fb1881a1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:46.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.632+0000 7fb18ce00700 1 --2- 172.21.15.97:0/4204065185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb188102eb0 0x7fb18819a4b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:46.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.632+0000 7fb18ce00700 1 -- 172.21.15.97:0/4204065185 >> 172.21.15.97:0/4204065185 conn(0x7fb1880fd210 msgr2=0x7fb188073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:46.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.632+0000 7fb18ce00700 1 -- 172.21.15.97:0/4204065185 shutdown_connections 2024-03-11T19:58:46.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:46.633+0000 7fb18ce00700 1 -- 172.21.15.97:0/4204065185 wait complete. 2024-03-11T19:58:46.633 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-03-11T19:58:47.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-11T19:58:47.629 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:47 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:47.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:47 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:47.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:47 smithi097 ceph-mon[31898]: pgmap v414: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:47.630 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:47 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/4204065185' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-11T19:58:47.737 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:47 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:47.738 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:47 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:47.738 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:47 smithi114 ceph-mon[36376]: pgmap v414: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:47.738 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:47 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/4204065185' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-11T19:58:47.783 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:47 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:47.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:47 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:47.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:47 smithi161 ceph-mon[36332]: pgmap v414: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:47.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:47 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/4204065185' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-11T19:58:47.970 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:58:48.665 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:58:48.665 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:58:48.665 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:48 smithi097 ceph-mon[31898]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:48.737 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:58:48.737 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:58:48.737 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:48 smithi114 ceph-mon[36376]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:48.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:48 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:58:48.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:48 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:58:48.784 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:48 smithi161 ceph-mon[36332]: from='mgr.14152 172.21.15.97:0/3716618585' entity='mgr.x' 2024-03-11T19:58:49.665 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:49 smithi097 ceph-mon[31898]: pgmap v415: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:49.737 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:49 smithi114 ceph-mon[36376]: pgmap v415: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:49.783 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:49 smithi161 ceph-mon[36332]: pgmap v415: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:50.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.427+0000 7efd2ba00700 1 -- 172.21.15.97:0/3017255035 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd24101cb0 msgr2=0x7efd241020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:50.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.427+0000 7efd2ba00700 1 --2- 172.21.15.97:0/3017255035 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd24101cb0 0x7efd241020d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efd180099c0 tx=0x7efd18009cd0 comp rx=0 tx=0).stop 2024-03-11T19:58:50.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.427+0000 7efd2ba00700 1 -- 172.21.15.97:0/3017255035 shutdown_connections 2024-03-11T19:58:50.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.427+0000 7efd2ba00700 1 --2- 172.21.15.97:0/3017255035 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd24101cb0 0x7efd241020d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:50.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.427+0000 7efd2ba00700 1 --2- 172.21.15.97:0/3017255035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd24103850 0x7efd2410a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:50.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.427+0000 7efd2ba00700 1 --2- 172.21.15.97:0/3017255035 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd24102eb0 0x7efd24103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:50.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.427+0000 7efd2ba00700 1 -- 172.21.15.97:0/3017255035 >> 172.21.15.97:0/3017255035 conn(0x7efd240fd210 msgr2=0x7efd240ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:50.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.427+0000 7efd2ba00700 1 -- 172.21.15.97:0/3017255035 shutdown_connections 2024-03-11T19:58:50.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.428+0000 7efd2ba00700 1 -- 172.21.15.97:0/3017255035 wait complete. 2024-03-11T19:58:50.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.429+0000 7efd2ba00700 1 Processor -- start 2024-03-11T19:58:50.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.429+0000 7efd2ba00700 1 -- start start 2024-03-11T19:58:50.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.429+0000 7efd2ba00700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd24101cb0 0x7efd24199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:50.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.429+0000 7efd2ba00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd24102eb0 0x7efd2419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:50.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.429+0000 7efd2ba00700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd24103850 0x7efd241a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:50.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.429+0000 7efd2ba00700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd241a1a40 con 0x7efd24103850 2024-03-11T19:58:50.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.429+0000 7efd2ba00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd241a1b80 con 0x7efd24102eb0 2024-03-11T19:58:50.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.429+0000 7efd2ba00700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd241a1d10 con 0x7efd24101cb0 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.429+0000 7efd23e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd24102eb0 0x7efd2419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.429+0000 7efd29000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd24101cb0 0x7efd24199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.429+0000 7efd23e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd24102eb0 0x7efd2419a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:57584/0 (socket says 172.21.15.97:57584) 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.429+0000 7efd29000700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd24101cb0 0x7efd24199f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:40082/0 (socket says 172.21.15.97:40082) 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.430+0000 7efd23e00700 1 -- 172.21.15.97:0/227991110 learned_addr learned my addr 172.21.15.97:0/227991110 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.430+0000 7efd29a00700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd24103850 0x7efd241a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.430+0000 7efd29000700 1 -- 172.21.15.97:0/227991110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd24102eb0 msgr2=0x7efd2419a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.430+0000 7efd29000700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd24102eb0 0x7efd2419a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.430+0000 7efd29000700 1 -- 172.21.15.97:0/227991110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd24103850 msgr2=0x7efd241a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.430+0000 7efd29000700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd24103850 0x7efd241a1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.430+0000 7efd29000700 1 -- 172.21.15.97:0/227991110 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd180095f0 con 0x7efd24101cb0 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.430+0000 7efd23e00700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd24102eb0 0x7efd2419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.430+0000 7efd29000700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd24101cb0 0x7efd24199f00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efd18003760 tx=0x7efd1800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:50.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.431+0000 7efd21600700 1 -- 172.21.15.97:0/227991110 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1801d070 con 0x7efd24101cb0 2024-03-11T19:58:50.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.431+0000 7efd21600700 1 -- 172.21.15.97:0/227991110 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd18022470 con 0x7efd24101cb0 2024-03-11T19:58:50.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.431+0000 7efd2ba00700 1 -- 172.21.15.97:0/227991110 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd241a1f90 con 0x7efd24101cb0 2024-03-11T19:58:50.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.431+0000 7efd21600700 1 -- 172.21.15.97:0/227991110 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18017650 con 0x7efd24101cb0 2024-03-11T19:58:50.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.431+0000 7efd2ba00700 1 -- 172.21.15.97:0/227991110 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd241a22f0 con 0x7efd24101cb0 2024-03-11T19:58:50.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.432+0000 7efd2ba00700 1 -- 172.21.15.97:0/227991110 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd2410d640 con 0x7efd24101cb0 2024-03-11T19:58:50.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.434+0000 7efd21600700 1 -- 172.21.15.97:0/227991110 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7efd180225e0 con 0x7efd24101cb0 2024-03-11T19:58:50.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.435+0000 7efd21600700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efd0c0a1ac0 0x7efd0c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:50.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.435+0000 7efd21600700 1 -- 172.21.15.97:0/227991110 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7efd180ceff0 con 0x7efd24101cb0 2024-03-11T19:58:50.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.436+0000 7efd23e00700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efd0c0a1ac0 0x7efd0c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:50.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.436+0000 7efd23e00700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efd0c0a1ac0 0x7efd0c0a3f80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7efd10011550 tx=0x7efd10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:50.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.437+0000 7efd21600700 1 -- 172.21.15.97:0/227991110 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efd18097f90 con 0x7efd24101cb0 2024-03-11T19:58:50.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.658+0000 7efd2ba00700 1 -- 172.21.15.97:0/227991110 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7efd2410d7d0 con 0x7efd24101cb0 2024-03-11T19:58:50.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.660+0000 7efd21600700 1 -- 172.21.15.97:0/227991110 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) v1 ==== 56+0+5692 (secure 0 0 0) 0x7efd1809b650 con 0x7efd24101cb0 2024-03-11T19:58:50.658 INFO:teuthology.orchestra.run.smithi097.stdout:epoch 93 2024-03-11T19:58:50.658 INFO:teuthology.orchestra.run.smithi097.stdout:fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T19:58:50.658 INFO:teuthology.orchestra.run.smithi097.stdout:created 2024-03-11T19:46:35.573122+0000 2024-03-11T19:58:50.658 INFO:teuthology.orchestra.run.smithi097.stdout:modified 2024-03-11T19:57:46.674203+0000 2024-03-11T19:58:50.658 INFO:teuthology.orchestra.run.smithi097.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-11T19:58:50.658 INFO:teuthology.orchestra.run.smithi097.stdout:crush_version 27 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:full_ratio 0.95 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:backfillfull_ratio 0.9 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:nearfull_ratio 0.85 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:require_min_compat_client luminous 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:min_compat_client luminous 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:require_osd_release squid 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:stretch_mode_enabled false 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:max_osd 12 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6802/2786748520,v1:172.21.15.97:6803/2786748520] [v2:172.21.15.97:6804/2786748520,v1:172.21.15.97:6805/2786748520] exists,up aaa3b17b-f7a0-4cf1-857c-5e3415f224a9 2024-03-11T19:58:50.659 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6800/2861651844,v1:172.21.15.114:6801/2861651844] [v2:172.21.15.114:6802/2861651844,v1:172.21.15.114:6803/2861651844] exists,up 259f580e-e340-4048-a461-5da3946ed1d7 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6800/4070161993,v1:172.21.15.161:6801/4070161993] [v2:172.21.15.161:6802/4070161993,v1:172.21.15.161:6803/4070161993] exists,up 64012698-9df0-4d30-b4e4-1bebdbde0a0c 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6810/2971789214,v1:172.21.15.97:6811/2971789214] [v2:172.21.15.97:6812/2971789214,v1:172.21.15.97:6813/2971789214] exists,up 794e7a63-96da-4d6e-b020-29d61db19359 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 up in weight 1 up_from 33 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6808/1414652177,v1:172.21.15.114:6809/1414652177] [v2:172.21.15.114:6810/1414652177,v1:172.21.15.114:6811/1414652177] exists,up 601a3bf7-cfee-4e6d-a838-7f04eccdb73f 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6808/2197634465,v1:172.21.15.161:6809/2197634465] [v2:172.21.15.161:6810/2197634465,v1:172.21.15.161:6811/2197634465] exists,up 0ce520f9-2aef-4901-aa82-edd7ca858028 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 up in weight 1 up_from 46 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6818/2094652517,v1:172.21.15.97:6819/2094652517] [v2:172.21.15.97:6820/2094652517,v1:172.21.15.97:6821/2094652517] exists,up 609b0dd7-7cff-4f8e-b656-2a11e7652150 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 up in weight 1 up_from 51 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6816/3855344963,v1:172.21.15.114:6817/3855344963] [v2:172.21.15.114:6818/3855344963,v1:172.21.15.114:6819/3855344963] exists,up 5a0d4de9-3628-4421-91b0-0c8cb7cdcf93 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:osd.8 up in weight 1 up_from 59 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6816/4061144818,v1:172.21.15.161:6817/4061144818] [v2:172.21.15.161:6818/4061144818,v1:172.21.15.161:6819/4061144818] exists,up 2c0abbf6-d848-4911-beff-7b4d3ce7b330 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.97:6826/2375353428,v1:172.21.15.97:6827/2375353428] [v2:172.21.15.97:6828/2375353428,v1:172.21.15.97:6829/2375353428] exists,up d55f416c-9d0f-4938-8e34-b364abeaedcc 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6824/275586537,v1:172.21.15.114:6825/275586537] [v2:172.21.15.114:6826/275586537,v1:172.21.15.114:6827/275586537] exists,up 27d60b08-8d2b-453e-841d-abb1339ce6e3 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6824/1523383703,v1:172.21.15.161:6825/1523383703] [v2:172.21.15.161:6826/1523383703,v1:172.21.15.161:6827/1523383703] exists,up f16c85e3-0e4a-4ae3-894f-6dfe489e7325 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 2.e [2,1] 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 2.1c [8,11] 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 2.1d [2,4,8,11] 2024-03-11T19:58:50.660 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 3.2 [3,9] 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 3.e [4,8] 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 3.17 [4,7] 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 3.29 [3,9] 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 3.2c [4,9] 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 3.39 [5,6] 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 4.5 [5,10] 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 4.10 [9,8] 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 4.16 [9,8] 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:pg_upmap_items 4.1b [9,8] 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:0/4129219534 expires 2024-03-12T19:47:44.356323+0000 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:0/3932792931 expires 2024-03-12T19:47:44.356323+0000 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:6801/1708150324 expires 2024-03-12T19:47:44.356323+0000 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:6800/1708150324 expires 2024-03-12T19:47:44.356323+0000 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:6800/3216682632 expires 2024-03-12T19:47:09.368819+0000 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:0/3236990094 expires 2024-03-12T19:47:09.368819+0000 2024-03-11T19:58:50.661 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:0/1301912133 expires 2024-03-12T19:47:09.368819+0000 2024-03-11T19:58:50.662 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:6801/3216682632 expires 2024-03-12T19:47:09.368819+0000 2024-03-11T19:58:50.662 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:0/46019613 expires 2024-03-12T19:47:44.356323+0000 2024-03-11T19:58:50.662 INFO:teuthology.orchestra.run.smithi097.stdout:blocklist 172.21.15.97:0/4194903751 expires 2024-03-12T19:47:09.368819+0000 2024-03-11T19:58:50.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.663+0000 7efd2ba00700 1 -- 172.21.15.97:0/227991110 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efd0c0a1ac0 msgr2=0x7efd0c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:50.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.663+0000 7efd2ba00700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efd0c0a1ac0 0x7efd0c0a3f80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7efd10011550 tx=0x7efd10009260 comp rx=0 tx=0).stop 2024-03-11T19:58:50.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.663+0000 7efd2ba00700 1 -- 172.21.15.97:0/227991110 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd24101cb0 msgr2=0x7efd24199f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.663+0000 7efd2ba00700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd24101cb0 0x7efd24199f00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efd18003760 tx=0x7efd1800f690 comp rx=0 tx=0).stop 2024-03-11T19:58:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.663+0000 7efd2ba00700 1 -- 172.21.15.97:0/227991110 shutdown_connections 2024-03-11T19:58:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.663+0000 7efd2ba00700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7efd0c0a1ac0 0x7efd0c0a3f80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.663+0000 7efd2ba00700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7efd24103850 0x7efd241a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.663+0000 7efd2ba00700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd24101cb0 0x7efd24199f00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.663+0000 7efd2ba00700 1 --2- 172.21.15.97:0/227991110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd24102eb0 0x7efd2419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.663+0000 7efd2ba00700 1 -- 172.21.15.97:0/227991110 >> 172.21.15.97:0/227991110 conn(0x7efd240fd210 msgr2=0x7efd24108650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.663+0000 7efd2ba00700 1 -- 172.21.15.97:0/227991110 shutdown_connections 2024-03-11T19:58:50.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:50.663+0000 7efd2ba00700 1 -- 172.21.15.97:0/227991110 wait complete. 2024-03-11T19:58:51.606 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-11T19:58:51.620 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:58:51.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.841+0000 7f239aa00640 1 -- 172.21.15.97:0/3410647310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23941033c0 msgr2=0x7f23941037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:51.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.841+0000 7f239aa00640 1 --2- 172.21.15.97:0/3410647310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23941033c0 0x7f23941037e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2384009900 tx=0x7f238402bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:51.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.841+0000 7f239aa00640 1 -- 172.21.15.97:0/3410647310 shutdown_connections 2024-03-11T19:58:51.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.841+0000 7f239aa00640 1 --2- 172.21.15.97:0/3410647310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2394104f70 0x7f239410b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:51.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.841+0000 7f239aa00640 1 --2- 172.21.15.97:0/3410647310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23941045b0 0x7f2394104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:51.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.841+0000 7f239aa00640 1 --2- 172.21.15.97:0/3410647310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23941033c0 0x7f23941037e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:51.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.841+0000 7f239aa00640 1 -- 172.21.15.97:0/3410647310 >> 172.21.15.97:0/3410647310 conn(0x7f23940fe970 msgr2=0x7f2394100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:51.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.841+0000 7f239aa00640 1 -- 172.21.15.97:0/3410647310 shutdown_connections 2024-03-11T19:58:51.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.841+0000 7f239aa00640 1 -- 172.21.15.97:0/3410647310 wait complete. 2024-03-11T19:58:51.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.842+0000 7f239aa00640 1 Processor -- start 2024-03-11T19:58:51.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.842+0000 7f239aa00640 1 -- start start 2024-03-11T19:58:51.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.842+0000 7f239aa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23941033c0 0x7f2394199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:51.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.842+0000 7f239aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23941045b0 0x7f23941999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:51.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.842+0000 7f239aa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2394104f70 0x7f23941a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:51.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.842+0000 7f239aa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23941a0f70 con 0x7f23941033c0 2024-03-11T19:58:51.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.842+0000 7f239aa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23941a10e0 con 0x7f23941045b0 2024-03-11T19:58:51.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.842+0000 7f239aa00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23941a12a0 con 0x7f2394104f70 2024-03-11T19:58:51.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.842+0000 7f2393e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23941033c0 0x7f2394199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:51.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.842+0000 7f2398a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2394104f70 0x7f23941a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:51.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.843+0000 7f2393e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23941033c0 0x7f2394199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58300/0 (socket says 172.21.15.97:58300) 2024-03-11T19:58:51.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.843+0000 7f2393e00640 1 -- 172.21.15.97:0/4164707155 learned_addr learned my addr 172.21.15.97:0/4164707155 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:51.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.843+0000 7f238b400640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23941045b0 0x7f23941999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:51.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.843+0000 7f2398a00640 1 -- 172.21.15.97:0/4164707155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23941045b0 msgr2=0x7f23941999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:51.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.843+0000 7f2398a00640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23941045b0 0x7f23941999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:51.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.843+0000 7f2398a00640 1 -- 172.21.15.97:0/4164707155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23941033c0 msgr2=0x7f2394199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:51.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.843+0000 7f2398a00640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23941033c0 0x7f2394199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:51.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.843+0000 7f2398a00640 1 -- 172.21.15.97:0/4164707155 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2384009560 con 0x7f2394104f70 2024-03-11T19:58:51.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.843+0000 7f2393e00640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23941033c0 0x7f2394199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:51.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.843+0000 7f2398a00640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2394104f70 0x7f23941a0a30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f238c009fd0 tx=0x7f238c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:51.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.843+0000 7f2391600640 1 -- 172.21.15.97:0/4164707155 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c00ce20 con 0x7f2394104f70 2024-03-11T19:58:51.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.844+0000 7f239aa00640 1 -- 172.21.15.97:0/4164707155 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23941a1580 con 0x7f2394104f70 2024-03-11T19:58:51.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.844+0000 7f2391600640 1 -- 172.21.15.97:0/4164707155 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f238c005020 con 0x7f2394104f70 2024-03-11T19:58:51.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.844+0000 7f2391600640 1 -- 172.21.15.97:0/4164707155 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c010690 con 0x7f2394104f70 2024-03-11T19:58:51.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.844+0000 7f239aa00640 1 -- 172.21.15.97:0/4164707155 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23941a1a90 con 0x7f2394104f70 2024-03-11T19:58:51.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.845+0000 7f239aa00640 1 -- 172.21.15.97:0/4164707155 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239410a6e0 con 0x7f2394104f70 2024-03-11T19:58:51.844 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.847+0000 7f2391600640 1 -- 172.21.15.97:0/4164707155 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f238c005500 con 0x7f2394104f70 2024-03-11T19:58:51.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.848+0000 7f2391600640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f23780a1a30 0x7f23780a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:51.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.848+0000 7f2391600640 1 -- 172.21.15.97:0/4164707155 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f238c014070 con 0x7f2394104f70 2024-03-11T19:58:51.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.848+0000 7f2393e00640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f23780a1a30 0x7f23780a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:51.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.849+0000 7f2393e00640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f23780a1a30 0x7f23780a3ef0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f238402c840 tx=0x7f238402cd10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:51.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:51.852+0000 7f2391600640 1 -- 172.21.15.97:0/4164707155 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f238c096d00 con 0x7f2394104f70 2024-03-11T19:58:51.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:51 smithi097 ceph-mon[31898]: pgmap v416: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:51.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:51 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/227991110' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T19:58:51.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:51 smithi114 ceph-mon[36376]: pgmap v416: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:51.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:51 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/227991110' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T19:58:52.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:51 smithi161 ceph-mon[36332]: pgmap v416: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:58:52.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:51 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/227991110' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T19:58:52.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.054+0000 7f239aa00640 1 -- 172.21.15.97:0/4164707155 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f239402ce40 con 0x7f2394104f70 2024-03-11T19:58:52.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.056+0000 7f2391600640 1 -- 172.21.15.97:0/4164707155 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24838 (secure 0 0 0) 0x7f238c005190 con 0x7f2394104f70 2024-03-11T19:58:52.055 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:52.055 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":93,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:57:46.674203+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:58:52.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.059+0000 7f239aa00640 1 -- 172.21.15.97:0/4164707155 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f23780a1a30 msgr2=0x7f23780a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:52.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.059+0000 7f239aa00640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f23780a1a30 0x7f23780a3ef0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f238402c840 tx=0x7f238402cd10 comp rx=0 tx=0).stop 2024-03-11T19:58:52.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.059+0000 7f239aa00640 1 -- 172.21.15.97:0/4164707155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2394104f70 msgr2=0x7f23941a0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:52.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.059+0000 7f239aa00640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2394104f70 0x7f23941a0a30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f238c009fd0 tx=0x7f238c00ef10 comp rx=0 tx=0).stop 2024-03-11T19:58:52.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.059+0000 7f239aa00640 1 -- 172.21.15.97:0/4164707155 shutdown_connections 2024-03-11T19:58:52.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.059+0000 7f239aa00640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2394104f70 0x7f23941a0a30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.059+0000 7f239aa00640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23941045b0 0x7f23941999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.059+0000 7f239aa00640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f23780a1a30 0x7f23780a3ef0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.059+0000 7f239aa00640 1 --2- 172.21.15.97:0/4164707155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23941033c0 0x7f2394199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.059+0000 7f239aa00640 1 -- 172.21.15.97:0/4164707155 >> 172.21.15.97:0/4164707155 conn(0x7f23940fe970 msgr2=0x7f23941078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:52.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.060+0000 7f239aa00640 1 -- 172.21.15.97:0/4164707155 shutdown_connections 2024-03-11T19:58:52.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.060+0000 7f239aa00640 1 -- 172.21.15.97:0/4164707155 wait complete. 2024-03-11T19:58:52.077 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:50:29.051985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:55:43.424076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:55:44.652160+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T19:55:48.598077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T19:58:52.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:58:52.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.300+0000 7fd662000640 1 -- 172.21.15.97:0/747861740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd65c109350 msgr2=0x7fd65c10fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:52.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.300+0000 7fd662000640 1 --2- 172.21.15.97:0/747861740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd65c109350 0x7fd65c10fbe0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd654009be0 tx=0x7fd65402bef0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.301+0000 7fd662000640 1 -- 172.21.15.97:0/747861740 shutdown_connections 2024-03-11T19:58:52.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.301+0000 7fd662000640 1 --2- 172.21.15.97:0/747861740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd65c109350 0x7fd65c10fbe0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.301+0000 7fd662000640 1 --2- 172.21.15.97:0/747861740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd65c108990 0x7fd65c108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.301+0000 7fd662000640 1 --2- 172.21.15.97:0/747861740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd65c1077a0 0x7fd65c107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.301+0000 7fd662000640 1 -- 172.21.15.97:0/747861740 >> 172.21.15.97:0/747861740 conn(0x7fd65c076740 msgr2=0x7fd65c078b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:52.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.301+0000 7fd662000640 1 -- 172.21.15.97:0/747861740 shutdown_connections 2024-03-11T19:58:52.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.301+0000 7fd662000640 1 -- 172.21.15.97:0/747861740 wait complete. 2024-03-11T19:58:52.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.302+0000 7fd662000640 1 Processor -- start 2024-03-11T19:58:52.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.302+0000 7fd662000640 1 -- start start 2024-03-11T19:58:52.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.302+0000 7fd662000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd65c1077a0 0x7fd65c19d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:52.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.302+0000 7fd662000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd65c108990 0x7fd65c19dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:52.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.302+0000 7fd662000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd65c109350 0x7fd65c1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:52.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.302+0000 7fd662000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c1a5280 con 0x7fd65c108990 2024-03-11T19:58:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.302+0000 7fd662000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c1a53f0 con 0x7fd65c109350 2024-03-11T19:58:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.302+0000 7fd662000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65c1a55b0 con 0x7fd65c1077a0 2024-03-11T19:58:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.302+0000 7fd652a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd65c108990 0x7fd65c19dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.302+0000 7fd65be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd65c109350 0x7fd65c1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.302+0000 7fd652a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd65c108990 0x7fd65c19dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58324/0 (socket says 172.21.15.97:58324) 2024-03-11T19:58:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.302+0000 7fd652a00640 1 -- 172.21.15.97:0/3122408902 learned_addr learned my addr 172.21.15.97:0/3122408902 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.302+0000 7fd65b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd65c1077a0 0x7fd65c19d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.303+0000 7fd652a00640 1 -- 172.21.15.97:0/3122408902 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd65c1077a0 msgr2=0x7fd65c19d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.303+0000 7fd652a00640 1 --2- 172.21.15.97:0/3122408902 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd65c1077a0 0x7fd65c19d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.303+0000 7fd652a00640 1 -- 172.21.15.97:0/3122408902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd65c109350 msgr2=0x7fd65c1a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.303+0000 7fd652a00640 1 --2- 172.21.15.97:0/3122408902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd65c109350 0x7fd65c1a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.303+0000 7fd652a00640 1 -- 172.21.15.97:0/3122408902 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c009560 con 0x7fd65c108990 2024-03-11T19:58:52.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.303+0000 7fd65be00640 1 --2- 172.21.15.97:0/3122408902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd65c109350 0x7fd65c1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:52.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.303+0000 7fd652a00640 1 --2- 172.21.15.97:0/3122408902 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd65c108990 0x7fd65c19dcd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd64400e9f0 tx=0x7fd64400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:52.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.303+0000 7fd658c00640 1 -- 172.21.15.97:0/3122408902 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd644009720 con 0x7fd65c108990 2024-03-11T19:58:52.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.303+0000 7fd662000640 1 -- 172.21.15.97:0/3122408902 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd654009840 con 0x7fd65c108990 2024-03-11T19:58:52.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.303+0000 7fd658c00640 1 -- 172.21.15.97:0/3122408902 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd64400cd80 con 0x7fd65c108990 2024-03-11T19:58:52.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.303+0000 7fd658c00640 1 -- 172.21.15.97:0/3122408902 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd644010690 con 0x7fd65c108990 2024-03-11T19:58:52.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.304+0000 7fd662000640 1 -- 172.21.15.97:0/3122408902 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd65c1a5bc0 con 0x7fd65c108990 2024-03-11T19:58:52.303 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.305+0000 7fd662000640 1 -- 172.21.15.97:0/3122408902 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd65c10eac0 con 0x7fd65c108990 2024-03-11T19:58:52.304 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.307+0000 7fd658c00640 1 -- 172.21.15.97:0/3122408902 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd6440046c0 con 0x7fd65c108990 2024-03-11T19:58:52.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.308+0000 7fd658c00640 1 --2- 172.21.15.97:0/3122408902 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd6400a1a30 0x7fd6400a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:52.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.308+0000 7fd658c00640 1 -- 172.21.15.97:0/3122408902 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fd644014070 con 0x7fd65c108990 2024-03-11T19:58:52.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.308+0000 7fd65b400640 1 --2- 172.21.15.97:0/3122408902 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd6400a1a30 0x7fd6400a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:52.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.309+0000 7fd65b400640 1 --2- 172.21.15.97:0/3122408902 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd6400a1a30 0x7fd6400a3ef0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009530 tx=0x7fd64c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:52.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.311+0000 7fd658c00640 1 -- 172.21.15.97:0/3122408902 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd644096ba0 con 0x7fd65c108990 2024-03-11T19:58:52.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.506+0000 7fd662000640 1 -- 172.21.15.97:0/3122408902 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd65c113330 con 0x7fd65c108990 2024-03-11T19:58:52.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.507+0000 7fd658c00640 1 -- 172.21.15.97:0/3122408902 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd65c113330 con 0x7fd65c108990 2024-03-11T19:58:52.504 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-11T19:58:52.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.509+0000 7fd662000640 1 -- 172.21.15.97:0/3122408902 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd6400a1a30 msgr2=0x7fd6400a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:52.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.509+0000 7fd662000640 1 --2- 172.21.15.97:0/3122408902 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd6400a1a30 0x7fd6400a3ef0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009530 tx=0x7fd64c009260 comp rx=0 tx=0).stop 2024-03-11T19:58:52.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.509+0000 7fd662000640 1 -- 172.21.15.97:0/3122408902 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd65c108990 msgr2=0x7fd65c19dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:52.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.509+0000 7fd662000640 1 --2- 172.21.15.97:0/3122408902 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd65c108990 0x7fd65c19dcd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd64400e9f0 tx=0x7fd64400edf0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.509+0000 7fd662000640 1 -- 172.21.15.97:0/3122408902 shutdown_connections 2024-03-11T19:58:52.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.509+0000 7fd662000640 1 --2- 172.21.15.97:0/3122408902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd65c109350 0x7fd65c1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.509+0000 7fd662000640 1 --2- 172.21.15.97:0/3122408902 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fd6400a1a30 0x7fd6400a3ef0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.509+0000 7fd662000640 1 --2- 172.21.15.97:0/3122408902 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd65c108990 0x7fd65c19dcd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.509+0000 7fd662000640 1 --2- 172.21.15.97:0/3122408902 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd65c1077a0 0x7fd65c19d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.509+0000 7fd662000640 1 -- 172.21.15.97:0/3122408902 >> 172.21.15.97:0/3122408902 conn(0x7fd65c076740 msgr2=0x7fd65c10bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:52.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.509+0000 7fd662000640 1 -- 172.21.15.97:0/3122408902 shutdown_connections 2024-03-11T19:58:52.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.509+0000 7fd662000640 1 -- 172.21.15.97:0/3122408902 wait complete. 2024-03-11T19:58:52.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:58:52.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.748+0000 7faf14c00640 1 -- 172.21.15.97:0/372076847 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf101045b0 msgr2=0x7faf10104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:52.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.748+0000 7faf14c00640 1 --2- 172.21.15.97:0/372076847 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf101045b0 0x7faf10104a30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faef8009900 tx=0x7faef802bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:52.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.748+0000 7faf14c00640 1 -- 172.21.15.97:0/372076847 shutdown_connections 2024-03-11T19:58:52.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.748+0000 7faf14c00640 1 --2- 172.21.15.97:0/372076847 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf10104f70 0x7faf1010b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.748+0000 7faf14c00640 1 --2- 172.21.15.97:0/372076847 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf101045b0 0x7faf10104a30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.748+0000 7faf14c00640 1 --2- 172.21.15.97:0/372076847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf101033c0 0x7faf101037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.748+0000 7faf14c00640 1 -- 172.21.15.97:0/372076847 >> 172.21.15.97:0/372076847 conn(0x7faf100fe930 msgr2=0x7faf10100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:52.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.749+0000 7faf14c00640 1 -- 172.21.15.97:0/372076847 shutdown_connections 2024-03-11T19:58:52.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.749+0000 7faf14c00640 1 -- 172.21.15.97:0/372076847 wait complete. 2024-03-11T19:58:52.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.750+0000 7faf14c00640 1 Processor -- start 2024-03-11T19:58:52.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.750+0000 7faf14c00640 1 -- start start 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.750+0000 7faf14c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf101033c0 0x7faf101994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.750+0000 7faf14c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf101045b0 0x7faf101999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.750+0000 7faf14c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf10104f70 0x7faf101a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.750+0000 7faf14c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf101a0fa0 con 0x7faf101033c0 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.750+0000 7faf14c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf101a1110 con 0x7faf10104f70 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.750+0000 7faf14c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf101a12d0 con 0x7faf101045b0 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.750+0000 7faf0de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf101033c0 0x7faf101994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.751+0000 7faf0de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf101033c0 0x7faf101994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58342/0 (socket says 172.21.15.97:58342) 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.751+0000 7faf0de00640 1 -- 172.21.15.97:0/2893812479 learned_addr learned my addr 172.21.15.97:0/2893812479 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.751+0000 7faf0d400640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf101045b0 0x7faf101999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.751+0000 7faf0e800640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf10104f70 0x7faf101a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.751+0000 7faf0de00640 1 -- 172.21.15.97:0/2893812479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf101045b0 msgr2=0x7faf101999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.751+0000 7faf0de00640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf101045b0 0x7faf101999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.751+0000 7faf0de00640 1 -- 172.21.15.97:0/2893812479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf10104f70 msgr2=0x7faf101a0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:52.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.751+0000 7faf0de00640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf10104f70 0x7faf101a0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.751+0000 7faf0de00640 1 -- 172.21.15.97:0/2893812479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef8009560 con 0x7faf101033c0 2024-03-11T19:58:52.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.751+0000 7faf0d400640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf101045b0 0x7faf101999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:52.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.751+0000 7faf0de00640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf101033c0 0x7faf101994b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faf0000e9c0 tx=0x7faf0000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:52.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.752+0000 7faf06a00640 1 -- 172.21.15.97:0/2893812479 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf000096e0 con 0x7faf101033c0 2024-03-11T19:58:52.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.752+0000 7faf06a00640 1 -- 172.21.15.97:0/2893812479 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf0000cd80 con 0x7faf101033c0 2024-03-11T19:58:52.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.752+0000 7faf14c00640 1 -- 172.21.15.97:0/2893812479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf101a15b0 con 0x7faf101033c0 2024-03-11T19:58:52.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.752+0000 7faf06a00640 1 -- 172.21.15.97:0/2893812479 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00010430 con 0x7faf101033c0 2024-03-11T19:58:52.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.752+0000 7faf14c00640 1 -- 172.21.15.97:0/2893812479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf101a1a40 con 0x7faf101033c0 2024-03-11T19:58:52.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.753+0000 7faf14c00640 1 -- 172.21.15.97:0/2893812479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1010a680 con 0x7faf101033c0 2024-03-11T19:58:52.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.755+0000 7faf06a00640 1 -- 172.21.15.97:0/2893812479 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7faf00003680 con 0x7faf101033c0 2024-03-11T19:58:52.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.757+0000 7faf06a00640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7faef40a1ab0 0x7faef40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:52.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.757+0000 7faf06a00640 1 -- 172.21.15.97:0/2893812479 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7faf00014070 con 0x7faf101033c0 2024-03-11T19:58:52.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.757+0000 7faf0d400640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7faef40a1ab0 0x7faef40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:52.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.758+0000 7faf0d400640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7faef40a1ab0 0x7faef40a3f70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7faef8000c00 tx=0x7faef802c740 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:52.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.759+0000 7faf06a00640 1 -- 172.21.15.97:0/2893812479 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faf00096a70 con 0x7faf101033c0 2024-03-11T19:58:52.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:52 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/4164707155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:52.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:52 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3122408902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:58:52.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.958+0000 7faf14c00640 1 -- 172.21.15.97:0/2893812479 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faf1002ce40 con 0x7faf101033c0 2024-03-11T19:58:52.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.958+0000 7faf06a00640 1 -- 172.21.15.97:0/2893812479 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7faf0009a130 con 0x7faf101033c0 2024-03-11T19:58:52.956 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T19:58:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.961+0000 7faf14c00640 1 -- 172.21.15.97:0/2893812479 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7faef40a1ab0 msgr2=0x7faef40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.961+0000 7faf14c00640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7faef40a1ab0 0x7faef40a3f70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7faef8000c00 tx=0x7faef802c740 comp rx=0 tx=0).stop 2024-03-11T19:58:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.961+0000 7faf14c00640 1 -- 172.21.15.97:0/2893812479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf101033c0 msgr2=0x7faf101994b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.961+0000 7faf14c00640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf101033c0 0x7faf101994b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faf0000e9c0 tx=0x7faf0000ee80 comp rx=0 tx=0).stop 2024-03-11T19:58:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.961+0000 7faf14c00640 1 -- 172.21.15.97:0/2893812479 shutdown_connections 2024-03-11T19:58:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.961+0000 7faf14c00640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf10104f70 0x7faf101a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.961+0000 7faf14c00640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf101045b0 0x7faf101999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.961+0000 7faf14c00640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7faef40a1ab0 0x7faef40a3f70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.961+0000 7faf14c00640 1 --2- 172.21.15.97:0/2893812479 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faf101033c0 0x7faf101994b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:52.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.961+0000 7faf14c00640 1 -- 172.21.15.97:0/2893812479 >> 172.21.15.97:0/2893812479 conn(0x7faf100fe930 msgr2=0x7faf10107850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:52.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.961+0000 7faf14c00640 1 -- 172.21.15.97:0/2893812479 shutdown_connections 2024-03-11T19:58:52.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:52.961+0000 7faf14c00640 1 -- 172.21.15.97:0/2893812479 wait complete. 2024-03-11T19:58:52.980 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:58:52.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:52 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/4164707155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:52.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:52 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3122408902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:58:53.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:52 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/4164707155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:53.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:52 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3122408902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:58:53.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.193+0000 7fc82d600640 1 -- 172.21.15.97:0/2329756170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc828077720 msgr2=0x7fc828077fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:53.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.193+0000 7fc82d600640 1 --2- 172.21.15.97:0/2329756170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc828077720 0x7fc828077fc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc818009ca0 tx=0x7fc81802c0f0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.193+0000 7fc82d600640 1 -- 172.21.15.97:0/2329756170 shutdown_connections 2024-03-11T19:58:53.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.193+0000 7fc82d600640 1 --2- 172.21.15.97:0/2329756170 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc828077720 0x7fc828077fc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.193+0000 7fc82d600640 1 --2- 172.21.15.97:0/2329756170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc828076d80 0x7fc8280771e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.193+0000 7fc82d600640 1 --2- 172.21.15.97:0/2329756170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc828075b40 0x7fc828075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.193+0000 7fc82d600640 1 -- 172.21.15.97:0/2329756170 >> 172.21.15.97:0/2329756170 conn(0x7fc8280fe930 msgr2=0x7fc828100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:53.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.193+0000 7fc82d600640 1 -- 172.21.15.97:0/2329756170 shutdown_connections 2024-03-11T19:58:53.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.193+0000 7fc82d600640 1 -- 172.21.15.97:0/2329756170 wait complete. 2024-03-11T19:58:53.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.194+0000 7fc82d600640 1 Processor -- start 2024-03-11T19:58:53.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.194+0000 7fc82d600640 1 -- start start 2024-03-11T19:58:53.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc82d600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc828075b40 0x7fc82819d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:53.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc82d600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc828076d80 0x7fc82819dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:53.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc82d600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc828077720 0x7fc8281a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:53.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc82d600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8281a52a0 con 0x7fc828075b40 2024-03-11T19:58:53.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc82d600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8281a5410 con 0x7fc828077720 2024-03-11T19:58:53.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc82d600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8281a55d0 con 0x7fc828076d80 2024-03-11T19:58:53.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc826a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc828075b40 0x7fc82819d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:53.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc826a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc828075b40 0x7fc82819d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58362/0 (socket says 172.21.15.97:58362) 2024-03-11T19:58:53.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc826a00640 1 -- 172.21.15.97:0/433578327 learned_addr learned my addr 172.21.15.97:0/433578327 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:53.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc826000640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc828076d80 0x7fc82819dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:53.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc827400640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc828077720 0x7fc8281a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:53.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc826a00640 1 -- 172.21.15.97:0/433578327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc828076d80 msgr2=0x7fc82819dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:53.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc826a00640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc828076d80 0x7fc82819dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc826a00640 1 -- 172.21.15.97:0/433578327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc828077720 msgr2=0x7fc8281a4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:53.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc826a00640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc828077720 0x7fc8281a4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.195+0000 7fc826a00640 1 -- 172.21.15.97:0/433578327 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc810009490 con 0x7fc828075b40 2024-03-11T19:58:53.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.196+0000 7fc826a00640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc828075b40 0x7fc82819d7b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc82000e9f0 tx=0x7fc82000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:53.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.196+0000 7fc81f400640 1 -- 172.21.15.97:0/433578327 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc820009700 con 0x7fc828075b40 2024-03-11T19:58:53.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.196+0000 7fc81f400640 1 -- 172.21.15.97:0/433578327 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc82000cd80 con 0x7fc828075b40 2024-03-11T19:58:53.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.196+0000 7fc82d600640 1 -- 172.21.15.97:0/433578327 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc818009900 con 0x7fc828075b40 2024-03-11T19:58:53.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.196+0000 7fc81f400640 1 -- 172.21.15.97:0/433578327 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc820018920 con 0x7fc828075b40 2024-03-11T19:58:53.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.196+0000 7fc82d600640 1 -- 172.21.15.97:0/433578327 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8281a5c30 con 0x7fc828075b40 2024-03-11T19:58:53.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.197+0000 7fc82d600640 1 -- 172.21.15.97:0/433578327 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc82810fd30 con 0x7fc828075b40 2024-03-11T19:58:53.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.199+0000 7fc81f400640 1 -- 172.21.15.97:0/433578327 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc820018a80 con 0x7fc828075b40 2024-03-11T19:58:53.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.200+0000 7fc81f400640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc80c0a1a30 0x7fc80c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:53.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.201+0000 7fc81f400640 1 -- 172.21.15.97:0/433578327 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fc820014070 con 0x7fc828075b40 2024-03-11T19:58:53.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.201+0000 7fc826000640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc80c0a1a30 0x7fc80c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:53.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.202+0000 7fc826000640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc80c0a1a30 0x7fc80c0a3ef0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc82819ecc0 tx=0x7fc810005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:53.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.203+0000 7fc81f400640 1 -- 172.21.15.97:0/433578327 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc820096ed0 con 0x7fc828075b40 2024-03-11T19:58:53.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.401+0000 7fc82d600640 1 -- 172.21.15.97:0/433578327 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc828113470 con 0x7fc828075b40 2024-03-11T19:58:53.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.402+0000 7fc81f400640 1 -- 172.21.15.97:0/433578327 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc828113470 con 0x7fc828075b40 2024-03-11T19:58:53.399 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-11T19:58:53.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.404+0000 7fc82d600640 1 -- 172.21.15.97:0/433578327 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc80c0a1a30 msgr2=0x7fc80c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:53.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.404+0000 7fc82d600640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc80c0a1a30 0x7fc80c0a3ef0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc82819ecc0 tx=0x7fc810005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:53.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.404+0000 7fc82d600640 1 -- 172.21.15.97:0/433578327 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc828075b40 msgr2=0x7fc82819d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:53.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.404+0000 7fc82d600640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc828075b40 0x7fc82819d7b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc82000e9f0 tx=0x7fc82000eeb0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.404+0000 7fc82d600640 1 -- 172.21.15.97:0/433578327 shutdown_connections 2024-03-11T19:58:53.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.404+0000 7fc82d600640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc828077720 0x7fc8281a4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.404+0000 7fc82d600640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc828076d80 0x7fc82819dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.404+0000 7fc82d600640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc80c0a1a30 0x7fc80c0a3ef0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.404+0000 7fc82d600640 1 --2- 172.21.15.97:0/433578327 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc828075b40 0x7fc82819d7b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.404+0000 7fc82d600640 1 -- 172.21.15.97:0/433578327 >> 172.21.15.97:0/433578327 conn(0x7fc8280fe930 msgr2=0x7fc828100430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:53.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.405+0000 7fc82d600640 1 -- 172.21.15.97:0/433578327 shutdown_connections 2024-03-11T19:58:53.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.405+0000 7fc82d600640 1 -- 172.21.15.97:0/433578327 wait complete. 2024-03-11T19:58:53.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T19:58:53.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.628+0000 7f0f12400640 1 -- 172.21.15.97:0/1054809944 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f0c1033a0 msgr2=0x7f0f0c1037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:53.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.628+0000 7f0f12400640 1 --2- 172.21.15.97:0/1054809944 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f0c1033a0 0x7f0f0c1037c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009900 tx=0x7f0efc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:53.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.629+0000 7f0f12400640 1 -- 172.21.15.97:0/1054809944 shutdown_connections 2024-03-11T19:58:53.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.629+0000 7f0f12400640 1 --2- 172.21.15.97:0/1054809944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f0c104f50 0x7f0f0c10b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.629+0000 7f0f12400640 1 --2- 172.21.15.97:0/1054809944 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f0c104590 0x7f0f0c104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.629+0000 7f0f12400640 1 --2- 172.21.15.97:0/1054809944 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f0c1033a0 0x7f0f0c1037c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.629+0000 7f0f12400640 1 -- 172.21.15.97:0/1054809944 >> 172.21.15.97:0/1054809944 conn(0x7f0f0c0fe930 msgr2=0x7f0f0c100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:53.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.629+0000 7f0f12400640 1 -- 172.21.15.97:0/1054809944 shutdown_connections 2024-03-11T19:58:53.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.629+0000 7f0f12400640 1 -- 172.21.15.97:0/1054809944 wait complete. 2024-03-11T19:58:53.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.630+0000 7f0f12400640 1 Processor -- start 2024-03-11T19:58:53.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.630+0000 7f0f12400640 1 -- start start 2024-03-11T19:58:53.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.630+0000 7f0f12400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f0c1033a0 0x7f0f0c199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:53.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.630+0000 7f0f12400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f0c104590 0x7f0f0c199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.630+0000 7f0f12400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f0c104f50 0x7f0f0c1a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.630+0000 7f0f12400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c1a0f20 con 0x7f0f0c1033a0 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.630+0000 7f0f12400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c1a1090 con 0x7f0f0c104f50 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.630+0000 7f0f12400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c1a1250 con 0x7f0f0c104590 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.630+0000 7f0f0b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f0c1033a0 0x7f0f0c199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.630+0000 7f0f0b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f0c1033a0 0x7f0f0c199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:58374/0 (socket says 172.21.15.97:58374) 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.630+0000 7f0f0b400640 1 -- 172.21.15.97:0/2173619488 learned_addr learned my addr 172.21.15.97:0/2173619488 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.630+0000 7f0f0be00640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f0c104f50 0x7f0f0c1a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.631+0000 7f0f03e00640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f0c104590 0x7f0f0c199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.631+0000 7f0f0b400640 1 -- 172.21.15.97:0/2173619488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f0c104590 msgr2=0x7f0f0c199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.631+0000 7f0f0b400640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f0c104590 0x7f0f0c199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.631+0000 7f0f0b400640 1 -- 172.21.15.97:0/2173619488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f0c104f50 msgr2=0x7f0f0c1a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.631+0000 7f0f0b400640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f0c104f50 0x7f0f0c1a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.631+0000 7f0f0b400640 1 -- 172.21.15.97:0/2173619488 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc009560 con 0x7f0f0c1033a0 2024-03-11T19:58:53.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.631+0000 7f0f0b400640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f0c1033a0 0x7f0f0c199430 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0efc000c00 tx=0x7f0efc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:53.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.631+0000 7f0f08c00640 1 -- 172.21.15.97:0/2173619488 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc0044c0 con 0x7f0f0c1033a0 2024-03-11T19:58:53.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.631+0000 7f0f08c00640 1 -- 172.21.15.97:0/2173619488 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0efc0052d0 con 0x7f0f0c1033a0 2024-03-11T19:58:53.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.632+0000 7f0f12400640 1 -- 172.21.15.97:0/2173619488 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0c1a14d0 con 0x7f0f0c1033a0 2024-03-11T19:58:53.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.632+0000 7f0f08c00640 1 -- 172.21.15.97:0/2173619488 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc03e450 con 0x7f0f0c1033a0 2024-03-11T19:58:53.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.632+0000 7f0f12400640 1 -- 172.21.15.97:0/2173619488 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0c1a1a30 con 0x7f0f0c1033a0 2024-03-11T19:58:53.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.633+0000 7f0f12400640 1 -- 172.21.15.97:0/2173619488 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0c10a660 con 0x7f0f0c1033a0 2024-03-11T19:58:53.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.635+0000 7f0f08c00640 1 -- 172.21.15.97:0/2173619488 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0efc005440 con 0x7f0f0c1033a0 2024-03-11T19:58:53.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.636+0000 7f0f08c00640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0ef00a1ab0 0x7f0ef00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:53.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.636+0000 7f0f08c00640 1 -- 172.21.15.97:0/2173619488 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f0efc0f0060 con 0x7f0f0c1033a0 2024-03-11T19:58:53.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.636+0000 7f0f03e00640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0ef00a1ab0 0x7f0ef00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:53.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.637+0000 7f0f03e00640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0ef00a1ab0 0x7f0ef00a3f70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4005d00 tx=0x7f0ef4005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:53.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.640+0000 7f0f08c00640 1 -- 172.21.15.97:0/2173619488 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0efc0b8e50 con 0x7f0f0c1033a0 2024-03-11T19:58:53.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.839+0000 7f0f12400640 1 -- 172.21.15.97:0/2173619488 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0f0c1a1660 con 0x7f0f0c1033a0 2024-03-11T19:58:53.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.840+0000 7f0f08c00640 1 -- 172.21.15.97:0/2173619488 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0f0c1a1660 con 0x7f0f0c1033a0 2024-03-11T19:58:53.838 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T19:58:53.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.843+0000 7f0f12400640 1 -- 172.21.15.97:0/2173619488 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0ef00a1ab0 msgr2=0x7f0ef00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:53.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.843+0000 7f0f12400640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0ef00a1ab0 0x7f0ef00a3f70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4005d00 tx=0x7f0ef4005c50 comp rx=0 tx=0).stop 2024-03-11T19:58:53.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.843+0000 7f0f12400640 1 -- 172.21.15.97:0/2173619488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f0c1033a0 msgr2=0x7f0f0c199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:53.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.843+0000 7f0f12400640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f0c1033a0 0x7f0f0c199430 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0efc000c00 tx=0x7f0efc0040c0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.843+0000 7f0f12400640 1 -- 172.21.15.97:0/2173619488 shutdown_connections 2024-03-11T19:58:53.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.843+0000 7f0f12400640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f0c104f50 0x7f0f0c1a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.843+0000 7f0f12400640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f0c104590 0x7f0f0c199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.843+0000 7f0f12400640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f0ef00a1ab0 0x7f0ef00a3f70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.843+0000 7f0f12400640 1 --2- 172.21.15.97:0/2173619488 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0f0c1033a0 0x7f0f0c199430 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:53.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.843+0000 7f0f12400640 1 -- 172.21.15.97:0/2173619488 >> 172.21.15.97:0/2173619488 conn(0x7f0f0c0fe930 msgr2=0x7f0f0c107830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:53.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.843+0000 7f0f12400640 1 -- 172.21.15.97:0/2173619488 shutdown_connections 2024-03-11T19:58:53.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:53.843+0000 7f0f12400640 1 -- 172.21.15.97:0/2173619488 wait complete. 2024-03-11T19:58:53.857 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:53 smithi097 ceph-mon[31898]: pgmap v417: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:53.857 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:53 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2893812479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:58:53.857 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:53 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/433578327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:58:53.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:58:53.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:53 smithi114 ceph-mon[36376]: pgmap v417: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:53.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:53 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2893812479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:58:53.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:53 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/433578327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:58:54.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:53 smithi161 ceph-mon[36332]: pgmap v417: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:58:54.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:53 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2893812479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:58:54.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:53 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/433578327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:58:54.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.069+0000 7f39d0a00640 1 -- 172.21.15.97:0/1254967987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc1045b0 msgr2=0x7f39cc104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:54.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.069+0000 7f39d0a00640 1 --2- 172.21.15.97:0/1254967987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc1045b0 0x7f39cc104a30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f39b4009900 tx=0x7f39b402bc50 comp rx=0 tx=0).stop 2024-03-11T19:58:54.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.069+0000 7f39d0a00640 1 -- 172.21.15.97:0/1254967987 shutdown_connections 2024-03-11T19:58:54.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.069+0000 7f39d0a00640 1 --2- 172.21.15.97:0/1254967987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39cc104f70 0x7f39cc10b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.069+0000 7f39d0a00640 1 --2- 172.21.15.97:0/1254967987 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc1045b0 0x7f39cc104a30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.069+0000 7f39d0a00640 1 --2- 172.21.15.97:0/1254967987 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39cc1033c0 0x7f39cc1037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.069+0000 7f39d0a00640 1 -- 172.21.15.97:0/1254967987 >> 172.21.15.97:0/1254967987 conn(0x7f39cc0fe970 msgr2=0x7f39cc100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:54.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.069+0000 7f39d0a00640 1 -- 172.21.15.97:0/1254967987 shutdown_connections 2024-03-11T19:58:54.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.069+0000 7f39d0a00640 1 -- 172.21.15.97:0/1254967987 wait complete. 2024-03-11T19:58:54.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.070+0000 7f39d0a00640 1 Processor -- start 2024-03-11T19:58:54.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.070+0000 7f39d0a00640 1 -- start start 2024-03-11T19:58:54.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.070+0000 7f39d0a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc1033c0 0x7f39cc199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.070+0000 7f39d0a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39cc1045b0 0x7f39cc1999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.070+0000 7f39d0a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39cc104f70 0x7f39cc1a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.070+0000 7f39d0a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc1a0f50 con 0x7f39cc1033c0 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.070+0000 7f39d0a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc1a10c0 con 0x7f39cc1045b0 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.070+0000 7f39d0a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc1a1280 con 0x7f39cc104f70 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.071+0000 7f39c9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc1033c0 0x7f39cc199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.071+0000 7f39c9e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc1033c0 0x7f39cc199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49306/0 (socket says 172.21.15.97:49306) 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.071+0000 7f39c9e00640 1 -- 172.21.15.97:0/1252225118 learned_addr learned my addr 172.21.15.97:0/1252225118 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.071+0000 7f39ca800640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39cc104f70 0x7f39cc1a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.071+0000 7f39c1400640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39cc1045b0 0x7f39cc1999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.071+0000 7f39c9e00640 1 -- 172.21.15.97:0/1252225118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39cc104f70 msgr2=0x7f39cc1a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.071+0000 7f39c9e00640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39cc104f70 0x7f39cc1a0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.071+0000 7f39c9e00640 1 -- 172.21.15.97:0/1252225118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39cc1045b0 msgr2=0x7f39cc1999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.071+0000 7f39c9e00640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39cc1045b0 0x7f39cc1999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.071+0000 7f39c9e00640 1 -- 172.21.15.97:0/1252225118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c4009840 con 0x7f39cc1033c0 2024-03-11T19:58:54.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.071+0000 7f39c9e00640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc1033c0 0x7f39cc199460 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00e9c0 tx=0x7f39bc00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:54.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.072+0000 7f39c3400640 1 -- 172.21.15.97:0/1252225118 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc0096e0 con 0x7f39cc1033c0 2024-03-11T19:58:54.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.072+0000 7f39c3400640 1 -- 172.21.15.97:0/1252225118 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39bc00ccc0 con 0x7f39cc1033c0 2024-03-11T19:58:54.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.072+0000 7f39d0a00640 1 -- 172.21.15.97:0/1252225118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b4009560 con 0x7f39cc1033c0 2024-03-11T19:58:54.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.072+0000 7f39c3400640 1 -- 172.21.15.97:0/1252225118 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc010430 con 0x7f39cc1033c0 2024-03-11T19:58:54.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.072+0000 7f39d0a00640 1 -- 172.21.15.97:0/1252225118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39cc1a1860 con 0x7f39cc1033c0 2024-03-11T19:58:54.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.073+0000 7f39d0a00640 1 -- 172.21.15.97:0/1252225118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39cc10a700 con 0x7f39cc1033c0 2024-03-11T19:58:54.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.075+0000 7f39c3400640 1 -- 172.21.15.97:0/1252225118 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f39bc003f80 con 0x7f39cc1033c0 2024-03-11T19:58:54.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.076+0000 7f39c3400640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39b00a1a30 0x7f39b00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:54.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.076+0000 7f39c3400640 1 -- 172.21.15.97:0/1252225118 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f39bc014070 con 0x7f39cc1033c0 2024-03-11T19:58:54.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.076+0000 7f39c1400640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39b00a1a30 0x7f39b00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:54.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.077+0000 7f39c1400640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39b00a1a30 0x7f39b00a3ef0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f39b4009230 tx=0x7f39b4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:54.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.081+0000 7f39c3400640 1 -- 172.21.15.97:0/1252225118 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f39bc096a00 con 0x7f39cc1033c0 2024-03-11T19:58:54.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.320+0000 7f39d0a00640 1 -- 172.21.15.97:0/1252225118 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39cc10edd0 con 0x7f39cc1033c0 2024-03-11T19:58:54.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.322+0000 7f39c3400640 1 -- 172.21.15.97:0/1252225118 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11401 (secure 0 0 0) 0x7f39bc09a0c0 con 0x7f39cc1033c0 2024-03-11T19:58:54.320 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:54.320 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:55:45.149335+0000","modified":"2024-03-11T19:56:19.631638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24583,"mds_2":24547,"mds_3":24545,"mds_4":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24583":{"gid":24583,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.97:6841/2531110572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2531110572},{"type":"v1","addr":"172.21.15.97:6841","nonce":2531110572}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6839/1624532248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":1624532248},{"type":"v1","addr":"172.21.15.114:6839","nonce":1624532248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:58:54.320 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-03-11T19:58:54.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.325+0000 7f39d0a00640 1 -- 172.21.15.97:0/1252225118 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39b00a1a30 msgr2=0x7f39b00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.325+0000 7f39d0a00640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39b00a1a30 0x7f39b00a3ef0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f39b4009230 tx=0x7f39b4005fd0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.325+0000 7f39d0a00640 1 -- 172.21.15.97:0/1252225118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc1033c0 msgr2=0x7f39cc199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.325+0000 7f39d0a00640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc1033c0 0x7f39cc199460 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00e9c0 tx=0x7f39bc00edc0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.325+0000 7f39d0a00640 1 -- 172.21.15.97:0/1252225118 shutdown_connections 2024-03-11T19:58:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.325+0000 7f39d0a00640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f39cc104f70 0x7f39cc1a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.325+0000 7f39d0a00640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39cc1045b0 0x7f39cc1999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.325+0000 7f39d0a00640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f39b00a1a30 0x7f39b00a3ef0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.325+0000 7f39d0a00640 1 --2- 172.21.15.97:0/1252225118 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f39cc1033c0 0x7f39cc199460 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.325+0000 7f39d0a00640 1 -- 172.21.15.97:0/1252225118 >> 172.21.15.97:0/1252225118 conn(0x7f39cc0fe970 msgr2=0x7f39cc1078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.325+0000 7f39d0a00640 1 -- 172.21.15.97:0/1252225118 shutdown_connections 2024-03-11T19:58:54.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.325+0000 7f39d0a00640 1 -- 172.21.15.97:0/1252225118 wait complete. 2024-03-11T19:58:54.342 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:58:54.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.559+0000 7f561e800640 1 -- 172.21.15.97:0/1962363089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56181077c0 msgr2=0x7f5618107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:54.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.559+0000 7f561e800640 1 --2- 172.21.15.97:0/1962363089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56181077c0 0x7f5618107be0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5608009900 tx=0x7f560802bc40 comp rx=0 tx=0).stop 2024-03-11T19:58:54.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.559+0000 7f561e800640 1 -- 172.21.15.97:0/1962363089 shutdown_connections 2024-03-11T19:58:54.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.559+0000 7f561e800640 1 --2- 172.21.15.97:0/1962363089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5618109370 0x7f561810fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.559+0000 7f561e800640 1 --2- 172.21.15.97:0/1962363089 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f56181089b0 0x7f5618108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.559+0000 7f561e800640 1 --2- 172.21.15.97:0/1962363089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56181077c0 0x7f5618107be0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.559+0000 7f561e800640 1 -- 172.21.15.97:0/1962363089 >> 172.21.15.97:0/1962363089 conn(0x7f5618076790 msgr2=0x7f5618078bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:54.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.560+0000 7f561e800640 1 -- 172.21.15.97:0/1962363089 shutdown_connections 2024-03-11T19:58:54.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.560+0000 7f561e800640 1 -- 172.21.15.97:0/1962363089 wait complete. 2024-03-11T19:58:54.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.561+0000 7f561e800640 1 Processor -- start 2024-03-11T19:58:54.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.561+0000 7f561e800640 1 -- start start 2024-03-11T19:58:54.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.561+0000 7f561e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56181077c0 0x7f561819d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:54.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.561+0000 7f561e800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56181089b0 0x7f561819dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:54.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.561+0000 7f561e800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618109370 0x7f56181a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:54.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.561+0000 7f561e800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56181a5330 con 0x7f5618109370 2024-03-11T19:58:54.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.561+0000 7f561e800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56181a54a0 con 0x7f56181077c0 2024-03-11T19:58:54.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.562+0000 7f561e800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56181a5660 con 0x7f56181089b0 2024-03-11T19:58:54.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.562+0000 7f5617400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56181077c0 0x7f561819d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:54.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.562+0000 7f5617400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56181077c0 0x7f561819d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:50942/0 (socket says 172.21.15.97:50942) 2024-03-11T19:58:54.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.562+0000 7f5617400640 1 -- 172.21.15.97:0/1800124756 learned_addr learned my addr 172.21.15.97:0/1800124756 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:54.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.562+0000 7f5617e00640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618109370 0x7f56181a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:54.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.562+0000 7f560fe00640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56181089b0 0x7f561819dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:54.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.562+0000 7f5617400640 1 -- 172.21.15.97:0/1800124756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56181089b0 msgr2=0x7f561819dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:54.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.562+0000 7f5617400640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56181089b0 0x7f561819dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.562+0000 7f5617400640 1 -- 172.21.15.97:0/1800124756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618109370 msgr2=0x7f56181a4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:54.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.562+0000 7f5617400640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618109370 0x7f56181a4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.562+0000 7f5617400640 1 -- 172.21.15.97:0/1800124756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5600009490 con 0x7f56181077c0 2024-03-11T19:58:54.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.563+0000 7f5617400640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56181077c0 0x7f561819d840 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5608005ab0 tx=0x7f5608005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:54.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.563+0000 7f5614c00640 1 -- 172.21.15.97:0/1800124756 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56080043d0 con 0x7f56181077c0 2024-03-11T19:58:54.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.563+0000 7f5614c00640 1 -- 172.21.15.97:0/1800124756 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56080052d0 con 0x7f56181077c0 2024-03-11T19:58:54.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.563+0000 7f561e800640 1 -- 172.21.15.97:0/1800124756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5608009560 con 0x7f56181077c0 2024-03-11T19:58:54.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.563+0000 7f5614c00640 1 -- 172.21.15.97:0/1800124756 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5608003d40 con 0x7f56181077c0 2024-03-11T19:58:54.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.563+0000 7f561e800640 1 -- 172.21.15.97:0/1800124756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56181a5c60 con 0x7f56181077c0 2024-03-11T19:58:54.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.564+0000 7f561e800640 1 -- 172.21.15.97:0/1800124756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f561810eae0 con 0x7f56181077c0 2024-03-11T19:58:54.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.566+0000 7f5614c00640 1 -- 172.21.15.97:0/1800124756 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5608005440 con 0x7f56181077c0 2024-03-11T19:58:54.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.567+0000 7f5614c00640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f55fc0a1a30 0x7f55fc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:54.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.568+0000 7f5614c00640 1 -- 172.21.15.97:0/1800124756 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f56080efef0 con 0x7f56181077c0 2024-03-11T19:58:54.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.568+0000 7f560fe00640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f55fc0a1a30 0x7f55fc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:54.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.568+0000 7f560fe00640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f55fc0a1a30 0x7f55fc0a3ef0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f561819ed50 tx=0x7f5600005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:54.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.570+0000 7f5614c00640 1 -- 172.21.15.97:0/1800124756 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f56080b8ce0 con 0x7f56181077c0 2024-03-11T19:58:54.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.768+0000 7f561e800640 1 -- 172.21.15.97:0/1800124756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f561802cdd0 con 0x7f56181077c0 2024-03-11T19:58:54.768 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.771+0000 7f5614c00640 1 -- 172.21.15.97:0/1800124756 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24838 (secure 0 0 0) 0x7f5608043090 con 0x7f56181077c0 2024-03-11T19:58:54.769 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:54.769 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":93,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:57:46.674203+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:58:54.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.773+0000 7f561e800640 1 -- 172.21.15.97:0/1800124756 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f55fc0a1a30 msgr2=0x7f55fc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:54.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.773+0000 7f561e800640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f55fc0a1a30 0x7f55fc0a3ef0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f561819ed50 tx=0x7f5600005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:54.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.773+0000 7f561e800640 1 -- 172.21.15.97:0/1800124756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56181077c0 msgr2=0x7f561819d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:54.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.773+0000 7f561e800640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56181077c0 0x7f561819d840 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5608005ab0 tx=0x7f5608005e50 comp rx=0 tx=0).stop 2024-03-11T19:58:54.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.773+0000 7f561e800640 1 -- 172.21.15.97:0/1800124756 shutdown_connections 2024-03-11T19:58:54.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.773+0000 7f561e800640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f55fc0a1a30 0x7f55fc0a3ef0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.773+0000 7f561e800640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5618109370 0x7f56181a4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.773+0000 7f561e800640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f56181089b0 0x7f561819dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.774+0000 7f561e800640 1 --2- 172.21.15.97:0/1800124756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56181077c0 0x7f561819d840 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:54.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.774+0000 7f561e800640 1 -- 172.21.15.97:0/1800124756 >> 172.21.15.97:0/1800124756 conn(0x7f5618076790 msgr2=0x7f561810bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:54.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.774+0000 7f561e800640 1 -- 172.21.15.97:0/1800124756 shutdown_connections 2024-03-11T19:58:54.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:54.774+0000 7f561e800640 1 -- 172.21.15.97:0/1800124756 wait complete. 2024-03-11T19:58:54.790 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:50:29.051985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:55:43.424076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:55:44.652160+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T19:55:48.598077+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T19:58:54.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T19:58:54.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:54 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2173619488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:58:54.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:54 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1252225118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:54.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:54 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2173619488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:58:54.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:54 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1252225118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:55.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.007+0000 7f0191800640 1 -- 172.21.15.97:0/1993136901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c106d40 msgr2=0x7f018c109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:55.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.007+0000 7f0191800640 1 --2- 172.21.15.97:0/1993136901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c106d40 0x7f018c109190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0184009ca0 tx=0x7f018402c100 comp rx=0 tx=0).stop 2024-03-11T19:58:55.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.008+0000 7f0191800640 1 -- 172.21.15.97:0/1993136901 shutdown_connections 2024-03-11T19:58:55.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.008+0000 7f0191800640 1 --2- 172.21.15.97:0/1993136901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c106d40 0x7f018c109190 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.008+0000 7f0191800640 1 --2- 172.21.15.97:0/1993136901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f018c104410 0x7f018c106800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.008+0000 7f0191800640 1 --2- 172.21.15.97:0/1993136901 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f018c101ae0 0x7f018c103ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.008+0000 7f0191800640 1 -- 172.21.15.97:0/1993136901 >> 172.21.15.97:0/1993136901 conn(0x7f018c0fb4f0 msgr2=0x7f018c0fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:55.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.008+0000 7f0191800640 1 -- 172.21.15.97:0/1993136901 shutdown_connections 2024-03-11T19:58:55.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.008+0000 7f0191800640 1 -- 172.21.15.97:0/1993136901 wait complete. 2024-03-11T19:58:55.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.009+0000 7f0191800640 1 Processor -- start 2024-03-11T19:58:55.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.009+0000 7f0191800640 1 -- start start 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.009+0000 7f0191800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f018c101ae0 0x7f018c1970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.009+0000 7f0191800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c104410 0x7f018c1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.009+0000 7f0191800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f018c106d40 0x7f018c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.009+0000 7f0191800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c19eba0 con 0x7f018c106d40 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.009+0000 7f0191800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c19ed10 con 0x7f018c104410 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.009+0000 7f0191800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c19eed0 con 0x7f018c101ae0 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.010+0000 7f0190e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f018c106d40 0x7f018c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.010+0000 7f018b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c104410 0x7f018c1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.010+0000 7f018be00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f018c101ae0 0x7f018c1970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.010+0000 7f0190e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f018c106d40 0x7f018c19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49344/0 (socket says 172.21.15.97:49344) 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.010+0000 7f018b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c104410 0x7f018c1975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:50960/0 (socket says 172.21.15.97:50960) 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.010+0000 7f0190e00640 1 -- 172.21.15.97:0/3585401355 learned_addr learned my addr 172.21.15.97:0/3585401355 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.010+0000 7f018be00640 1 -- 172.21.15.97:0/3585401355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c104410 msgr2=0x7f018c1975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:55.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.010+0000 7f018be00640 1 --2- 172.21.15.97:0/3585401355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c104410 0x7f018c1975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.010+0000 7f018be00640 1 -- 172.21.15.97:0/3585401355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f018c106d40 msgr2=0x7f018c19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:55.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.010+0000 7f018be00640 1 --2- 172.21.15.97:0/3585401355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f018c106d40 0x7f018c19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.010+0000 7f018be00640 1 -- 172.21.15.97:0/3585401355 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0184009900 con 0x7f018c101ae0 2024-03-11T19:58:55.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.011+0000 7f018be00640 1 --2- 172.21.15.97:0/3585401355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f018c101ae0 0x7f018c1970b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f017c00e9f0 tx=0x7f017c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:55.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.011+0000 7f0188c00640 1 -- 172.21.15.97:0/3585401355 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c00ccc0 con 0x7f018c101ae0 2024-03-11T19:58:55.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.011+0000 7f0191800640 1 -- 172.21.15.97:0/3585401355 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018c19f1b0 con 0x7f018c101ae0 2024-03-11T19:58:55.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.011+0000 7f0188c00640 1 -- 172.21.15.97:0/3585401355 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f017c00ce20 con 0x7f018c101ae0 2024-03-11T19:58:55.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.011+0000 7f0188c00640 1 -- 172.21.15.97:0/3585401355 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c004a00 con 0x7f018c101ae0 2024-03-11T19:58:55.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.011+0000 7f0191800640 1 -- 172.21.15.97:0/3585401355 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018c19f6c0 con 0x7f018c101ae0 2024-03-11T19:58:55.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.012+0000 7f0191800640 1 -- 172.21.15.97:0/3585401355 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018c0fcfe0 con 0x7f018c101ae0 2024-03-11T19:58:55.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.014+0000 7f0188c00640 1 -- 172.21.15.97:0/3585401355 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f017c0050b0 con 0x7f018c101ae0 2024-03-11T19:58:55.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.015+0000 7f0188c00640 1 --2- 172.21.15.97:0/3585401355 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f01700a1a30 0x7f01700a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:55.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.016+0000 7f0188c00640 1 -- 172.21.15.97:0/3585401355 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f017c014070 con 0x7f018c101ae0 2024-03-11T19:58:55.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.016+0000 7f018b400640 1 --2- 172.21.15.97:0/3585401355 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f01700a1a30 0x7f01700a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:55.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.016+0000 7f018b400640 1 --2- 172.21.15.97:0/3585401355 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f01700a1a30 0x7f01700a3ef0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f018c1985c0 tx=0x7f0178005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:55.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.018+0000 7f0188c00640 1 -- 172.21.15.97:0/3585401355 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f017c0966d0 con 0x7f018c101ae0 2024-03-11T19:58:55.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:54 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2173619488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:58:55.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:54 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1252225118' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:55.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.199+0000 7f0191800640 1 -- 172.21.15.97:0/3585401355 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f018c10ca80 con 0x7f018c101ae0 2024-03-11T19:58:55.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.200+0000 7f0188c00640 1 -- 172.21.15.97:0/3585401355 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f017c099d90 con 0x7f018c101ae0 2024-03-11T19:58:55.198 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-11T19:58:55.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.203+0000 7f0191800640 1 -- 172.21.15.97:0/3585401355 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f01700a1a30 msgr2=0x7f01700a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:55.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.203+0000 7f0191800640 1 --2- 172.21.15.97:0/3585401355 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f01700a1a30 0x7f01700a3ef0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f018c1985c0 tx=0x7f0178005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:55.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.203+0000 7f0191800640 1 -- 172.21.15.97:0/3585401355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f018c101ae0 msgr2=0x7f018c1970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.203+0000 7f0191800640 1 --2- 172.21.15.97:0/3585401355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f018c101ae0 0x7f018c1970b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f017c00e9f0 tx=0x7f017c00eeb0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.203+0000 7f0191800640 1 -- 172.21.15.97:0/3585401355 shutdown_connections 2024-03-11T19:58:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.203+0000 7f0191800640 1 --2- 172.21.15.97:0/3585401355 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f01700a1a30 0x7f01700a3ef0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.203+0000 7f0191800640 1 --2- 172.21.15.97:0/3585401355 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f018c106d40 0x7f018c19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.203+0000 7f0191800640 1 --2- 172.21.15.97:0/3585401355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f018c104410 0x7f018c1975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.203+0000 7f0191800640 1 --2- 172.21.15.97:0/3585401355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f018c101ae0 0x7f018c1970b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.203+0000 7f0191800640 1 -- 172.21.15.97:0/3585401355 >> 172.21.15.97:0/3585401355 conn(0x7f018c0fb4f0 msgr2=0x7f018c102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:55.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.203+0000 7f0191800640 1 -- 172.21.15.97:0/3585401355 shutdown_connections 2024-03-11T19:58:55.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.203+0000 7f0191800640 1 -- 172.21.15.97:0/3585401355 wait complete. 2024-03-11T19:58:55.220 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T19:58:55.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.431+0000 7fcb7e600640 1 -- 172.21.15.97:0/3213513405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb78104490 msgr2=0x7fcb78104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:55.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.431+0000 7fcb7e600640 1 --2- 172.21.15.97:0/3213513405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb78104490 0x7fcb78104910 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb60009860 tx=0x7fcb6002bb70 comp rx=0 tx=0).stop 2024-03-11T19:58:55.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.431+0000 7fcb7e600640 1 -- 172.21.15.97:0/3213513405 shutdown_connections 2024-03-11T19:58:55.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.431+0000 7fcb7e600640 1 --2- 172.21.15.97:0/3213513405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb78104e50 0x7fcb7810b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.431+0000 7fcb7e600640 1 --2- 172.21.15.97:0/3213513405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb78104490 0x7fcb78104910 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.431+0000 7fcb7e600640 1 --2- 172.21.15.97:0/3213513405 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb781032a0 0x7fcb781036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.431+0000 7fcb7e600640 1 -- 172.21.15.97:0/3213513405 >> 172.21.15.97:0/3213513405 conn(0x7fcb780fe830 msgr2=0x7fcb78100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:55.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.431+0000 7fcb7e600640 1 -- 172.21.15.97:0/3213513405 shutdown_connections 2024-03-11T19:58:55.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.431+0000 7fcb7e600640 1 -- 172.21.15.97:0/3213513405 wait complete. 2024-03-11T19:58:55.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.432+0000 7fcb7e600640 1 Processor -- start 2024-03-11T19:58:55.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb7e600640 1 -- start start 2024-03-11T19:58:55.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb7e600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb781032a0 0x7fcb78199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:55.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb7e600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb78104490 0x7fcb78199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:55.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb7e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb78104e50 0x7fcb781a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:55.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb7e600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb781a0df0 con 0x7fcb781032a0 2024-03-11T19:58:55.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb7e600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb781a0f60 con 0x7fcb78104e50 2024-03-11T19:58:55.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb7e600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb781a1120 con 0x7fcb78104490 2024-03-11T19:58:55.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb7d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb781032a0 0x7fcb78199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:55.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb6fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb78104490 0x7fcb78199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb6fe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb78104490 0x7fcb78199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:57786/0 (socket says 172.21.15.97:57786) 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb6fe00640 1 -- 172.21.15.97:0/3116910930 learned_addr learned my addr 172.21.15.97:0/3116910930 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb7dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb78104e50 0x7fcb781a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb7d200640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb781032a0 0x7fcb78199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49356/0 (socket says 172.21.15.97:49356) 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb6fe00640 1 -- 172.21.15.97:0/3116910930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb78104e50 msgr2=0x7fcb781a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb6fe00640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb78104e50 0x7fcb781a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb6fe00640 1 -- 172.21.15.97:0/3116910930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb781032a0 msgr2=0x7fcb78199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb6fe00640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb781032a0 0x7fcb78199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.433+0000 7fcb6fe00640 1 -- 172.21.15.97:0/3116910930 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb68009560 con 0x7fcb78104490 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.434+0000 7fcb7dc00640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb78104e50 0x7fcb781a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.434+0000 7fcb7d200640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb781032a0 0x7fcb78199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.434+0000 7fcb6fe00640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb78104490 0x7fcb78199840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb60005940 tx=0x7fcb600045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.434+0000 7fcb76000640 1 -- 172.21.15.97:0/3116910930 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6002c510 con 0x7fcb78104490 2024-03-11T19:58:55.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.434+0000 7fcb76000640 1 -- 172.21.15.97:0/3116910930 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb6002cb50 con 0x7fcb78104490 2024-03-11T19:58:55.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.434+0000 7fcb7e600640 1 -- 172.21.15.97:0/3116910930 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb60009490 con 0x7fcb78104490 2024-03-11T19:58:55.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.434+0000 7fcb76000640 1 -- 172.21.15.97:0/3116910930 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb60032860 con 0x7fcb78104490 2024-03-11T19:58:55.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.434+0000 7fcb7e600640 1 -- 172.21.15.97:0/3116910930 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb781a1580 con 0x7fcb78104490 2024-03-11T19:58:55.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.435+0000 7fcb7e600640 1 -- 172.21.15.97:0/3116910930 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7810a5c0 con 0x7fcb78104490 2024-03-11T19:58:55.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.437+0000 7fcb76000640 1 -- 172.21.15.97:0/3116910930 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcb6003b040 con 0x7fcb78104490 2024-03-11T19:58:55.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.439+0000 7fcb76000640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb5c0a1a30 0x7fcb5c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:55.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.439+0000 7fcb76000640 1 -- 172.21.15.97:0/3116910930 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fcb600ef8e0 con 0x7fcb78104490 2024-03-11T19:58:55.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.439+0000 7fcb7d200640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb5c0a1a30 0x7fcb5c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.439+0000 7fcb7d200640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb5c0a1a30 0x7fcb5c0a3ef0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009530 tx=0x7fcb68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:55.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.441+0000 7fcb76000640 1 -- 172.21.15.97:0/3116910930 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcb600b8780 con 0x7fcb78104490 2024-03-11T19:58:55.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.622+0000 7fcb7e600640 1 -- 172.21.15.97:0/3116910930 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcb7810ef00 con 0x7fcb78104490 2024-03-11T19:58:55.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.623+0000 7fcb76000640 1 -- 172.21.15.97:0/3116910930 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcb60041020 con 0x7fcb78104490 2024-03-11T19:58:55.621 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T19:58:55.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.626+0000 7fcb7e600640 1 -- 172.21.15.97:0/3116910930 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb5c0a1a30 msgr2=0x7fcb5c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:55.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.626+0000 7fcb7e600640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb5c0a1a30 0x7fcb5c0a3ef0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009530 tx=0x7fcb68009260 comp rx=0 tx=0).stop 2024-03-11T19:58:55.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.626+0000 7fcb7e600640 1 -- 172.21.15.97:0/3116910930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb78104490 msgr2=0x7fcb78199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:55.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.626+0000 7fcb7e600640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb78104490 0x7fcb78199840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb60005940 tx=0x7fcb600045e0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.626+0000 7fcb7e600640 1 -- 172.21.15.97:0/3116910930 shutdown_connections 2024-03-11T19:58:55.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.626+0000 7fcb7e600640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb78104e50 0x7fcb781a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.626+0000 7fcb7e600640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb78104490 0x7fcb78199840 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.626+0000 7fcb7e600640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb5c0a1a30 0x7fcb5c0a3ef0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.626+0000 7fcb7e600640 1 --2- 172.21.15.97:0/3116910930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb781032a0 0x7fcb78199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.626+0000 7fcb7e600640 1 -- 172.21.15.97:0/3116910930 >> 172.21.15.97:0/3116910930 conn(0x7fcb780fe830 msgr2=0x7fcb78107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:55.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.626+0000 7fcb7e600640 1 -- 172.21.15.97:0/3116910930 shutdown_connections 2024-03-11T19:58:55.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.626+0000 7fcb7e600640 1 -- 172.21.15.97:0/3116910930 wait complete. 2024-03-11T19:58:55.643 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T19:58:55.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.847+0000 7fcb94c00640 1 -- 172.21.15.97:0/970990019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb901045b0 msgr2=0x7fcb90104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:55.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.847+0000 7fcb94c00640 1 --2- 172.21.15.97:0/970990019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb901045b0 0x7fcb90104a30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009900 tx=0x7fcb7802bc50 comp rx=0 tx=0).stop 2024-03-11T19:58:55.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.848+0000 7fcb94c00640 1 -- 172.21.15.97:0/970990019 shutdown_connections 2024-03-11T19:58:55.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.848+0000 7fcb94c00640 1 --2- 172.21.15.97:0/970990019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb90104f70 0x7fcb9010b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.848+0000 7fcb94c00640 1 --2- 172.21.15.97:0/970990019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb901045b0 0x7fcb90104a30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.848+0000 7fcb94c00640 1 --2- 172.21.15.97:0/970990019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb901033c0 0x7fcb901037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.848+0000 7fcb94c00640 1 -- 172.21.15.97:0/970990019 >> 172.21.15.97:0/970990019 conn(0x7fcb900fe930 msgr2=0x7fcb90100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:55.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.848+0000 7fcb94c00640 1 -- 172.21.15.97:0/970990019 shutdown_connections 2024-03-11T19:58:55.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.848+0000 7fcb94c00640 1 -- 172.21.15.97:0/970990019 wait complete. 2024-03-11T19:58:55.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.849+0000 7fcb94c00640 1 Processor -- start 2024-03-11T19:58:55.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.849+0000 7fcb94c00640 1 -- start start 2024-03-11T19:58:55.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.849+0000 7fcb94c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb901033c0 0x7fcb90199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:55.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.849+0000 7fcb94c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb901045b0 0x7fcb90199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.849+0000 7fcb94c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb90104f70 0x7fcb901a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.849+0000 7fcb94c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb901a0f10 con 0x7fcb901033c0 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.849+0000 7fcb94c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb901a1080 con 0x7fcb901045b0 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.849+0000 7fcb94c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb901a1240 con 0x7fcb90104f70 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.849+0000 7fcb8de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb901033c0 0x7fcb90199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.849+0000 7fcb8de00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb901033c0 0x7fcb90199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49368/0 (socket says 172.21.15.97:49368) 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.849+0000 7fcb8de00640 1 -- 172.21.15.97:0/532681272 learned_addr learned my addr 172.21.15.97:0/532681272 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.849+0000 7fcb8d400640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb901045b0 0x7fcb90199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.849+0000 7fcb8e800640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb90104f70 0x7fcb901a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.850+0000 7fcb8de00640 1 -- 172.21.15.97:0/532681272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb90104f70 msgr2=0x7fcb901a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.850+0000 7fcb8de00640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb90104f70 0x7fcb901a09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.850+0000 7fcb8de00640 1 -- 172.21.15.97:0/532681272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb901045b0 msgr2=0x7fcb90199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.850+0000 7fcb8de00640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb901045b0 0x7fcb90199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.850+0000 7fcb8de00640 1 -- 172.21.15.97:0/532681272 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb78009560 con 0x7fcb901033c0 2024-03-11T19:58:55.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.850+0000 7fcb8de00640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb901033c0 0x7fcb90199420 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000eb70 tx=0x7fcb8000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:55.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.850+0000 7fcb86a00640 1 -- 172.21.15.97:0/532681272 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb80003b70 con 0x7fcb901033c0 2024-03-11T19:58:55.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.850+0000 7fcb86a00640 1 -- 172.21.15.97:0/532681272 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb8000ee70 con 0x7fcb901033c0 2024-03-11T19:58:55.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.850+0000 7fcb94c00640 1 -- 172.21.15.97:0/532681272 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb901a1520 con 0x7fcb901033c0 2024-03-11T19:58:55.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.850+0000 7fcb86a00640 1 -- 172.21.15.97:0/532681272 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb800107f0 con 0x7fcb901033c0 2024-03-11T19:58:55.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.850+0000 7fcb94c00640 1 -- 172.21.15.97:0/532681272 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb901a19b0 con 0x7fcb901033c0 2024-03-11T19:58:55.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.851+0000 7fcb94c00640 1 -- 172.21.15.97:0/532681272 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9010a680 con 0x7fcb901033c0 2024-03-11T19:58:55.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.853+0000 7fcb86a00640 1 -- 172.21.15.97:0/532681272 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcb80004020 con 0x7fcb901033c0 2024-03-11T19:58:55.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.854+0000 7fcb86a00640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb740a1a30 0x7fcb740a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:55.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.855+0000 7fcb86a00640 1 -- 172.21.15.97:0/532681272 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fcb80014070 con 0x7fcb901033c0 2024-03-11T19:58:55.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.855+0000 7fcb8d400640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb740a1a30 0x7fcb740a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:55.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.855+0000 7fcb8d400640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb740a1a30 0x7fcb740a3ef0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcb78000c00 tx=0x7fcb7802c800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:55.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:55.858+0000 7fcb86a00640 1 -- 172.21.15.97:0/532681272 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcb800d3020 con 0x7fcb901033c0 2024-03-11T19:58:55.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:55 smithi097 ceph-mon[31898]: pgmap v418: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:55.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:55 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1800124756' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:55.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:55 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3585401355' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:58:55.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:55 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3116910930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:58:55.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:55 smithi114 ceph-mon[36376]: pgmap v418: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:55.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:55 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1800124756' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:55.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:55 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3585401355' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:58:55.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:55 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3116910930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:58:56.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:55 smithi161 ceph-mon[36332]: pgmap v418: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:58:56.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:55 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1800124756' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:56.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:55 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3585401355' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T19:58:56.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:55 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3116910930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T19:58:56.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.055+0000 7fcb94c00640 1 -- 172.21.15.97:0/532681272 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb9002ce70 con 0x7fcb901033c0 2024-03-11T19:58:56.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.056+0000 7fcb86a00640 1 -- 172.21.15.97:0/532681272 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcb80096cc0 con 0x7fcb901033c0 2024-03-11T19:58:56.054 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-11T19:58:56.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.059+0000 7fcb94c00640 1 -- 172.21.15.97:0/532681272 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb740a1a30 msgr2=0x7fcb740a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:56.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.059+0000 7fcb94c00640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb740a1a30 0x7fcb740a3ef0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcb78000c00 tx=0x7fcb7802c800 comp rx=0 tx=0).stop 2024-03-11T19:58:56.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.059+0000 7fcb94c00640 1 -- 172.21.15.97:0/532681272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb901033c0 msgr2=0x7fcb90199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:56.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.059+0000 7fcb94c00640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb901033c0 0x7fcb90199420 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000eb70 tx=0x7fcb8000c6a0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.059+0000 7fcb94c00640 1 -- 172.21.15.97:0/532681272 shutdown_connections 2024-03-11T19:58:56.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.059+0000 7fcb94c00640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcb90104f70 0x7fcb901a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.059+0000 7fcb94c00640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcb901045b0 0x7fcb90199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.059+0000 7fcb94c00640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fcb740a1a30 0x7fcb740a3ef0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.059+0000 7fcb94c00640 1 --2- 172.21.15.97:0/532681272 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcb901033c0 0x7fcb90199420 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.059+0000 7fcb94c00640 1 -- 172.21.15.97:0/532681272 >> 172.21.15.97:0/532681272 conn(0x7fcb900fe930 msgr2=0x7fcb90107850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:56.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.059+0000 7fcb94c00640 1 -- 172.21.15.97:0/532681272 shutdown_connections 2024-03-11T19:58:56.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.059+0000 7fcb94c00640 1 -- 172.21.15.97:0/532681272 wait complete. 2024-03-11T19:58:56.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T19:58:56.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.294+0000 7f5162a00640 1 -- 172.21.15.97:0/799902341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f515c073b80 msgr2=0x7f515c073fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:56.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.294+0000 7f5162a00640 1 --2- 172.21.15.97:0/799902341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f515c073b80 0x7f515c073fe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5144009900 tx=0x7f514402bcd0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.294+0000 7f5162a00640 1 -- 172.21.15.97:0/799902341 shutdown_connections 2024-03-11T19:58:56.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.294+0000 7f5162a00640 1 --2- 172.21.15.97:0/799902341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f515c109020 0x7f515c10b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.294+0000 7f5162a00640 1 --2- 172.21.15.97:0/799902341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f515c073b80 0x7f515c073fe0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.294+0000 7f5162a00640 1 --2- 172.21.15.97:0/799902341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f515c075160 0x7f515c0735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.294+0000 7f5162a00640 1 -- 172.21.15.97:0/799902341 >> 172.21.15.97:0/799902341 conn(0x7f515c0fc720 msgr2=0x7f515c0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:56.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.294+0000 7f5162a00640 1 -- 172.21.15.97:0/799902341 shutdown_connections 2024-03-11T19:58:56.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.294+0000 7f5162a00640 1 -- 172.21.15.97:0/799902341 wait complete. 2024-03-11T19:58:56.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.295+0000 7f5162a00640 1 Processor -- start 2024-03-11T19:58:56.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.295+0000 7f5162a00640 1 -- start start 2024-03-11T19:58:56.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.296+0000 7f5162a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f515c073b80 0x7f515c199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:56.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.296+0000 7f5162a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f515c075160 0x7f515c199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:56.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.296+0000 7f5162a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f515c109020 0x7f515c1a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:56.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.296+0000 7f5162a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c1a0f00 con 0x7f515c109020 2024-03-11T19:58:56.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.296+0000 7f5162a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c1a1070 con 0x7f515c073b80 2024-03-11T19:58:56.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.296+0000 7f5162a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c1a1230 con 0x7f515c075160 2024-03-11T19:58:56.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.296+0000 7f5160a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f515c109020 0x7f515c1a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.296+0000 7f515b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f515c075160 0x7f515c199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.296+0000 7f5160a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f515c109020 0x7f515c1a09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49374/0 (socket says 172.21.15.97:49374) 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.296+0000 7f5160a00640 1 -- 172.21.15.97:0/730704970 learned_addr learned my addr 172.21.15.97:0/730704970 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.296+0000 7f515be00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f515c073b80 0x7f515c199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.297+0000 7f5160a00640 1 -- 172.21.15.97:0/730704970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f515c075160 msgr2=0x7f515c199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.297+0000 7f5160a00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f515c075160 0x7f515c199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.297+0000 7f5160a00640 1 -- 172.21.15.97:0/730704970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f515c073b80 msgr2=0x7f515c199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.297+0000 7f5160a00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f515c073b80 0x7f515c199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.297+0000 7f5160a00640 1 -- 172.21.15.97:0/730704970 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c009840 con 0x7f515c109020 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.297+0000 7f515be00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f515c073b80 0x7f515c199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.297+0000 7f5160a00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f515c109020 0x7f515c1a09c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f515400e970 tx=0x7f515400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.297+0000 7f5158c00640 1 -- 172.21.15.97:0/730704970 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154009720 con 0x7f515c109020 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.297+0000 7f5162a00640 1 -- 172.21.15.97:0/730704970 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5144009560 con 0x7f515c109020 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.297+0000 7f5158c00640 1 -- 172.21.15.97:0/730704970 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5154005500 con 0x7f515c109020 2024-03-11T19:58:56.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.297+0000 7f5158c00640 1 -- 172.21.15.97:0/730704970 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154010430 con 0x7f515c109020 2024-03-11T19:58:56.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.297+0000 7f5162a00640 1 -- 172.21.15.97:0/730704970 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515c1a1860 con 0x7f515c109020 2024-03-11T19:58:56.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.298+0000 7f5162a00640 1 -- 172.21.15.97:0/730704970 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515c0683c0 con 0x7f515c109020 2024-03-11T19:58:56.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.301+0000 7f5158c00640 1 -- 172.21.15.97:0/730704970 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5154005020 con 0x7f515c109020 2024-03-11T19:58:56.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.303+0000 7f5158c00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f51400a1a30 0x7f51400a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:56.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.303+0000 7f5158c00640 1 -- 172.21.15.97:0/730704970 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f5154014070 con 0x7f515c109020 2024-03-11T19:58:56.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.304+0000 7f515be00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f51400a1a30 0x7f51400a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:56.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.304+0000 7f515be00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f51400a1a30 0x7f51400a3ef0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f514c009fd0 tx=0x7f514c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:56.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.308+0000 7f5158c00640 1 -- 172.21.15.97:0/730704970 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5154096980 con 0x7f515c109020 2024-03-11T19:58:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.506+0000 7f5162a00640 1 -- 172.21.15.97:0/730704970 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f515c107de0 con 0x7f515c109020 2024-03-11T19:58:56.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.507+0000 7f5158c00640 1 -- 172.21.15.97:0/730704970 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f515c107de0 con 0x7f515c109020 2024-03-11T19:58:56.504 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T19:58:56.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.510+0000 7f5162a00640 1 -- 172.21.15.97:0/730704970 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f51400a1a30 msgr2=0x7f51400a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:56.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.510+0000 7f5162a00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f51400a1a30 0x7f51400a3ef0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f514c009fd0 tx=0x7f514c009500 comp rx=0 tx=0).stop 2024-03-11T19:58:56.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.510+0000 7f5162a00640 1 -- 172.21.15.97:0/730704970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f515c109020 msgr2=0x7f515c1a09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:56.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.510+0000 7f5162a00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f515c109020 0x7f515c1a09c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f515400e970 tx=0x7f515400ed70 comp rx=0 tx=0).stop 2024-03-11T19:58:56.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.510+0000 7f5162a00640 1 -- 172.21.15.97:0/730704970 shutdown_connections 2024-03-11T19:58:56.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.510+0000 7f5162a00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f51400a1a30 0x7f51400a3ef0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.510+0000 7f5162a00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f515c109020 0x7f515c1a09c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.510+0000 7f5162a00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f515c075160 0x7f515c199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.510+0000 7f5162a00640 1 --2- 172.21.15.97:0/730704970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f515c073b80 0x7f515c199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.510+0000 7f5162a00640 1 -- 172.21.15.97:0/730704970 >> 172.21.15.97:0/730704970 conn(0x7f515c0fc720 msgr2=0x7f515c104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:56.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.510+0000 7f5162a00640 1 -- 172.21.15.97:0/730704970 shutdown_connections 2024-03-11T19:58:56.508 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.510+0000 7f5162a00640 1 -- 172.21.15.97:0/730704970 wait complete. 2024-03-11T19:58:56.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T19:58:56.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.743+0000 7f7bddc00640 1 -- 172.21.15.97:0/2792859544 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd81077c0 msgr2=0x7f7bd8107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.743+0000 7f7bddc00640 1 --2- 172.21.15.97:0/2792859544 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd81077c0 0x7f7bd8107be0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009900 tx=0x7f7bc802bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.744+0000 7f7bddc00640 1 -- 172.21.15.97:0/2792859544 shutdown_connections 2024-03-11T19:58:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.744+0000 7f7bddc00640 1 --2- 172.21.15.97:0/2792859544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd8109370 0x7f7bd810fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.744+0000 7f7bddc00640 1 --2- 172.21.15.97:0/2792859544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd81089b0 0x7f7bd8108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.744+0000 7f7bddc00640 1 --2- 172.21.15.97:0/2792859544 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd81077c0 0x7f7bd8107be0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.744+0000 7f7bddc00640 1 -- 172.21.15.97:0/2792859544 >> 172.21.15.97:0/2792859544 conn(0x7f7bd8076790 msgr2=0x7f7bd8078bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.744+0000 7f7bddc00640 1 -- 172.21.15.97:0/2792859544 shutdown_connections 2024-03-11T19:58:56.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.744+0000 7f7bddc00640 1 -- 172.21.15.97:0/2792859544 wait complete. 2024-03-11T19:58:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.745+0000 7f7bddc00640 1 Processor -- start 2024-03-11T19:58:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.745+0000 7f7bddc00640 1 -- start start 2024-03-11T19:58:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.745+0000 7f7bddc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd81077c0 0x7f7bd819d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:56.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.745+0000 7f7bddc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd81089b0 0x7f7bd819ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.745+0000 7f7bddc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd8109370 0x7f7bd81a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.745+0000 7f7bddc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd81a5370 con 0x7f7bd81089b0 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.745+0000 7f7bddc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd81a54e0 con 0x7f7bd8109370 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.745+0000 7f7bddc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd81a56a0 con 0x7f7bd81077c0 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.745+0000 7f7bce000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd81089b0 0x7f7bd819ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.745+0000 7f7bce000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd81089b0 0x7f7bd819ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49394/0 (socket says 172.21.15.97:49394) 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.745+0000 7f7bce000640 1 -- 172.21.15.97:0/548711598 learned_addr learned my addr 172.21.15.97:0/548711598 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.745+0000 7f7bd6a00640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd81077c0 0x7f7bd819d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.745+0000 7f7bd7400640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd8109370 0x7f7bd81a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.746+0000 7f7bce000640 1 -- 172.21.15.97:0/548711598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd81077c0 msgr2=0x7f7bd819d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.746+0000 7f7bce000640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd81077c0 0x7f7bd819d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.746+0000 7f7bce000640 1 -- 172.21.15.97:0/548711598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd8109370 msgr2=0x7f7bd81a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.746+0000 7f7bce000640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd8109370 0x7f7bd81a4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.746+0000 7f7bce000640 1 -- 172.21.15.97:0/548711598 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc8009560 con 0x7f7bd81089b0 2024-03-11T19:58:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.746+0000 7f7bd7400640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd8109370 0x7f7bd81a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.746+0000 7f7bd6a00640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd81077c0 0x7f7bd819d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.746+0000 7f7bce000640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd81089b0 0x7f7bd819ddc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7bc000eb90 tx=0x7f7bc000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.746+0000 7f7bcfe00640 1 -- 172.21.15.97:0/548711598 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc00099b0 con 0x7f7bd81089b0 2024-03-11T19:58:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.746+0000 7f7bcfe00640 1 -- 172.21.15.97:0/548711598 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7bc000ee90 con 0x7f7bd81089b0 2024-03-11T19:58:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.746+0000 7f7bcfe00640 1 -- 172.21.15.97:0/548711598 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc00107a0 con 0x7f7bd81089b0 2024-03-11T19:58:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.746+0000 7f7bddc00640 1 -- 172.21.15.97:0/548711598 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd81a5980 con 0x7f7bd81089b0 2024-03-11T19:58:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.747+0000 7f7bddc00640 1 -- 172.21.15.97:0/548711598 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd81a5e40 con 0x7f7bd81089b0 2024-03-11T19:58:56.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.748+0000 7f7bddc00640 1 -- 172.21.15.97:0/548711598 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd810eae0 con 0x7f7bd81089b0 2024-03-11T19:58:56.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.749+0000 7f7bcfe00640 1 -- 172.21.15.97:0/548711598 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f7bc0005020 con 0x7f7bd81089b0 2024-03-11T19:58:56.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.751+0000 7f7bcfe00640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7bbc0a1a30 0x7f7bbc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:56.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.751+0000 7f7bcfe00640 1 -- 172.21.15.97:0/548711598 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f7bc0014070 con 0x7f7bd81089b0 2024-03-11T19:58:56.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.751+0000 7f7bd6a00640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7bbc0a1a30 0x7f7bbc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:56.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.752+0000 7f7bd6a00640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7bbc0a1a30 0x7f7bbc0a3ef0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7bc802c240 tx=0x7f7bc802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:56.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.754+0000 7f7bcfe00640 1 -- 172.21.15.97:0/548711598 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7bc0096e80 con 0x7f7bd81089b0 2024-03-11T19:58:56.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:56 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/532681272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:58:56.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:56 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/730704970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:58:56.986 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.988+0000 7f7bddc00640 1 -- 172.21.15.97:0/548711598 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bd8113200 con 0x7f7bd81089b0 2024-03-11T19:58:56.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:56 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/532681272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:58:56.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:56 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/730704970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:58:56.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.990+0000 7f7bcfe00640 1 -- 172.21.15.97:0/548711598 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11401 (secure 0 0 0) 0x7f7bc009a540 con 0x7f7bd81089b0 2024-03-11T19:58:56.988 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:56.988 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:55:45.149335+0000","modified":"2024-03-11T19:56:19.631638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24583,"mds_2":24547,"mds_3":24545,"mds_4":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24583":{"gid":24583,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.97:6841/2531110572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2531110572},{"type":"v1","addr":"172.21.15.97:6841","nonce":2531110572}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6839/1624532248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":1624532248},{"type":"v1","addr":"172.21.15.114:6839","nonce":1624532248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T19:58:56.989 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 28 2024-03-11T19:58:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.993+0000 7f7bddc00640 1 -- 172.21.15.97:0/548711598 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7bbc0a1a30 msgr2=0x7f7bbc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.993+0000 7f7bddc00640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7bbc0a1a30 0x7f7bbc0a3ef0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7bc802c240 tx=0x7f7bc802c7e0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.993+0000 7f7bddc00640 1 -- 172.21.15.97:0/548711598 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd81089b0 msgr2=0x7f7bd819ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.993+0000 7f7bddc00640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd81089b0 0x7f7bd819ddc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7bc000eb90 tx=0x7f7bc000c6a0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.993+0000 7f7bddc00640 1 -- 172.21.15.97:0/548711598 shutdown_connections 2024-03-11T19:58:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.993+0000 7f7bddc00640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bd8109370 0x7f7bd81a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.993+0000 7f7bddc00640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7bbc0a1a30 0x7f7bbc0a3ef0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.993+0000 7f7bddc00640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7bd81089b0 0x7f7bd819ddc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.993+0000 7f7bddc00640 1 --2- 172.21.15.97:0/548711598 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd81077c0 0x7f7bd819d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:56.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.993+0000 7f7bddc00640 1 -- 172.21.15.97:0/548711598 >> 172.21.15.97:0/548711598 conn(0x7f7bd8076790 msgr2=0x7f7bd810bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:56.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.994+0000 7f7bddc00640 1 -- 172.21.15.97:0/548711598 shutdown_connections 2024-03-11T19:58:56.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:56.994+0000 7f7bddc00640 1 -- 172.21.15.97:0/548711598 wait complete. 2024-03-11T19:58:57.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T19:58:57.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:56 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/532681272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T19:58:57.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:56 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/730704970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T19:58:57.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.209+0000 7f2ce6c00640 1 -- 172.21.15.97:0/3819738700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce0100d60 msgr2=0x7f2ce01011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.209+0000 7f2ce6c00640 1 --2- 172.21.15.97:0/3819738700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce0100d60 0x7f2ce01011e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8009ca0 tx=0x7f2cd802c100 comp rx=0 tx=0).stop 2024-03-11T19:58:57.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.209+0000 7f2ce6c00640 1 -- 172.21.15.97:0/3819738700 shutdown_connections 2024-03-11T19:58:57.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.209+0000 7f2ce6c00640 1 --2- 172.21.15.97:0/3819738700 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce0100d60 0x7f2ce01011e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.209+0000 7f2ce6c00640 1 --2- 172.21.15.97:0/3819738700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ce01003c0 0x7f2ce0100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.209+0000 7f2ce6c00640 1 --2- 172.21.15.97:0/3819738700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ce00ffa60 0x7f2ce00ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.209+0000 7f2ce6c00640 1 -- 172.21.15.97:0/3819738700 >> 172.21.15.97:0/3819738700 conn(0x7f2ce00fb600 msgr2=0x7f2ce00fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.209+0000 7f2ce6c00640 1 -- 172.21.15.97:0/3819738700 shutdown_connections 2024-03-11T19:58:57.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.210+0000 7f2ce6c00640 1 -- 172.21.15.97:0/3819738700 wait complete. 2024-03-11T19:58:57.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.210+0000 7f2ce6c00640 1 Processor -- start 2024-03-11T19:58:57.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.210+0000 7f2ce6c00640 1 -- start start 2024-03-11T19:58:57.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2ce6c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce00ffa60 0x7f2ce01993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2ce6c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ce01003c0 0x7f2ce01998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2ce6c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ce0100d60 0x7f2ce01a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2ce6c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce01a0e90 con 0x7f2ce00ffa60 2024-03-11T19:58:57.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2ce6c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce01a1000 con 0x7f2ce0100d60 2024-03-11T19:58:57.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2ce6c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce01a11c0 con 0x7f2ce01003c0 2024-03-11T19:58:57.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2cdfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce00ffa60 0x7f2ce01993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2cdfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce00ffa60 0x7f2ce01993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49410/0 (socket says 172.21.15.97:49410) 2024-03-11T19:58:57.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2cdfe00640 1 -- 172.21.15.97:0/1021023062 learned_addr learned my addr 172.21.15.97:0/1021023062 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:57.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2cdf400640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ce01003c0 0x7f2ce01998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2ce4c00640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ce0100d60 0x7f2ce01a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2cdfe00640 1 -- 172.21.15.97:0/1021023062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ce01003c0 msgr2=0x7f2ce01998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2cdfe00640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ce01003c0 0x7f2ce01998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2cdfe00640 1 -- 172.21.15.97:0/1021023062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ce0100d60 msgr2=0x7f2ce01a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2cdfe00640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ce0100d60 0x7f2ce01a0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2cdfe00640 1 -- 172.21.15.97:0/1021023062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc009490 con 0x7f2ce00ffa60 2024-03-11T19:58:57.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.211+0000 7f2ce4c00640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ce0100d60 0x7f2ce01a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:57.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.212+0000 7f2cdfe00640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce00ffa60 0x7f2ce01993a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000e9f0 tx=0x7f2cd000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.212+0000 7f2cdcc00640 1 -- 172.21.15.97:0/1021023062 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0009720 con 0x7f2ce00ffa60 2024-03-11T19:58:57.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.212+0000 7f2ce6c00640 1 -- 172.21.15.97:0/1021023062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd8009900 con 0x7f2ce00ffa60 2024-03-11T19:58:57.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.212+0000 7f2cdcc00640 1 -- 172.21.15.97:0/1021023062 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cd000cd80 con 0x7f2ce00ffa60 2024-03-11T19:58:57.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.212+0000 7f2cdcc00640 1 -- 172.21.15.97:0/1021023062 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0010690 con 0x7f2ce00ffa60 2024-03-11T19:58:57.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.212+0000 7f2ce6c00640 1 -- 172.21.15.97:0/1021023062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce01a17d0 con 0x7f2ce00ffa60 2024-03-11T19:58:57.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.213+0000 7f2ce6c00640 1 -- 172.21.15.97:0/1021023062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce010a8f0 con 0x7f2ce00ffa60 2024-03-11T19:58:57.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.215+0000 7f2cdcc00640 1 -- 172.21.15.97:0/1021023062 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2cd00046c0 con 0x7f2ce00ffa60 2024-03-11T19:58:57.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.216+0000 7f2cdcc00640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2cc40a1a30 0x7f2cc40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.216+0000 7f2cdcc00640 1 -- 172.21.15.97:0/1021023062 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f2cd0014070 con 0x7f2ce00ffa60 2024-03-11T19:58:57.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.217+0000 7f2cdf400640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2cc40a1a30 0x7f2cc40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.217+0000 7f2cdf400640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2cc40a1a30 0x7f2cc40a3ef0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2ce019a8b0 tx=0x7f2ccc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.220+0000 7f2cdcc00640 1 -- 172.21.15.97:0/1021023062 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2cd0096a30 con 0x7f2ce00ffa60 2024-03-11T19:58:57.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.419+0000 7f2ce6c00640 1 -- 172.21.15.97:0/1021023062 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ce002ce40 con 0x7f2ce00ffa60 2024-03-11T19:58:57.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.423+0000 7f2cdcc00640 1 -- 172.21.15.97:0/1021023062 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24838 (secure 0 0 0) 0x7f2cd009a0f0 con 0x7f2ce00ffa60 2024-03-11T19:58:57.421 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T19:58:57.421 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":93,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T19:57:46.674203+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T19:58:57.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.426+0000 7f2ce6c00640 1 -- 172.21.15.97:0/1021023062 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2cc40a1a30 msgr2=0x7f2cc40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.426+0000 7f2ce6c00640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2cc40a1a30 0x7f2cc40a3ef0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2ce019a8b0 tx=0x7f2ccc005f90 comp rx=0 tx=0).stop 2024-03-11T19:58:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.426+0000 7f2ce6c00640 1 -- 172.21.15.97:0/1021023062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce00ffa60 msgr2=0x7f2ce01993a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.426+0000 7f2ce6c00640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce00ffa60 0x7f2ce01993a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000e9f0 tx=0x7f2cd000edf0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.426+0000 7f2ce6c00640 1 -- 172.21.15.97:0/1021023062 shutdown_connections 2024-03-11T19:58:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.426+0000 7f2ce6c00640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ce0100d60 0x7f2ce01a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.426+0000 7f2ce6c00640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ce01003c0 0x7f2ce01998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.426+0000 7f2ce6c00640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f2cc40a1a30 0x7f2cc40a3ef0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.426+0000 7f2ce6c00640 1 --2- 172.21.15.97:0/1021023062 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2ce00ffa60 0x7f2ce01993a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.426+0000 7f2ce6c00640 1 -- 172.21.15.97:0/1021023062 >> 172.21.15.97:0/1021023062 conn(0x7f2ce00fb600 msgr2=0x7f2ce00fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.426+0000 7f2ce6c00640 1 -- 172.21.15.97:0/1021023062 shutdown_connections 2024-03-11T19:58:57.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.426+0000 7f2ce6c00640 1 -- 172.21.15.97:0/1021023062 wait complete. 2024-03-11T19:58:57.441 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-11T19:55:45.149335+0000', 'modified': '2024-03-11T19:56:19.631638+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24599, 'mds_1': 24583, 'mds_2': 24547, 'mds_3': 24545, 'mds_4': 24464}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24454': {'gid': 24454, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6833/2213740234', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6832', 'nonce': 2213740234}, {'type': 'v1', 'addr': '172.21.15.161:6833', 'nonce': 2213740234}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24464': {'gid': 24464, 'name': 'k', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.114:6833/2550912009', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6832', 'nonce': 2550912009}, {'type': 'v1', 'addr': '172.21.15.114:6833', 'nonce': 2550912009}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24499': {'gid': 24499, 'name': 'd', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.97:6837/267259401', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 267259401}, {'type': 'v1', 'addr': '172.21.15.97:6837', 'nonce': 267259401}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24512': {'gid': 24512, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.114:6835/2623690462', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 2623690462}, {'type': 'v1', 'addr': '172.21.15.114:6835', 'nonce': 2623690462}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24517': {'gid': 24517, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6835/3358622196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6834', 'nonce': 3358622196}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 3358622196}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24529': {'gid': 24529, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.161:6837/1600156948', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6836', 'nonce': 1600156948}, {'type': 'v1', 'addr': '172.21.15.161:6837', 'nonce': 1600156948}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24545': {'gid': 24545, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.114:6837/1112816294', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6836', 'nonce': 1112816294}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1112816294}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.161:6839/565994162', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6838', 'nonce': 565994162}, {'type': 'v1', 'addr': '172.21.15.161:6839', 'nonce': 565994162}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24583': {'gid': 24583, 'name': 'a', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.97:6841/2531110572', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6840', 'nonce': 2531110572}, {'type': 'v1', 'addr': '172.21.15.97:6841', 'nonce': 2531110572}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24599': {'gid': 24599, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.114:6839/1624532248', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6838', 'nonce': 1624532248}, {'type': 'v1', 'addr': '172.21.15.114:6839', 'nonce': 1624532248}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-11T19:58:57.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-11T19:58:57.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-11T19:58:57.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.654+0000 7f92b5e00640 1 -- 172.21.15.97:0/3409186692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b0100360 msgr2=0x7f92b01007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.654+0000 7f92b5e00640 1 --2- 172.21.15.97:0/3409186692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b0100360 0x7f92b01007c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9298009900 tx=0x7f929802bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:57.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.654+0000 7f92b5e00640 1 -- 172.21.15.97:0/3409186692 shutdown_connections 2024-03-11T19:58:57.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.654+0000 7f92b5e00640 1 --2- 172.21.15.97:0/3409186692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b0100d00 0x7f92b0101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.654+0000 7f92b5e00640 1 --2- 172.21.15.97:0/3409186692 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b0100360 0x7f92b01007c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.654+0000 7f92b5e00640 1 --2- 172.21.15.97:0/3409186692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92b00ffa00 0x7f92b00ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.654+0000 7f92b5e00640 1 -- 172.21.15.97:0/3409186692 >> 172.21.15.97:0/3409186692 conn(0x7f92b00fb620 msgr2=0x7f92b00fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.654+0000 7f92b5e00640 1 -- 172.21.15.97:0/3409186692 shutdown_connections 2024-03-11T19:58:57.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.655+0000 7f92b5e00640 1 -- 172.21.15.97:0/3409186692 wait complete. 2024-03-11T19:58:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.655+0000 7f92b5e00640 1 Processor -- start 2024-03-11T19:58:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.656+0000 7f92b5e00640 1 -- start start 2024-03-11T19:58:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.656+0000 7f92b5e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b00ffa00 0x7f92b0199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.656+0000 7f92b5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92b0100360 0x7f92b0199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.656+0000 7f92b5e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b0100d00 0x7f92b01a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.656+0000 7f92b5e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b01a0f30 con 0x7f92b0100d00 2024-03-11T19:58:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.656+0000 7f92b5e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b01a10a0 con 0x7f92b0100360 2024-03-11T19:58:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.656+0000 7f92b5e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b01a1260 con 0x7f92b00ffa00 2024-03-11T19:58:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.656+0000 7f92af400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b0100d00 0x7f92b01a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.656+0000 7f92aea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b00ffa00 0x7f92b0199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.656+0000 7f92aea00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b00ffa00 0x7f92b0199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:57856/0 (socket says 172.21.15.97:57856) 2024-03-11T19:58:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.656+0000 7f92aea00640 1 -- 172.21.15.97:0/1164885878 learned_addr learned my addr 172.21.15.97:0/1164885878 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.656+0000 7f92a6000640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92b0100360 0x7f92b0199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.657+0000 7f92aea00640 1 -- 172.21.15.97:0/1164885878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92b0100360 msgr2=0x7f92b0199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.657+0000 7f92aea00640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92b0100360 0x7f92b0199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.657+0000 7f92aea00640 1 -- 172.21.15.97:0/1164885878 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b0100d00 msgr2=0x7f92b01a09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.657+0000 7f92aea00640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b0100d00 0x7f92b01a09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.657+0000 7f92aea00640 1 -- 172.21.15.97:0/1164885878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9298009560 con 0x7f92b00ffa00 2024-03-11T19:58:57.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.657+0000 7f92a6000640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92b0100360 0x7f92b0199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:57.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.657+0000 7f92aea00640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b00ffa00 0x7f92b0199440 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f92a000eb90 tx=0x7f92a000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.657+0000 7f92b4a00640 1 -- 172.21.15.97:0/1164885878 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a00099b0 con 0x7f92b00ffa00 2024-03-11T19:58:57.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.657+0000 7f92b5e00640 1 -- 172.21.15.97:0/1164885878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b01a1540 con 0x7f92b00ffa00 2024-03-11T19:58:57.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.657+0000 7f92b4a00640 1 -- 172.21.15.97:0/1164885878 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92a000ee90 con 0x7f92b00ffa00 2024-03-11T19:58:57.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.657+0000 7f92b4a00640 1 -- 172.21.15.97:0/1164885878 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a00107a0 con 0x7f92b00ffa00 2024-03-11T19:58:57.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.657+0000 7f92b5e00640 1 -- 172.21.15.97:0/1164885878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b01a1a00 con 0x7f92b00ffa00 2024-03-11T19:58:57.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.661+0000 7f92b8b68740 1 Processor -- start 2024-03-11T19:58:57.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.661+0000 7f92b8b68740 1 -- start start 2024-03-11T19:58:57.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92b8b68740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268a580 0x5644c268a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92b8b68740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268af70 0x5644c268fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92b8b68740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c26900e0 0x5644c2692570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92b8b68740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c25f9b30 con 0x5644c268af70 2024-03-11T19:58:57.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92b8b68740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c25f7060 con 0x5644c26900e0 2024-03-11T19:58:57.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92b8b68740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c25f2840 con 0x5644c268a580 2024-03-11T19:58:57.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92b4a00640 1 -- 172.21.15.97:0/1164885878 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f92a0005020 con 0x7f92b00ffa00 2024-03-11T19:58:57.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92af400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268a580 0x5644c268a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92af400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268a580 0x5644c268a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:57858/0 (socket says 172.21.15.97:57858) 2024-03-11T19:58:57.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92af400640 1 -- 172.21.15.97:0/952879421 learned_addr learned my addr 172.21.15.97:0/952879421 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:57.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92a6000640 1 --2- 172.21.15.97:0/952879421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c26900e0 0x5644c2692570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92aea00640 1 --2- 172.21.15.97:0/952879421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268af70 0x5644c268fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92af400640 1 -- 172.21.15.97:0/952879421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c26900e0 msgr2=0x5644c2692570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92af400640 1 --2- 172.21.15.97:0/952879421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c26900e0 0x5644c2692570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92af400640 1 -- 172.21.15.97:0/952879421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268af70 msgr2=0x5644c268fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92af400640 1 --2- 172.21.15.97:0/952879421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268af70 0x5644c268fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92af400640 1 -- 172.21.15.97:0/952879421 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644c25ba4c0 con 0x5644c268a580 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.662+0000 7f92aea00640 1 --2- 172.21.15.97:0/952879421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268af70 0x5644c268fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.663+0000 7f92af400640 1 --2- 172.21.15.97:0/952879421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268a580 0x5644c268a9a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009ca0 tx=0x7f92a8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=618b109bdbb40bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.663+0000 7f929fe00640 1 -- 172.21.15.97:0/952879421 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8019630 con 0x5644c268a580 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.663+0000 7f929fe00640 1 -- 172.21.15.97:0/952879421 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92a8009e50 con 0x5644c268a580 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.663+0000 7f929fe00640 1 -- 172.21.15.97:0/952879421 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a800ec00 con 0x5644c268a580 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.663+0000 7f92b4a00640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f92940a1a30 0x7f92940a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92b4a00640 1 -- 172.21.15.97:0/1164885878 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f92a0014070 con 0x7f92b00ffa00 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92aea00640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f92940a1a30 0x7f92940a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92b8b68740 1 -- 172.21.15.97:0/952879421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268a580 msgr2=0x5644c268a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92b8b68740 1 --2- 172.21.15.97:0/952879421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268a580 0x5644c268a9a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009ca0 tx=0x7f92a8011810 comp rx=0 tx=0).stop 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92b8b68740 1 -- 172.21.15.97:0/952879421 shutdown_connections 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92b8b68740 1 --2- 172.21.15.97:0/952879421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c26900e0 0x5644c2692570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92b8b68740 1 --2- 172.21.15.97:0/952879421 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268af70 0x5644c268fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92b8b68740 1 --2- 172.21.15.97:0/952879421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268a580 0x5644c268a9a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92b8b68740 1 -- 172.21.15.97:0/952879421 >> 172.21.15.97:0/952879421 conn(0x5644c2685af0 msgr2=0x5644c2685ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92b8b68740 1 -- 172.21.15.97:0/952879421 shutdown_connections 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92b8b68740 1 -- 172.21.15.97:0/952879421 wait complete. 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92aea00640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f92940a1a30 0x7f92940a3ef0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f92a00cbae0 tx=0x7f92a000cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92b8b68740 1 -- start start 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.664+0000 7f92b8b68740 1 Processor -- start 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.665+0000 7f92b8b68740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268a580 0x5644c26943b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.665+0000 7f92b8b68740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c268af70 0x5644c26948f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.665+0000 7f92b8b68740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c26900e0 0x5644c269b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.665+0000 7f92b8b68740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c2597ea0 con 0x5644c268a580 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.665+0000 7f92b8b68740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c2597b40 con 0x5644c268af70 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.665+0000 7f92b8b68740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c233af20 con 0x5644c26900e0 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.665+0000 7f92b8b68740 10 client.0 _ll_register_callbacks cb 0x5644c25ab950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.665+0000 7f92af400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268a580 0x5644c26943b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.665+0000 7f92af400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268a580 0x5644c26943b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49454/0 (socket says 172.21.15.97:49454) 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.665+0000 7f92af400640 1 -- 172.21.15.97:0/3400845041 learned_addr learned my addr 172.21.15.97:0/3400845041 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.665+0000 7f92a6000640 1 --2- 172.21.15.97:0/3400845041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c268af70 0x5644c26948f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.665+0000 7f92af400640 1 --2- 172.21.15.97:0/3400845041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c26900e0 0x5644c269b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.666+0000 7f92af400640 1 -- 172.21.15.97:0/3400845041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c26900e0 msgr2=0x5644c269b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.666+0000 7f92af400640 1 --2- 172.21.15.97:0/3400845041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c26900e0 0x5644c269b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.666+0000 7f92af400640 1 -- 172.21.15.97:0/3400845041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c268af70 msgr2=0x5644c26948f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.666+0000 7f92af400640 1 --2- 172.21.15.97:0/3400845041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c268af70 0x5644c26948f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.666+0000 7f92af400640 1 -- 172.21.15.97:0/3400845041 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a8009950 con 0x5644c268a580 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.666+0000 7f92af400640 1 --2- 172.21.15.97:0/3400845041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268a580 0x5644c26943b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f92a8011d10 tx=0x7f92a8005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21db2c4ff1082381 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.666+0000 7f929cc00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.666+0000 7f929cc00640 1 -- 172.21.15.97:0/3400845041 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8005050 con 0x5644c268a580 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.666+0000 7f929cc00640 1 -- 172.21.15.97:0/3400845041 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f92a80051b0 con 0x5644c268a580 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.666+0000 7f92b8b68740 1 -- 172.21.15.97:0/3400845041 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5644c2616c40 con 0x5644c268a580 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.666+0000 7f929cc00640 1 -- 172.21.15.97:0/3400845041 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92a8005370 con 0x5644c268a580 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.666+0000 7f92b8b68740 10 client.15288 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.666+0000 7f92b8b68740 1 -- 172.21.15.97:0/3400845041 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5644c25f0ec0 con 0x5644c268a580 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.667+0000 7f929cc00640 1 -- 172.21.15.97:0/3400845041 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5644c2616c40 con 0x5644c268a580 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.667+0000 7f929cc00640 1 -- 172.21.15.97:0/3400845041 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f92a8020070 con 0x5644c268a580 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.667+0000 7f92b8b68740 10 client.15288 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.667+0000 7f92b8b68740 10 client.15288 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.667+0000 7f92b8b68740 20 client.15288 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.667+0000 7f92b8b68740 1 --2- 172.21.15.97:0/3400845041 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5644c26a01c0 0x5644c26a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.667+0000 7f92b8b68740 4 client.15288 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:58:57.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.667+0000 7f92b8b68740 1 -- 172.21.15.97:0/3400845041 --> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5644c22f26c0 con 0x5644c26a01c0 2024-03-11T19:58:57.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.667+0000 7f92a6000640 1 --2- 172.21.15.97:0/3400845041 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5644c26a01c0 0x5644c26a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.668+0000 7f929cc00640 10 client.15288 ms_handle_connect on v2:172.21.15.114:6832/2550912009 2024-03-11T19:58:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.668+0000 7f92a6000640 1 --2- 172.21.15.97:0/3400845041 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5644c26a01c0 0x5644c26a4c60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cec607a39b2a4459 server_cookie=5965237618630467 in_seq=0 out_seq=0 2024-03-11T19:58:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.669+0000 7f929cc00640 1 -- 172.21.15.97:0/3400845041 <== mds.4 v2:172.21.15.114:6832/2550912009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5644c22f26c0 con 0x5644c26a01c0 2024-03-11T19:58:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.669+0000 7f929cc00640 10 client.15288 handle_command_reply: tid=0 2024-03-11T19:58:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.669+0000 7f92b8b68740 1 client.15288 shutdown 2024-03-11T19:58:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.669+0000 7f92b8b68740 1 -- 172.21.15.97:0/3400845041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268a580 msgr2=0x5644c26943b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.669+0000 7f92b8b68740 1 --2- 172.21.15.97:0/3400845041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268a580 0x5644c26943b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f92a8011d10 tx=0x7f92a8005c50 comp rx=0 tx=0).stop 2024-03-11T19:58:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.669+0000 7f92b8b68740 1 -- 172.21.15.97:0/3400845041 shutdown_connections 2024-03-11T19:58:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.669+0000 7f92b8b68740 1 --2- 172.21.15.97:0/3400845041 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5644c26a01c0 0x5644c26a4c60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.669+0000 7f92b8b68740 1 --2- 172.21.15.97:0/3400845041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c26900e0 0x5644c269b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.669+0000 7f92b8b68740 1 --2- 172.21.15.97:0/3400845041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c268af70 0x5644c26948f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.669+0000 7f929cc00640 0 client.15288 ms_handle_reset on v2:172.21.15.114:6832/2550912009 2024-03-11T19:58:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.669+0000 7f92b8b68740 1 --2- 172.21.15.97:0/3400845041 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268a580 0x5644c26943b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.669+0000 7f92b8b68740 1 -- 172.21.15.97:0/3400845041 >> 172.21.15.97:0/3400845041 conn(0x5644c2685af0 msgr2=0x5644c2684e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.670+0000 7f92b8b68740 1 -- 172.21.15.97:0/3400845041 shutdown_connections 2024-03-11T19:58:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.670+0000 7f92b8b68740 1 -- 172.21.15.97:0/3400845041 wait complete. 2024-03-11T19:58:57.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.670+0000 7f92b8b68740 20 client.15288 trim_cache size 0 max 16384 2024-03-11T19:58:57.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.687+0000 7f92b8b68740 1 Processor -- start 2024-03-11T19:58:57.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.687+0000 7f92b8b68740 1 -- start start 2024-03-11T19:58:57.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.687+0000 7f92b8b68740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c2685af0 0x5644c2689460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.687+0000 7f92b8b68740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268a580 0x5644c269b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.687+0000 7f92b8b68740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268af70 0x5644c268ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.687+0000 7f92b8b68740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c2681100 con 0x5644c268a580 2024-03-11T19:58:57.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.687+0000 7f92b8b68740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c269fd70 con 0x5644c2685af0 2024-03-11T19:58:57.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.687+0000 7f92b8b68740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c26a27c0 con 0x5644c268af70 2024-03-11T19:58:57.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.688+0000 7f92af400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c2685af0 0x5644c2689460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.688+0000 7f92a6000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268a580 0x5644c269b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.688+0000 7f92af400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c2685af0 0x5644c2689460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:51110/0 (socket says 172.21.15.97:51110) 2024-03-11T19:58:57.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.688+0000 7f92af400640 1 -- 172.21.15.97:0/3020551830 learned_addr learned my addr 172.21.15.97:0/3020551830 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:57.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.688+0000 7f92af400640 1 --2- 172.21.15.97:0/3020551830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268af70 0x5644c268ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.688+0000 7f92a6000640 1 -- 172.21.15.97:0/3020551830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268af70 msgr2=0x5644c268ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.688+0000 7f92a6000640 1 --2- 172.21.15.97:0/3020551830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268af70 0x5644c268ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.688+0000 7f92a6000640 1 -- 172.21.15.97:0/3020551830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c2685af0 msgr2=0x5644c2689460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.688+0000 7f92a6000640 1 --2- 172.21.15.97:0/3020551830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c2685af0 0x5644c2689460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.688+0000 7f92a6000640 1 -- 172.21.15.97:0/3020551830 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644c269feb0 con 0x5644c268a580 2024-03-11T19:58:57.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.688+0000 7f92af400640 1 --2- 172.21.15.97:0/3020551830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c2685af0 0x5644c2689460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:57.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.688+0000 7f92a6000640 1 --2- 172.21.15.97:0/3020551830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268a580 0x5644c269b8c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9298036040 tx=0x7f92980058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfda113c4bde6718 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.689+0000 7f929f400640 1 -- 172.21.15.97:0/3020551830 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9298048de0 con 0x5644c268a580 2024-03-11T19:58:57.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.689+0000 7f929f400640 1 -- 172.21.15.97:0/3020551830 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f929802cca0 con 0x5644c268a580 2024-03-11T19:58:57.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.689+0000 7f929f400640 1 -- 172.21.15.97:0/3020551830 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9298048720 con 0x5644c268a580 2024-03-11T19:58:57.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.689+0000 7f92b8b68740 1 -- 172.21.15.97:0/3020551830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268a580 msgr2=0x5644c269b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.689+0000 7f92b8b68740 1 --2- 172.21.15.97:0/3020551830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268a580 0x5644c269b8c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9298036040 tx=0x7f92980058b0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.689+0000 7f92b8b68740 1 -- 172.21.15.97:0/3020551830 shutdown_connections 2024-03-11T19:58:57.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.689+0000 7f92b8b68740 1 --2- 172.21.15.97:0/3020551830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268af70 0x5644c268ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.689+0000 7f92b8b68740 1 --2- 172.21.15.97:0/3020551830 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c268a580 0x5644c269b8c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.689+0000 7f92b8b68740 1 --2- 172.21.15.97:0/3020551830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c2685af0 0x5644c2689460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.689+0000 7f92b8b68740 1 -- 172.21.15.97:0/3020551830 >> 172.21.15.97:0/3020551830 conn(0x5644c2686240 msgr2=0x5644c2688d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.690+0000 7f92b8b68740 1 -- 172.21.15.97:0/3020551830 shutdown_connections 2024-03-11T19:58:57.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.690+0000 7f92b8b68740 1 -- 172.21.15.97:0/3020551830 wait complete. 2024-03-11T19:58:57.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.690+0000 7f92b8b68740 1 -- start start 2024-03-11T19:58:57.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.690+0000 7f92b8b68740 1 Processor -- start 2024-03-11T19:58:57.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.690+0000 7f92b8b68740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c2685af0 0x5644c2699020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92b8b68740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c268a580 0x5644c2699560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92b8b68740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268af70 0x5644c269a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92b8b68740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c269aab0 con 0x5644c2685af0 2024-03-11T19:58:57.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92b8b68740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c269ac20 con 0x5644c268a580 2024-03-11T19:58:57.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92b8b68740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644c269ade0 con 0x5644c268af70 2024-03-11T19:58:57.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92b8b68740 10 client.0 _ll_register_callbacks cb 0x7f92a801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:57.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92af400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c2685af0 0x5644c2699020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92af400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c2685af0 0x5644c2699020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49460/0 (socket says 172.21.15.97:49460) 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92af400640 1 -- 172.21.15.97:0/1485040662 learned_addr learned my addr 172.21.15.97:0/1485040662 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92a6000640 1 --2- 172.21.15.97:0/1485040662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c268a580 0x5644c2699560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92af400640 1 --2- 172.21.15.97:0/1485040662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268af70 0x5644c269a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92af400640 1 -- 172.21.15.97:0/1485040662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268af70 msgr2=0x5644c269a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92af400640 1 --2- 172.21.15.97:0/1485040662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268af70 0x5644c269a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92af400640 1 -- 172.21.15.97:0/1485040662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c268a580 msgr2=0x5644c2699560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92af400640 1 --2- 172.21.15.97:0/1485040662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c268a580 0x5644c2699560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92af400640 1 -- 172.21.15.97:0/1485040662 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644c25f0ec0 con 0x5644c2685af0 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f92af400640 1 --2- 172.21.15.97:0/1485040662 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c2685af0 0x5644c2699020 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f92a8003c90 tx=0x7f92a8004350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bc73db57c3384f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.691+0000 7f929cc00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.692+0000 7f929cc00640 1 -- 172.21.15.97:0/1485040662 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8003d10 con 0x5644c2685af0 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.692+0000 7f929cc00640 1 -- 172.21.15.97:0/1485040662 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f92a8000e40 con 0x5644c2685af0 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.692+0000 7f92b8b68740 1 -- 172.21.15.97:0/1485040662 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f92a8009e50 con 0x5644c2685af0 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.692+0000 7f929cc00640 1 -- 172.21.15.97:0/1485040662 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92a8003e70 con 0x5644c2685af0 2024-03-11T19:58:57.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.692+0000 7f92b8b68740 10 client.15294 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.692+0000 7f92b8b68740 1 -- 172.21.15.97:0/1485040662 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f92b010c0d0 con 0x5644c2685af0 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.692+0000 7f929cc00640 1 -- 172.21.15.97:0/1485040662 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92a8009e50 con 0x5644c2685af0 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.692+0000 7f929cc00640 1 -- 172.21.15.97:0/1485040662 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f92a8020070 con 0x5644c2685af0 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.692+0000 7f92b8b68740 10 client.15294 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.692+0000 7f92b8b68740 10 client.15294 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.692+0000 7f92b8b68740 20 client.15294 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.692+0000 7f92b8b68740 1 --2- 172.21.15.97:0/1485040662 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5644c26900e0 0x5644c26f8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.693+0000 7f92b8b68740 4 client.15294 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.693+0000 7f92b8b68740 1 -- 172.21.15.97:0/1485040662 --> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5644c2343270 con 0x5644c26900e0 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.693+0000 7f92a6000640 1 --2- 172.21.15.97:0/1485040662 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5644c26900e0 0x5644c26f8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.693+0000 7f929cc00640 10 client.15294 ms_handle_connect on v2:172.21.15.114:6832/2550912009 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.693+0000 7f92a6000640 1 --2- 172.21.15.97:0/1485040662 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5644c26900e0 0x5644c26f8da0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4e8dc9a9432f1d06 server_cookie=18e3d2a1c6a42403 in_seq=0 out_seq=0 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f929cc00640 1 -- 172.21.15.97:0/1485040662 <== mds.4 v2:172.21.15.114:6832/2550912009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5644c2343270 con 0x5644c26900e0 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f929cc00640 10 client.15294 handle_command_reply: tid=0 2024-03-11T19:58:57.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f92b8b68740 1 client.15294 shutdown 2024-03-11T19:58:57.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f92b8b68740 1 -- 172.21.15.97:0/1485040662 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c2685af0 msgr2=0x5644c2699020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f92b8b68740 1 --2- 172.21.15.97:0/1485040662 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c2685af0 0x5644c2699020 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f92a8003c90 tx=0x7f92a8004350 comp rx=0 tx=0).stop 2024-03-11T19:58:57.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f92b8b68740 1 -- 172.21.15.97:0/1485040662 shutdown_connections 2024-03-11T19:58:57.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f92b8b68740 1 --2- 172.21.15.97:0/1485040662 >> [v2:172.21.15.114:6832/2550912009,v1:172.21.15.114:6833/2550912009] conn(0x5644c26900e0 0x5644c26f8da0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f92b8b68740 1 --2- 172.21.15.97:0/1485040662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5644c268af70 0x5644c269a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f92b8b68740 1 --2- 172.21.15.97:0/1485040662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644c268a580 0x5644c2699560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f929cc00640 0 client.15294 ms_handle_reset on v2:172.21.15.114:6832/2550912009 2024-03-11T19:58:57.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f92b8b68740 1 --2- 172.21.15.97:0/1485040662 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5644c2685af0 0x5644c2699020 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f92b8b68740 1 -- 172.21.15.97:0/1485040662 >> 172.21.15.97:0/1485040662 conn(0x5644c2686240 msgr2=0x5644c26895e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f92b8b68740 1 -- 172.21.15.97:0/1485040662 shutdown_connections 2024-03-11T19:58:57.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f92b8b68740 1 -- 172.21.15.97:0/1485040662 wait complete. 2024-03-11T19:58:57.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.694+0000 7f92b8b68740 20 client.15294 trim_cache size 0 max 16384 2024-03-11T19:58:57.697 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T19:58:57.697 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:58:57.697 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 4, 2024-03-11T19:58:57.697 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24464, 2024-03-11T19:58:57.697 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-11T19:58:57.697 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-11T19:58:57.697 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-11T19:58:57.697 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 159.067922176, 2024-03-11T19:58:57.697 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-03-11T19:58:57.697 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 93, 2024-03-11T19:58:57.697 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 89, 2024-03-11T19:58:57.697 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 232.36680073100001 2024-03-11T19:58:57.697 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T19:58:57.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.695+0000 7f92b5e00640 1 -- 172.21.15.97:0/1164885878 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f92940a1a30 msgr2=0x7f92940a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.695+0000 7f92b5e00640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f92940a1a30 0x7f92940a3ef0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f92a00cbae0 tx=0x7f92a000cf90 comp rx=0 tx=0).stop 2024-03-11T19:58:57.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.695+0000 7f92b5e00640 1 -- 172.21.15.97:0/1164885878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b00ffa00 msgr2=0x7f92b0199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.695+0000 7f92b5e00640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b00ffa00 0x7f92b0199440 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f92a000eb90 tx=0x7f92a000c6a0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.695+0000 7f92b5e00640 1 -- 172.21.15.97:0/1164885878 shutdown_connections 2024-03-11T19:58:57.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.695+0000 7f92b5e00640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f92940a1a30 0x7f92940a3ef0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.695+0000 7f92b5e00640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f92b0100d00 0x7f92b01a09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.695+0000 7f92b5e00640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92b0100360 0x7f92b0199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.695+0000 7f92b5e00640 1 --2- 172.21.15.97:0/1164885878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b00ffa00 0x7f92b0199440 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.695+0000 7f92b5e00640 1 -- 172.21.15.97:0/1164885878 >> 172.21.15.97:0/1164885878 conn(0x7f92b00fb620 msgr2=0x7f92b00fce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.695+0000 7f92b5e00640 1 -- 172.21.15.97:0/1164885878 shutdown_connections 2024-03-11T19:58:57.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.695+0000 7f92b5e00640 1 -- 172.21.15.97:0/1164885878 wait complete. 2024-03-11T19:58:57.708 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-11T19:58:57.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.903+0000 7f8699c00640 1 -- 172.21.15.97:0/922962948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86941045b0 msgr2=0x7f8694104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.903+0000 7f8699c00640 1 --2- 172.21.15.97:0/922962948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86941045b0 0x7f8694104a30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f867c009860 tx=0x7f867c02bb70 comp rx=0 tx=0).stop 2024-03-11T19:58:57.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.903+0000 7f8699c00640 1 -- 172.21.15.97:0/922962948 shutdown_connections 2024-03-11T19:58:57.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.903+0000 7f8699c00640 1 --2- 172.21.15.97:0/922962948 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8694104f70 0x7f869410b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.903+0000 7f8699c00640 1 --2- 172.21.15.97:0/922962948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86941045b0 0x7f8694104a30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.903+0000 7f8699c00640 1 --2- 172.21.15.97:0/922962948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86941033c0 0x7f86941037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.903+0000 7f8699c00640 1 -- 172.21.15.97:0/922962948 >> 172.21.15.97:0/922962948 conn(0x7f86940fe930 msgr2=0x7f8694100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.903+0000 7f8699c00640 1 -- 172.21.15.97:0/922962948 shutdown_connections 2024-03-11T19:58:57.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.904+0000 7f8699c00640 1 -- 172.21.15.97:0/922962948 wait complete. 2024-03-11T19:58:57.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.904+0000 7f8699c00640 1 Processor -- start 2024-03-11T19:58:57.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8699c00640 1 -- start start 2024-03-11T19:58:57.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8699c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86941033c0 0x7f8694199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8699c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86941045b0 0x7f8694199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8699c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8694104f70 0x7f86941a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8699c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86941a0f30 con 0x7f8694104f70 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8699c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86941a10a0 con 0x7f86941045b0 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8699c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86941a1260 con 0x7f86941033c0 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8692a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86941033c0 0x7f8694199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8692a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86941033c0 0x7f8694199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:57884/0 (socket says 172.21.15.97:57884) 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8692a00640 1 -- 172.21.15.97:0/2532553210 learned_addr learned my addr 172.21.15.97:0/2532553210 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8693400640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8694104f70 0x7f86941a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8692000640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86941045b0 0x7f8694199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8692a00640 1 -- 172.21.15.97:0/2532553210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86941045b0 msgr2=0x7f8694199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8692a00640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86941045b0 0x7f8694199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8692a00640 1 -- 172.21.15.97:0/2532553210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8694104f70 msgr2=0x7f86941a09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8692a00640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8694104f70 0x7f86941a09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.905+0000 7f8692a00640 1 -- 172.21.15.97:0/2532553210 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8684009560 con 0x7f86941033c0 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.906+0000 7f8693400640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8694104f70 0x7f86941a09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:57.905 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.906+0000 7f8692a00640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86941033c0 0x7f8694199440 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f868c00ebc0 tx=0x7f868c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.906+0000 7f868b400640 1 -- 172.21.15.97:0/2532553210 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c003b70 con 0x7f86941033c0 2024-03-11T19:58:57.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.906+0000 7f8699c00640 1 -- 172.21.15.97:0/2532553210 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867c009490 con 0x7f86941033c0 2024-03-11T19:58:57.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.906+0000 7f868b400640 1 -- 172.21.15.97:0/2532553210 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f868c00eec0 con 0x7f86941033c0 2024-03-11T19:58:57.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.906+0000 7f868b400640 1 -- 172.21.15.97:0/2532553210 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c019800 con 0x7f86941033c0 2024-03-11T19:58:57.906 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.906+0000 7f8699c00640 1 -- 172.21.15.97:0/2532553210 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86941a1890 con 0x7f86941033c0 2024-03-11T19:58:57.907 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.910+0000 7f869ca0c740 1 Processor -- start 2024-03-11T19:58:57.908 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.910+0000 7f869ca0c740 1 -- start start 2024-03-11T19:58:57.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.911+0000 7f869ca0c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0580 0x5652367c09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.911+0000 7f869ca0c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0f70 0x5652367c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.911+0000 7f869ca0c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367c8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.909 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.911+0000 7f869ca0c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x56523672fb30 con 0x5652367c0580 2024-03-11T19:58:57.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.911+0000 7f869ca0c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56523672d060 con 0x5652367c0f70 2024-03-11T19:58:57.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.911+0000 7f869ca0c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x565236728840 con 0x5652367c60e0 2024-03-11T19:58:57.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.911+0000 7f868b400640 1 -- 172.21.15.97:0/2532553210 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f868c019960 con 0x7f86941033c0 2024-03-11T19:58:57.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.911+0000 7f8693400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0580 0x5652367c09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.911+0000 7f8692000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.911+0000 7f8692000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:57890/0 (socket says 172.21.15.97:57890) 2024-03-11T19:58:57.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.911+0000 7f8693400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0580 0x5652367c09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49494/0 (socket says 172.21.15.97:49494) 2024-03-11T19:58:57.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.911+0000 7f8692000640 1 -- 172.21.15.97:0/3652011306 learned_addr learned my addr 172.21.15.97:0/3652011306 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:57.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.911+0000 7f8692a00640 1 --2- 172.21.15.97:0/3652011306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0f70 0x5652367c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.912+0000 7f8692000640 1 -- 172.21.15.97:0/3652011306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0f70 msgr2=0x5652367c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.912+0000 7f8692000640 1 --2- 172.21.15.97:0/3652011306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0f70 0x5652367c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.912+0000 7f8692000640 1 -- 172.21.15.97:0/3652011306 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0580 msgr2=0x5652367c09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.910 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.912+0000 7f8692000640 1 --2- 172.21.15.97:0/3652011306 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0580 0x5652367c09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.911 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.912+0000 7f8692000640 1 -- 172.21.15.97:0/3652011306 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652366f04c0 con 0x5652367c60e0 2024-03-11T19:58:57.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.912+0000 7f8692000640 1 --2- 172.21.15.97:0/3652011306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367c8570 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f867c000c00 tx=0x7f867c032450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8b3313595fe6b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.912+0000 7f8683e00640 1 -- 172.21.15.97:0/3652011306 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c004030 con 0x5652367c60e0 2024-03-11T19:58:57.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.912+0000 7f8683e00640 1 -- 172.21.15.97:0/3652011306 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f867c034440 con 0x5652367c60e0 2024-03-11T19:58:57.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.912+0000 7f8683e00640 1 -- 172.21.15.97:0/3652011306 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c02c510 con 0x5652367c60e0 2024-03-11T19:58:57.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.913+0000 7f868b400640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f86780a1a30 0x7f86780a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.913+0000 7f868b400640 1 -- 172.21.15.97:0/2532553210 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f868c014070 con 0x7f86941033c0 2024-03-11T19:58:57.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.913+0000 7f8693400640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f86780a1a30 0x7f86780a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.913+0000 7f869ca0c740 1 -- 172.21.15.97:0/3652011306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 msgr2=0x5652367c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.913+0000 7f869ca0c740 1 --2- 172.21.15.97:0/3652011306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367c8570 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f867c000c00 tx=0x7f867c032450 comp rx=0 tx=0).stop 2024-03-11T19:58:57.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.913+0000 7f869ca0c740 1 -- 172.21.15.97:0/3652011306 shutdown_connections 2024-03-11T19:58:57.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.913+0000 7f869ca0c740 1 --2- 172.21.15.97:0/3652011306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367c8570 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.913+0000 7f869ca0c740 1 --2- 172.21.15.97:0/3652011306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0f70 0x5652367c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.913+0000 7f869ca0c740 1 --2- 172.21.15.97:0/3652011306 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0580 0x5652367c09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.913+0000 7f869ca0c740 1 -- 172.21.15.97:0/3652011306 >> 172.21.15.97:0/3652011306 conn(0x5652367bbaf0 msgr2=0x5652367bbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.913+0000 7f869ca0c740 1 -- 172.21.15.97:0/3652011306 shutdown_connections 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.913+0000 7f8693400640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f86780a1a30 0x7f86780a3ef0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x5652367c17d0 tx=0x7f8684009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.913+0000 7f869ca0c740 1 -- 172.21.15.97:0/3652011306 wait complete. 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.914+0000 7f869ca0c740 1 -- start start 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.914+0000 7f869ca0c740 1 Processor -- start 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f869ca0c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0580 0x5652367ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f869ca0c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0f70 0x5652367ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f869ca0c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f869ca0c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652366cdea0 con 0x5652367c0580 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f869ca0c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652366cdb40 con 0x5652367c0f70 2024-03-11T19:58:57.916 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f869ca0c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x565236470f20 con 0x5652367c60e0 2024-03-11T19:58:57.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f869ca0c740 10 client.0 _ll_register_callbacks cb 0x5652366e1950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:57.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f8692a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0580 0x5652367ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f8692000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0f70 0x5652367ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f8692000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0f70 0x5652367ca8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:51156/0 (socket says 172.21.15.97:51156) 2024-03-11T19:58:57.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f8693400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f8692000640 1 -- 172.21.15.97:0/420808552 learned_addr learned my addr 172.21.15.97:0/420808552 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:57.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f8692a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0580 0x5652367ca3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49498/0 (socket says 172.21.15.97:49498) 2024-03-11T19:58:57.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f8692a00640 1 -- 172.21.15.97:0/420808552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 msgr2=0x5652367d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f8692a00640 1 --2- 172.21.15.97:0/420808552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367d1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f8692a00640 1 -- 172.21.15.97:0/420808552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0f70 msgr2=0x5652367ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f8692a00640 1 --2- 172.21.15.97:0/420808552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0f70 0x5652367ca8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f8692a00640 1 -- 172.21.15.97:0/420808552 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869410c100 con 0x5652367c0580 2024-03-11T19:58:57.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f8692000640 1 --2- 172.21.15.97:0/420808552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0f70 0x5652367ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:57.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.915+0000 7f8693400640 1 --2- 172.21.15.97:0/420808552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367d1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:57.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.916+0000 7f8692a00640 1 --2- 172.21.15.97:0/420808552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0580 0x5652367ca3a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x5652367c19e0 tx=0x7f868c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d344bda73eeef21e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.916+0000 7f8680c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:57.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.916+0000 7f8680c00640 1 -- 172.21.15.97:0/420808552 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c09a6d0 con 0x5652367c0580 2024-03-11T19:58:57.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.916+0000 7f8680c00640 1 -- 172.21.15.97:0/420808552 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f868c003cd0 con 0x5652367c0580 2024-03-11T19:58:57.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.916+0000 7f869ca0c740 1 -- 172.21.15.97:0/420808552 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56523674cc40 con 0x5652367c0580 2024-03-11T19:58:57.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.916+0000 7f8680c00640 1 -- 172.21.15.97:0/420808552 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f868c09ad70 con 0x5652367c0580 2024-03-11T19:58:57.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.916+0000 7f869ca0c740 10 client.15306 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:57.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.916+0000 7f869ca0c740 1 -- 172.21.15.97:0/420808552 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x565236726ec0 con 0x5652367c0580 2024-03-11T19:58:57.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.916+0000 7f8680c00640 1 -- 172.21.15.97:0/420808552 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56523674cc40 con 0x5652367c0580 2024-03-11T19:58:57.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.917+0000 7f8680c00640 1 -- 172.21.15.97:0/420808552 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f868c004b60 con 0x5652367c0580 2024-03-11T19:58:57.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.917+0000 7f869ca0c740 10 client.15306 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:57.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.917+0000 7f869ca0c740 10 client.15306 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-11T19:58:57.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.917+0000 7f869ca0c740 20 client.15306 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:57.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.917+0000 7f869ca0c740 1 --2- 172.21.15.97:0/420808552 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x5652367d5cb0 0x5652367dadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.917+0000 7f869ca0c740 4 client.15306 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:58:57.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.917+0000 7f869ca0c740 1 -- 172.21.15.97:0/420808552 --> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652364286c0 con 0x5652367d5cb0 2024-03-11T19:58:57.919 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.917+0000 7f8692000640 1 --2- 172.21.15.97:0/420808552 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x5652367d5cb0 0x5652367dadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.918+0000 7f8692000640 1 --2- 172.21.15.97:0/420808552 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x5652367d5cb0 0x5652367dadc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9322549a9939e6a0 server_cookie=39c0d4efdeb51bea in_seq=0 out_seq=0 2024-03-11T19:58:57.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.918+0000 7f8680c00640 10 client.15306 ms_handle_connect on v2:172.21.15.114:6836/1112816294 2024-03-11T19:58:57.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f8680c00640 1 -- 172.21.15.97:0/420808552 <== mds.3 v2:172.21.15.114:6836/1112816294 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5652364286c0 con 0x5652367d5cb0 2024-03-11T19:58:57.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f8680c00640 10 client.15306 handle_command_reply: tid=0 2024-03-11T19:58:57.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f869ca0c740 1 client.15306 shutdown 2024-03-11T19:58:57.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f869ca0c740 1 -- 172.21.15.97:0/420808552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0580 msgr2=0x5652367ca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f869ca0c740 1 --2- 172.21.15.97:0/420808552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0580 0x5652367ca3a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x5652367c19e0 tx=0x7f868c003f90 comp rx=0 tx=0).stop 2024-03-11T19:58:57.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f869ca0c740 1 -- 172.21.15.97:0/420808552 shutdown_connections 2024-03-11T19:58:57.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f869ca0c740 1 --2- 172.21.15.97:0/420808552 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x5652367d5cb0 0x5652367dadc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f869ca0c740 1 --2- 172.21.15.97:0/420808552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367d1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f869ca0c740 1 --2- 172.21.15.97:0/420808552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0f70 0x5652367ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f8680c00640 0 client.15306 ms_handle_reset on v2:172.21.15.114:6836/1112816294 2024-03-11T19:58:57.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f869ca0c740 1 --2- 172.21.15.97:0/420808552 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0580 0x5652367ca3a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f869ca0c740 1 -- 172.21.15.97:0/420808552 >> 172.21.15.97:0/420808552 conn(0x5652367bbaf0 msgr2=0x5652367bae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f869ca0c740 1 -- 172.21.15.97:0/420808552 shutdown_connections 2024-03-11T19:58:57.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f869ca0c740 1 -- 172.21.15.97:0/420808552 wait complete. 2024-03-11T19:58:57.921 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.919+0000 7f869ca0c740 20 client.15306 trim_cache size 0 max 16384 2024-03-11T19:58:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.936+0000 7f869ca0c740 1 Processor -- start 2024-03-11T19:58:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.936+0000 7f869ca0c740 1 -- start start 2024-03-11T19:58:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.936+0000 7f869ca0c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c0580 0x5652367ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.936+0000 7f869ca0c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 0x5652367d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.936+0000 7f869ca0c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c60e0 0x5652367dede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.936+0000 7f869ca0c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652367c9bf0 con 0x5652367c0f70 2024-03-11T19:58:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.936+0000 7f869ca0c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652367d8920 con 0x5652367c60e0 2024-03-11T19:58:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.936+0000 7f869ca0c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652367c8c30 con 0x5652367c0580 2024-03-11T19:58:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.937+0000 7f8692000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 0x5652367d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.937+0000 7f8692a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c0580 0x5652367ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.937+0000 7f8692000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 0x5652367d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49508/0 (socket says 172.21.15.97:49508) 2024-03-11T19:58:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.937+0000 7f8692000640 1 -- 172.21.15.97:0/2286251831 learned_addr learned my addr 172.21.15.97:0/2286251831 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:57.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.937+0000 7f8693400640 1 --2- 172.21.15.97:0/2286251831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c60e0 0x5652367dede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.937+0000 7f8692a00640 1 -- 172.21.15.97:0/2286251831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c60e0 msgr2=0x5652367dede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.937+0000 7f8692a00640 1 --2- 172.21.15.97:0/2286251831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c60e0 0x5652367dede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.937+0000 7f8692a00640 1 -- 172.21.15.97:0/2286251831 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 msgr2=0x5652367d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.937+0000 7f8692a00640 1 --2- 172.21.15.97:0/2286251831 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 0x5652367d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.937+0000 7f8692a00640 1 -- 172.21.15.97:0/2286251831 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652366c38b0 con 0x5652367c0580 2024-03-11T19:58:57.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.937+0000 7f8693400640 1 --2- 172.21.15.97:0/2286251831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c60e0 0x5652367dede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:57.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.937+0000 7f8692000640 1 --2- 172.21.15.97:0/2286251831 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 0x5652367d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:57.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.937+0000 7f8692a00640 1 --2- 172.21.15.97:0/2286251831 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c0580 0x5652367ca6c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f868c099020 tx=0x7f868c009a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f699fa8a13bad8fa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.938+0000 7f8683400640 1 -- 172.21.15.97:0/2286251831 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c0a3220 con 0x5652367c0580 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.938+0000 7f8683400640 1 -- 172.21.15.97:0/2286251831 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f868c09a6d0 con 0x5652367c0580 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.938+0000 7f8683400640 1 -- 172.21.15.97:0/2286251831 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c004c70 con 0x5652367c0580 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.938+0000 7f869ca0c740 1 -- 172.21.15.97:0/2286251831 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c0580 msgr2=0x5652367ca6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.938+0000 7f869ca0c740 1 --2- 172.21.15.97:0/2286251831 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c0580 0x5652367ca6c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f868c099020 tx=0x7f868c009a40 comp rx=0 tx=0).stop 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.938+0000 7f869ca0c740 1 -- 172.21.15.97:0/2286251831 shutdown_connections 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.938+0000 7f869ca0c740 1 --2- 172.21.15.97:0/2286251831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c60e0 0x5652367dede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.938+0000 7f869ca0c740 1 --2- 172.21.15.97:0/2286251831 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 0x5652367d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.938+0000 7f869ca0c740 1 --2- 172.21.15.97:0/2286251831 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c0580 0x5652367ca6c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.938+0000 7f869ca0c740 1 -- 172.21.15.97:0/2286251831 >> 172.21.15.97:0/2286251831 conn(0x5652367bbaf0 msgr2=0x5652367c9fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.939+0000 7f869ca0c740 1 -- 172.21.15.97:0/2286251831 shutdown_connections 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.939+0000 7f869ca0c740 1 -- 172.21.15.97:0/2286251831 wait complete. 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.939+0000 7f869ca0c740 1 -- start start 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.939+0000 7f869ca0c740 1 Processor -- start 2024-03-11T19:58:57.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.939+0000 7f869ca0c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0580 0x5652367cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.939+0000 7f869ca0c740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 0x5652367cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.939+0000 7f869ca0c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.939+0000 7f869ca0c740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652367d1430 con 0x5652367c0f70 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.939+0000 7f869ca0c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652367d15a0 con 0x5652367c0580 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.939+0000 7f869ca0c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652367d1760 con 0x5652367c60e0 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.939+0000 7f869ca0c740 10 client.0 _ll_register_callbacks cb 0x7f868c009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.940+0000 7f8692000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 0x5652367cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.940+0000 7f8692a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0580 0x5652367cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.940+0000 7f8692000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 0x5652367cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49518/0 (socket says 172.21.15.97:49518) 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.940+0000 7f8692000640 1 -- 172.21.15.97:0/246796743 learned_addr learned my addr 172.21.15.97:0/246796743 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.940+0000 7f8692a00640 1 --2- 172.21.15.97:0/246796743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0580 0x5652367cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:51176/0 (socket says 172.21.15.97:51176) 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.940+0000 7f8693400640 1 --2- 172.21.15.97:0/246796743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.940+0000 7f8692000640 1 -- 172.21.15.97:0/246796743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 msgr2=0x5652367d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.940+0000 7f8692000640 1 --2- 172.21.15.97:0/246796743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367d0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.940+0000 7f8692000640 1 -- 172.21.15.97:0/246796743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0580 msgr2=0x5652367cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.940+0000 7f8692000640 1 --2- 172.21.15.97:0/246796743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0580 0x5652367cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.940+0000 7f8692000640 1 -- 172.21.15.97:0/246796743 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652366f04c0 con 0x5652367c0f70 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.940+0000 7f8692000640 1 --2- 172.21.15.97:0/246796743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 0x5652367cdfb0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f867c02c180 tx=0x7f867c02c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c462ccb8d354f0c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.940+0000 7f8680c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.941+0000 7f8680c00640 1 -- 172.21.15.97:0/246796743 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c0345b0 con 0x5652367c0f70 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.941+0000 7f869ca0c740 1 -- 172.21.15.97:0/246796743 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5652367d6650 con 0x5652367c0f70 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.941+0000 7f8680c00640 1 -- 172.21.15.97:0/246796743 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f867c04c040 con 0x5652367c0f70 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.941+0000 7f8680c00640 1 -- 172.21.15.97:0/246796743 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f867c0348d0 con 0x5652367c0f70 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.941+0000 7f869ca0c740 10 client.15312 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.941+0000 7f869ca0c740 1 -- 172.21.15.97:0/246796743 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x565236726ec0 con 0x5652367c0f70 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.942+0000 7f8680c00640 1 -- 172.21.15.97:0/246796743 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5652367d6650 con 0x5652367c0f70 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.942+0000 7f8680c00640 1 -- 172.21.15.97:0/246796743 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f867c02c510 con 0x5652367c0f70 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.942+0000 7f869ca0c740 10 client.15312 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:57.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.942+0000 7f869ca0c740 10 client.15312 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-11T19:58:57.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.942+0000 7f869ca0c740 20 client.15312 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:57.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.942+0000 7f869ca0c740 1 --2- 172.21.15.97:0/246796743 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x565236831290 0x56523682d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:57.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.942+0000 7f869ca0c740 4 client.15312 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:58:57.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.942+0000 7f869ca0c740 1 -- 172.21.15.97:0/246796743 --> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f867c034440 con 0x565236831290 2024-03-11T19:58:57.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.942+0000 7f8692a00640 1 --2- 172.21.15.97:0/246796743 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x565236831290 0x56523682d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:57.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f8692a00640 1 --2- 172.21.15.97:0/246796743 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x565236831290 0x56523682d750 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8f05d69e54858391 server_cookie=c770eba4ef3ce9c3 in_seq=0 out_seq=0 2024-03-11T19:58:57.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f8680c00640 10 client.15312 ms_handle_connect on v2:172.21.15.114:6836/1112816294 2024-03-11T19:58:57.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f8680c00640 1 -- 172.21.15.97:0/246796743 <== mds.3 v2:172.21.15.114:6836/1112816294 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f867c034440 con 0x565236831290 2024-03-11T19:58:57.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f8680c00640 10 client.15312 handle_command_reply: tid=0 2024-03-11T19:58:57.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f869ca0c740 1 client.15312 shutdown 2024-03-11T19:58:57.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f869ca0c740 1 -- 172.21.15.97:0/246796743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 msgr2=0x5652367cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f869ca0c740 1 --2- 172.21.15.97:0/246796743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 0x5652367cdfb0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f867c02c180 tx=0x7f867c02c1f0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f869ca0c740 1 -- 172.21.15.97:0/246796743 shutdown_connections 2024-03-11T19:58:57.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f869ca0c740 1 --2- 172.21.15.97:0/246796743 >> [v2:172.21.15.114:6836/1112816294,v1:172.21.15.114:6837/1112816294] conn(0x565236831290 0x56523682d750 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f869ca0c740 1 --2- 172.21.15.97:0/246796743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5652367c60e0 0x5652367d0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f869ca0c740 1 --2- 172.21.15.97:0/246796743 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5652367c0f70 0x5652367cdfb0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f8680c00640 0 client.15312 ms_handle_reset on v2:172.21.15.114:6836/1112816294 2024-03-11T19:58:57.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f869ca0c740 1 --2- 172.21.15.97:0/246796743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5652367c0580 0x5652367cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.943+0000 7f869ca0c740 1 -- 172.21.15.97:0/246796743 >> 172.21.15.97:0/246796743 conn(0x5652367bbaf0 msgr2=0x5652367dede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.944+0000 7f869ca0c740 1 -- 172.21.15.97:0/246796743 shutdown_connections 2024-03-11T19:58:57.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.944+0000 7f869ca0c740 1 -- 172.21.15.97:0/246796743 wait complete. 2024-03-11T19:58:57.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.944+0000 7f869ca0c740 20 client.15312 trim_cache size 0 max 16384 2024-03-11T19:58:57.945 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T19:58:57.946 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:58:57.946 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 3, 2024-03-11T19:58:57.946 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24545, 2024-03-11T19:58:57.946 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-11T19:58:57.946 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-11T19:58:57.946 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-11T19:58:57.946 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 160.291150542, 2024-03-11T19:58:57.946 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-03-11T19:58:57.946 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 93, 2024-03-11T19:58:57.946 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 89, 2024-03-11T19:58:57.946 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 212.99677799200001 2024-03-11T19:58:57.946 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T19:58:57.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.944+0000 7f8699c00640 1 -- 172.21.15.97:0/2532553210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f86780a1a30 msgr2=0x7f86780a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.944+0000 7f8699c00640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f86780a1a30 0x7f86780a3ef0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x5652367c17d0 tx=0x7f8684009260 comp rx=0 tx=0).stop 2024-03-11T19:58:57.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.944+0000 7f8699c00640 1 -- 172.21.15.97:0/2532553210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86941033c0 msgr2=0x7f8694199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:57.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.944+0000 7f8699c00640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86941033c0 0x7f8694199440 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f868c00ebc0 tx=0x7f868c00c6a0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.944+0000 7f8699c00640 1 -- 172.21.15.97:0/2532553210 shutdown_connections 2024-03-11T19:58:57.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.944+0000 7f8699c00640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f86780a1a30 0x7f86780a3ef0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.944+0000 7f8699c00640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8694104f70 0x7f86941a09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.944+0000 7f8699c00640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86941045b0 0x7f8694199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.945+0000 7f8699c00640 1 --2- 172.21.15.97:0/2532553210 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86941033c0 0x7f8694199440 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:57.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.945+0000 7f8699c00640 1 -- 172.21.15.97:0/2532553210 >> 172.21.15.97:0/2532553210 conn(0x7f86940fe930 msgr2=0x7f86941078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:57.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.945+0000 7f8699c00640 1 -- 172.21.15.97:0/2532553210 shutdown_connections 2024-03-11T19:58:57.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:57.945+0000 7f8699c00640 1 -- 172.21.15.97:0/2532553210 wait complete. 2024-03-11T19:58:57.957 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:57 smithi097 ceph-mon[31898]: pgmap v419: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:57.957 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:57 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/548711598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:57.957 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:57 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1021023062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:57.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-11T19:58:57.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:57 smithi114 ceph-mon[36376]: pgmap v419: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:57.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:57 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/548711598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:57.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:57 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1021023062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:58.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:57 smithi161 ceph-mon[36332]: pgmap v419: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-11T19:58:58.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:57 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/548711598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T19:58:58.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:57 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1021023062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T19:58:58.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.162+0000 7f62e1600640 1 -- 172.21.15.97:0/2942502869 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62dc108ae0 msgr2=0x7f62dc10f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.162+0000 7f62e1600640 1 --2- 172.21.15.97:0/2942502869 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62dc108ae0 0x7f62dc10f4c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009900 tx=0x7f62c802bc40 comp rx=0 tx=0).stop 2024-03-11T19:58:58.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.163+0000 7f62e1600640 1 -- 172.21.15.97:0/2942502869 shutdown_connections 2024-03-11T19:58:58.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.163+0000 7f62e1600640 1 --2- 172.21.15.97:0/2942502869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62dc10fa00 0x7f62dc111df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.163+0000 7f62e1600640 1 --2- 172.21.15.97:0/2942502869 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62dc108ae0 0x7f62dc10f4c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.163+0000 7f62e1600640 1 --2- 172.21.15.97:0/2942502869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62dc108180 0x7f62dc1085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.163+0000 7f62e1600640 1 -- 172.21.15.97:0/2942502869 >> 172.21.15.97:0/2942502869 conn(0x7f62dc06e380 msgr2=0x7f62dc0707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.163+0000 7f62e1600640 1 -- 172.21.15.97:0/2942502869 shutdown_connections 2024-03-11T19:58:58.161 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.163+0000 7f62e1600640 1 -- 172.21.15.97:0/2942502869 wait complete. 2024-03-11T19:58:58.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.164+0000 7f62e1600640 1 Processor -- start 2024-03-11T19:58:58.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.164+0000 7f62e1600640 1 -- start start 2024-03-11T19:58:58.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.164+0000 7f62e1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62dc108180 0x7f62dc10ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.164+0000 7f62e1600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62dc108ae0 0x7f62dc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.164+0000 7f62e1600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62dc10fa00 0x7f62dc109990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.164+0000 7f62e1600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc10a0f0 con 0x7f62dc108ae0 2024-03-11T19:58:58.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.164+0000 7f62e1600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc10a260 con 0x7f62dc108180 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.164+0000 7f62e1600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc1a9a90 con 0x7f62dc10fa00 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.164+0000 7f62da000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62dc108ae0 0x7f62dc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.165+0000 7f62daa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62dc108180 0x7f62dc10ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.165+0000 7f62da000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62dc108ae0 0x7f62dc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49540/0 (socket says 172.21.15.97:49540) 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.165+0000 7f62da000640 1 -- 172.21.15.97:0/3939561818 learned_addr learned my addr 172.21.15.97:0/3939561818 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.165+0000 7f62db400640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62dc10fa00 0x7f62dc109990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.165+0000 7f62daa00640 1 -- 172.21.15.97:0/3939561818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62dc10fa00 msgr2=0x7f62dc109990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.165+0000 7f62daa00640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62dc10fa00 0x7f62dc109990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.165+0000 7f62daa00640 1 -- 172.21.15.97:0/3939561818 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62dc108ae0 msgr2=0x7f62dc109450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.165+0000 7f62daa00640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62dc108ae0 0x7f62dc109450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.165+0000 7f62daa00640 1 -- 172.21.15.97:0/3939561818 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c8009560 con 0x7f62dc108180 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.165+0000 7f62da000640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62dc108ae0 0x7f62dc109450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.165+0000 7f62daa00640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62dc108180 0x7f62dc10ae10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00ea20 tx=0x7f62cc00eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.165+0000 7f62d3400640 1 -- 172.21.15.97:0/3939561818 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc00ccc0 con 0x7f62dc108180 2024-03-11T19:58:58.164 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.166+0000 7f62d3400640 1 -- 172.21.15.97:0/3939561818 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62cc00ce20 con 0x7f62dc108180 2024-03-11T19:58:58.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.166+0000 7f62e1600640 1 -- 172.21.15.97:0/3939561818 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62dc1a9d40 con 0x7f62dc108180 2024-03-11T19:58:58.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.166+0000 7f62d3400640 1 -- 172.21.15.97:0/3939561818 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc018840 con 0x7f62dc108180 2024-03-11T19:58:58.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.166+0000 7f62e1600640 1 -- 172.21.15.97:0/3939561818 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62dc1aa1a0 con 0x7f62dc108180 2024-03-11T19:58:58.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62e41fa740 1 Processor -- start 2024-03-11T19:58:58.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62e41fa740 1 -- start start 2024-03-11T19:58:58.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62d3400640 1 -- 172.21.15.97:0/3939561818 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f62cc0189a0 con 0x7f62dc108180 2024-03-11T19:58:58.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62e41fa740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf182e580 0x560cf182e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62e41fa740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182ef70 0x560cf1833a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62e41fa740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf18340e0 0x560cf1836570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62e41fa740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf179db30 con 0x560cf18340e0 2024-03-11T19:58:58.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62e41fa740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf179b060 con 0x560cf182e580 2024-03-11T19:58:58.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62e41fa740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf1796840 con 0x560cf182ef70 2024-03-11T19:58:58.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62db400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf182e580 0x560cf182e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62da000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf18340e0 0x560cf1836570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62db400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf182e580 0x560cf182e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:51200/0 (socket says 172.21.15.97:51200) 2024-03-11T19:58:58.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62daa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182ef70 0x560cf1833a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.169+0000 7f62db400640 1 -- 172.21.15.97:0/3009358589 learned_addr learned my addr 172.21.15.97:0/3009358589 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.170+0000 7f62db400640 1 -- 172.21.15.97:0/3009358589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182ef70 msgr2=0x560cf1833a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.170+0000 7f62db400640 1 --2- 172.21.15.97:0/3009358589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182ef70 0x560cf1833a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.170+0000 7f62db400640 1 -- 172.21.15.97:0/3009358589 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf18340e0 msgr2=0x560cf1836570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.170+0000 7f62db400640 1 --2- 172.21.15.97:0/3009358589 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf18340e0 0x560cf1836570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.170+0000 7f62db400640 1 -- 172.21.15.97:0/3009358589 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cf175e4c0 con 0x560cf182e580 2024-03-11T19:58:58.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.170+0000 7f62db400640 1 --2- 172.21.15.97:0/3009358589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf182e580 0x560cf182e9a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f62d40039f0 tx=0x7f62d4003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bec12daafe388549 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.170+0000 7f62c7400640 1 -- 172.21.15.97:0/3009358589 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d401a470 con 0x560cf182e580 2024-03-11T19:58:58.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.170+0000 7f62c7400640 1 -- 172.21.15.97:0/3009358589 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62d4011920 con 0x560cf182e580 2024-03-11T19:58:58.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.170+0000 7f62c7400640 1 -- 172.21.15.97:0/3009358589 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d4010a70 con 0x560cf182e580 2024-03-11T19:58:58.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.171+0000 7f62d3400640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f62c00a1a30 0x7f62c00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.171+0000 7f62d3400640 1 -- 172.21.15.97:0/3939561818 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f62cc014070 con 0x7f62dc108180 2024-03-11T19:58:58.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.171+0000 7f62daa00640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f62c00a1a30 0x7f62c00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.171+0000 7f62e41fa740 1 -- 172.21.15.97:0/3009358589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf182e580 msgr2=0x560cf182e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.171+0000 7f62e41fa740 1 --2- 172.21.15.97:0/3009358589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf182e580 0x560cf182e9a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f62d40039f0 tx=0x7f62d4003eb0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.171+0000 7f62e41fa740 1 -- 172.21.15.97:0/3009358589 shutdown_connections 2024-03-11T19:58:58.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.171+0000 7f62e41fa740 1 --2- 172.21.15.97:0/3009358589 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf18340e0 0x560cf1836570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.171+0000 7f62e41fa740 1 --2- 172.21.15.97:0/3009358589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182ef70 0x560cf1833a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.171+0000 7f62e41fa740 1 --2- 172.21.15.97:0/3009358589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf182e580 0x560cf182e9a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.171+0000 7f62e41fa740 1 -- 172.21.15.97:0/3009358589 >> 172.21.15.97:0/3009358589 conn(0x560cf1829af0 msgr2=0x560cf1829ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.171+0000 7f62e41fa740 1 -- 172.21.15.97:0/3009358589 shutdown_connections 2024-03-11T19:58:58.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.171+0000 7f62e41fa740 1 -- 172.21.15.97:0/3009358589 wait complete. 2024-03-11T19:58:58.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.172+0000 7f62daa00640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f62c00a1a30 0x7f62c00a3ef0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f62cc096ab0 tx=0x7f62cc0d3530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.172+0000 7f62e41fa740 1 -- start start 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.172+0000 7f62e41fa740 1 Processor -- start 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.172+0000 7f62e41fa740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf182e580 0x560cf1838400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.172+0000 7f62e41fa740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182ef70 0x560cf1838940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.172+0000 7f62e41fa740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf18340e0 0x560cf183f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.172+0000 7f62e41fa740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf173bea0 con 0x560cf18340e0 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.172+0000 7f62e41fa740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf173bb40 con 0x560cf182e580 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.172+0000 7f62e41fa740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf14def20 con 0x560cf182ef70 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.172+0000 7f62e41fa740 10 client.0 _ll_register_callbacks cb 0x560cf174f950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.173+0000 7f62db400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf182e580 0x560cf1838400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.173+0000 7f62db400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf18340e0 0x560cf183f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.173+0000 7f62da000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182ef70 0x560cf1838940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.173+0000 7f62db400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf18340e0 0x560cf183f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49560/0 (socket says 172.21.15.97:49560) 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.173+0000 7f62db400640 1 -- 172.21.15.97:0/2474009441 learned_addr learned my addr 172.21.15.97:0/2474009441 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.173+0000 7f62db400640 1 -- 172.21.15.97:0/2474009441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182ef70 msgr2=0x560cf1838940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.173+0000 7f62db400640 1 --2- 172.21.15.97:0/2474009441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182ef70 0x560cf1838940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.173+0000 7f62db400640 1 -- 172.21.15.97:0/2474009441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf182e580 msgr2=0x560cf1838400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.173+0000 7f62db400640 1 --2- 172.21.15.97:0/2474009441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf182e580 0x560cf1838400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.173+0000 7f62db400640 1 -- 172.21.15.97:0/2474009441 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d4009c90 con 0x560cf18340e0 2024-03-11T19:58:58.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.174+0000 7f62db400640 1 --2- 172.21.15.97:0/2474009441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf18340e0 0x560cf183f980 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f62d4005fd0 tx=0x7f62d4005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1390efead628a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.174+0000 7f62bbe00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:58.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.174+0000 7f62bbe00640 1 -- 172.21.15.97:0/2474009441 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d401e040 con 0x560cf18340e0 2024-03-11T19:58:58.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.174+0000 7f62bbe00640 1 -- 172.21.15.97:0/2474009441 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f62d4004a40 con 0x560cf18340e0 2024-03-11T19:58:58.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.174+0000 7f62e41fa740 1 -- 172.21.15.97:0/2474009441 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560cf17bac40 con 0x560cf18340e0 2024-03-11T19:58:58.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.174+0000 7f62bbe00640 1 -- 172.21.15.97:0/2474009441 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62d4004c00 con 0x560cf18340e0 2024-03-11T19:58:58.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.174+0000 7f62e41fa740 10 client.15324 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:58.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.174+0000 7f62e41fa740 1 -- 172.21.15.97:0/2474009441 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560cf1794ec0 con 0x560cf18340e0 2024-03-11T19:58:58.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.174+0000 7f62bbe00640 1 -- 172.21.15.97:0/2474009441 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560cf17bac40 con 0x560cf18340e0 2024-03-11T19:58:58.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.175+0000 7f62bbe00640 1 -- 172.21.15.97:0/2474009441 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f62d40050e0 con 0x560cf18340e0 2024-03-11T19:58:58.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.175+0000 7f62e41fa740 10 client.15324 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:58.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.175+0000 7f62e41fa740 10 client.15324 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-11T19:58:58.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.175+0000 7f62e41fa740 20 client.15324 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:58.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.175+0000 7f62e41fa740 1 --2- 172.21.15.97:0/2474009441 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x560cf1843c40 0x560cf1848ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.175+0000 7f62e41fa740 4 client.15324 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:58:58.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.175+0000 7f62e41fa740 1 -- 172.21.15.97:0/2474009441 --> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560cf14966c0 con 0x560cf1843c40 2024-03-11T19:58:58.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.175+0000 7f62da000640 1 --2- 172.21.15.97:0/2474009441 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x560cf1843c40 0x560cf1848ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.176+0000 7f62da000640 1 --2- 172.21.15.97:0/2474009441 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x560cf1843c40 0x560cf1848ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6493702185013d98 server_cookie=376ea8392612c948 in_seq=0 out_seq=0 2024-03-11T19:58:58.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.176+0000 7f62bbe00640 10 client.15324 ms_handle_connect on v2:172.21.15.161:6838/565994162 2024-03-11T19:58:58.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.176+0000 7f62bbe00640 1 -- 172.21.15.97:0/2474009441 <== mds.2 v2:172.21.15.161:6838/565994162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560cf14966c0 con 0x560cf1843c40 2024-03-11T19:58:58.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.176+0000 7f62bbe00640 10 client.15324 handle_command_reply: tid=0 2024-03-11T19:58:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.177+0000 7f62e41fa740 1 client.15324 shutdown 2024-03-11T19:58:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.177+0000 7f62e41fa740 1 -- 172.21.15.97:0/2474009441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf18340e0 msgr2=0x560cf183f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.177+0000 7f62e41fa740 1 --2- 172.21.15.97:0/2474009441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf18340e0 0x560cf183f980 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f62d4005fd0 tx=0x7f62d4005c50 comp rx=0 tx=0).stop 2024-03-11T19:58:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.177+0000 7f62e41fa740 1 -- 172.21.15.97:0/2474009441 shutdown_connections 2024-03-11T19:58:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.177+0000 7f62e41fa740 1 --2- 172.21.15.97:0/2474009441 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x560cf1843c40 0x560cf1848ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.177+0000 7f62e41fa740 1 --2- 172.21.15.97:0/2474009441 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf18340e0 0x560cf183f980 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.177+0000 7f62e41fa740 1 --2- 172.21.15.97:0/2474009441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182ef70 0x560cf1838940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.177+0000 7f62e41fa740 1 --2- 172.21.15.97:0/2474009441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf182e580 0x560cf1838400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.177+0000 7f62bbe00640 0 client.15324 ms_handle_reset on v2:172.21.15.161:6838/565994162 2024-03-11T19:58:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.177+0000 7f62e41fa740 1 -- 172.21.15.97:0/2474009441 >> 172.21.15.97:0/2474009441 conn(0x560cf1829af0 msgr2=0x560cf1828e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.177+0000 7f62e41fa740 1 -- 172.21.15.97:0/2474009441 shutdown_connections 2024-03-11T19:58:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.177+0000 7f62e41fa740 1 -- 172.21.15.97:0/2474009441 wait complete. 2024-03-11T19:58:58.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.177+0000 7f62e41fa740 20 client.15324 trim_cache size 0 max 16384 2024-03-11T19:58:58.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.194+0000 7f62e41fa740 1 Processor -- start 2024-03-11T19:58:58.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.194+0000 7f62e41fa740 1 -- start start 2024-03-11T19:58:58.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.194+0000 7f62e41fa740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182e580 0x560cf184cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.194+0000 7f62e41fa740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 0x560cf183f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62e41fa740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf18340e0 0x560cf184d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62e41fa740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf14966c0 con 0x560cf182ef70 2024-03-11T19:58:58.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62e41fa740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf17bac40 con 0x560cf18340e0 2024-03-11T19:58:58.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62e41fa740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf178f540 con 0x560cf182e580 2024-03-11T19:58:58.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62da000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 0x560cf183f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62da000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 0x560cf183f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49572/0 (socket says 172.21.15.97:49572) 2024-03-11T19:58:58.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62da000640 1 -- 172.21.15.97:0/1102825378 learned_addr learned my addr 172.21.15.97:0/1102825378 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62db400640 1 --2- 172.21.15.97:0/1102825378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182e580 0x560cf184cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62db400640 1 --2- 172.21.15.97:0/1102825378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf18340e0 0x560cf184d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62da000640 1 -- 172.21.15.97:0/1102825378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182e580 msgr2=0x560cf184cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62da000640 1 --2- 172.21.15.97:0/1102825378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182e580 0x560cf184cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62da000640 1 -- 172.21.15.97:0/1102825378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf18340e0 msgr2=0x560cf184d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62da000640 1 --2- 172.21.15.97:0/1102825378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf18340e0 0x560cf184d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62da000640 1 -- 172.21.15.97:0/1102825378 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cf17318b0 con 0x560cf182ef70 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.195+0000 7f62db400640 1 --2- 172.21.15.97:0/1102825378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf18340e0 0x560cf184d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.196+0000 7f62da000640 1 --2- 172.21.15.97:0/1102825378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 0x560cf183f8a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f62c802c250 tx=0x7f62c802c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b58eaa4e7ff8e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.196+0000 7f62c6a00640 1 -- 172.21.15.97:0/1102825378 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8045040 con 0x560cf182ef70 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.196+0000 7f62c6a00640 1 -- 172.21.15.97:0/1102825378 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62c8047040 con 0x560cf182ef70 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.196+0000 7f62c6a00640 1 -- 172.21.15.97:0/1102825378 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8005050 con 0x560cf182ef70 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.196+0000 7f62e41fa740 1 -- 172.21.15.97:0/1102825378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 msgr2=0x560cf183f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.196+0000 7f62e41fa740 1 --2- 172.21.15.97:0/1102825378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 0x560cf183f8a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f62c802c250 tx=0x7f62c802c7f0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.196+0000 7f62e41fa740 1 -- 172.21.15.97:0/1102825378 shutdown_connections 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.196+0000 7f62e41fa740 1 --2- 172.21.15.97:0/1102825378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf18340e0 0x560cf184d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.196+0000 7f62e41fa740 1 --2- 172.21.15.97:0/1102825378 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 0x560cf183f8a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.196+0000 7f62e41fa740 1 --2- 172.21.15.97:0/1102825378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182e580 0x560cf184cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.196+0000 7f62e41fa740 1 -- 172.21.15.97:0/1102825378 >> 172.21.15.97:0/1102825378 conn(0x560cf1829af0 msgr2=0x560cf1833720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.197+0000 7f62e41fa740 1 -- 172.21.15.97:0/1102825378 shutdown_connections 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.197+0000 7f62e41fa740 1 -- 172.21.15.97:0/1102825378 wait complete. 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.197+0000 7f62e41fa740 1 -- start start 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.197+0000 7f62e41fa740 1 Processor -- start 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.197+0000 7f62e41fa740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182e580 0x560cf183f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.197+0000 7f62e41fa740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 0x560cf183c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.197+0000 7f62e41fa740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf18340e0 0x560cf183c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.197+0000 7f62e41fa740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf183cec0 con 0x560cf182ef70 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.197+0000 7f62e41fa740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf189a9a0 con 0x560cf18340e0 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.197+0000 7f62e41fa740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf189ab30 con 0x560cf182e580 2024-03-11T19:58:58.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62e41fa740 10 client.0 _ll_register_callbacks cb 0x7f62c8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:58.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62da000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 0x560cf183c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62da000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 0x560cf183c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49586/0 (socket says 172.21.15.97:49586) 2024-03-11T19:58:58.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62da000640 1 -- 172.21.15.97:0/3559279963 learned_addr learned my addr 172.21.15.97:0/3559279963 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62db400640 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182e580 0x560cf183f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62db400640 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf18340e0 0x560cf183c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62da000640 1 -- 172.21.15.97:0/3559279963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182e580 msgr2=0x560cf183f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62da000640 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182e580 0x560cf183f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62da000640 1 -- 172.21.15.97:0/3559279963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf18340e0 msgr2=0x560cf183c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62da000640 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf18340e0 0x560cf183c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62da000640 1 -- 172.21.15.97:0/3559279963 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62dc112680 con 0x560cf182ef70 2024-03-11T19:58:58.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62db400640 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182e580 0x560cf183f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:58.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62da000640 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 0x560cf183c2a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f62c802c220 tx=0x7f62c8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60c13fbadba30b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.198+0000 7f62bbe00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.199+0000 7f62bbe00640 1 -- 172.21.15.97:0/3559279963 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8045040 con 0x560cf182ef70 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.199+0000 7f62e41fa740 1 -- 172.21.15.97:0/3559279963 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560cf1845890 con 0x560cf182ef70 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.199+0000 7f62bbe00640 1 -- 172.21.15.97:0/3559279963 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f62c8004a40 con 0x560cf182ef70 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.199+0000 7f62bbe00640 1 -- 172.21.15.97:0/3559279963 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62c8004450 con 0x560cf182ef70 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.199+0000 7f62e41fa740 10 client.15330 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.199+0000 7f62e41fa740 1 -- 172.21.15.97:0/3559279963 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560cf1757510 con 0x560cf182ef70 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.199+0000 7f62bbe00640 1 -- 172.21.15.97:0/3559279963 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62c8067020 con 0x560cf182ef70 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.199+0000 7f62bbe00640 1 -- 172.21.15.97:0/3559279963 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f62c803c040 con 0x560cf182ef70 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.199+0000 7f62e41fa740 10 client.15330 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.200+0000 7f62e41fa740 10 client.15330 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.200+0000 7f62e41fa740 20 client.15330 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.200+0000 7f62e41fa740 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x560cf189fac0 0x560cf189bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.200+0000 7f62e41fa740 4 client.15330 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.200+0000 7f62e41fa740 1 -- 172.21.15.97:0/3559279963 --> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f62d4011920 con 0x560cf189fac0 2024-03-11T19:58:58.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.200+0000 7f62db400640 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x560cf189fac0 0x560cf189bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.200+0000 7f62db400640 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x560cf189fac0 0x560cf189bf40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2dfffe6f4b37260 server_cookie=7187f0ce618703e4 in_seq=0 out_seq=0 2024-03-11T19:58:58.203 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.200+0000 7f62bbe00640 10 client.15330 ms_handle_connect on v2:172.21.15.161:6838/565994162 2024-03-11T19:58:58.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62bbe00640 1 -- 172.21.15.97:0/3559279963 <== mds.2 v2:172.21.15.161:6838/565994162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f62d4011920 con 0x560cf189fac0 2024-03-11T19:58:58.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62bbe00640 10 client.15330 handle_command_reply: tid=0 2024-03-11T19:58:58.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62e41fa740 1 client.15330 shutdown 2024-03-11T19:58:58.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62e41fa740 1 -- 172.21.15.97:0/3559279963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 msgr2=0x560cf183c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62e41fa740 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 0x560cf183c2a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f62c802c220 tx=0x7f62c8005fd0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62e41fa740 1 -- 172.21.15.97:0/3559279963 shutdown_connections 2024-03-11T19:58:58.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62e41fa740 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.161:6838/565994162,v1:172.21.15.161:6839/565994162] conn(0x560cf189fac0 0x560cf189bf40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.204 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62e41fa740 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560cf18340e0 0x560cf183c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62e41fa740 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560cf182ef70 0x560cf183c2a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62e41fa740 1 --2- 172.21.15.97:0/3559279963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560cf182e580 0x560cf183f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62bbe00640 0 client.15330 ms_handle_reset on v2:172.21.15.161:6838/565994162 2024-03-11T19:58:58.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62e41fa740 1 -- 172.21.15.97:0/3559279963 >> 172.21.15.97:0/3559279963 conn(0x560cf1829af0 msgr2=0x560cf184cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62e41fa740 1 -- 172.21.15.97:0/3559279963 shutdown_connections 2024-03-11T19:58:58.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62e41fa740 1 -- 172.21.15.97:0/3559279963 wait complete. 2024-03-11T19:58:58.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.201+0000 7f62e41fa740 20 client.15330 trim_cache size 0 max 16384 2024-03-11T19:58:58.205 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T19:58:58.205 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:58:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 2, 2024-03-11T19:58:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24547, 2024-03-11T19:58:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-11T19:58:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-11T19:58:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-11T19:58:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 161.556067703, 2024-03-11T19:58:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-03-11T19:58:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 93, 2024-03-11T19:58:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 89, 2024-03-11T19:58:58.206 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 210.32439345700001 2024-03-11T19:58:58.207 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T19:58:58.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.202+0000 7f62e1600640 1 -- 172.21.15.97:0/3939561818 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f62c00a1a30 msgr2=0x7f62c00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.202+0000 7f62e1600640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f62c00a1a30 0x7f62c00a3ef0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f62cc096ab0 tx=0x7f62cc0d3530 comp rx=0 tx=0).stop 2024-03-11T19:58:58.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.202+0000 7f62e1600640 1 -- 172.21.15.97:0/3939561818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62dc108180 msgr2=0x7f62dc10ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.202+0000 7f62e1600640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62dc108180 0x7f62dc10ae10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00ea20 tx=0x7f62cc00eee0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.202+0000 7f62e1600640 1 -- 172.21.15.97:0/3939561818 shutdown_connections 2024-03-11T19:58:58.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.202+0000 7f62e1600640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62dc10fa00 0x7f62dc109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.202+0000 7f62e1600640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f62c00a1a30 0x7f62c00a3ef0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.202+0000 7f62e1600640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f62dc108ae0 0x7f62dc109450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.202+0000 7f62e1600640 1 --2- 172.21.15.97:0/3939561818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62dc108180 0x7f62dc10ae10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.202+0000 7f62e1600640 1 -- 172.21.15.97:0/3939561818 >> 172.21.15.97:0/3939561818 conn(0x7f62dc06e380 msgr2=0x7f62dc10b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.202+0000 7f62e1600640 1 -- 172.21.15.97:0/3939561818 shutdown_connections 2024-03-11T19:58:58.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.202+0000 7f62e1600640 1 -- 172.21.15.97:0/3939561818 wait complete. 2024-03-11T19:58:58.217 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-11T19:58:58.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.428+0000 7f5b6a000640 1 -- 172.21.15.97:0/13580083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b64104f70 msgr2=0x7f5b6410b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.428+0000 7f5b6a000640 1 --2- 172.21.15.97:0/13580083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b64104f70 0x7f5b6410b800 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5b54009860 tx=0x7f5b5402bb70 comp rx=0 tx=0).stop 2024-03-11T19:58:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.429+0000 7f5b6a000640 1 -- 172.21.15.97:0/13580083 shutdown_connections 2024-03-11T19:58:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.429+0000 7f5b6a000640 1 --2- 172.21.15.97:0/13580083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b64104f70 0x7f5b6410b800 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.429+0000 7f5b6a000640 1 --2- 172.21.15.97:0/13580083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b641045b0 0x7f5b64104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.429+0000 7f5b6a000640 1 --2- 172.21.15.97:0/13580083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b641033c0 0x7f5b641037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.429+0000 7f5b6a000640 1 -- 172.21.15.97:0/13580083 >> 172.21.15.97:0/13580083 conn(0x7f5b640fe930 msgr2=0x7f5b64100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.429+0000 7f5b6a000640 1 -- 172.21.15.97:0/13580083 shutdown_connections 2024-03-11T19:58:58.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.429+0000 7f5b6a000640 1 -- 172.21.15.97:0/13580083 wait complete. 2024-03-11T19:58:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.430+0000 7f5b6a000640 1 Processor -- start 2024-03-11T19:58:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.430+0000 7f5b6a000640 1 -- start start 2024-03-11T19:58:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.430+0000 7f5b6a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b641033c0 0x7f5b641993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.430+0000 7f5b6a000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b641045b0 0x7f5b64199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b6a000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b64104f70 0x7f5b641a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b6a000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b641a0ee0 con 0x7f5b64104f70 2024-03-11T19:58:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b6a000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b641a1050 con 0x7f5b641033c0 2024-03-11T19:58:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b6a000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b641a1210 con 0x7f5b641045b0 2024-03-11T19:58:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b63400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b641033c0 0x7f5b641993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b63400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b641033c0 0x7f5b641993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:51232/0 (socket says 172.21.15.97:51232) 2024-03-11T19:58:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b63400640 1 -- 172.21.15.97:0/3177072464 learned_addr learned my addr 172.21.15.97:0/3177072464 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b63e00640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b64104f70 0x7f5b641a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b62a00640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b641045b0 0x7f5b64199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b63400640 1 -- 172.21.15.97:0/3177072464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b641045b0 msgr2=0x7f5b64199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b63400640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b641045b0 0x7f5b64199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b63400640 1 -- 172.21.15.97:0/3177072464 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b64104f70 msgr2=0x7f5b641a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b63400640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b64104f70 0x7f5b641a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.431+0000 7f5b63400640 1 -- 172.21.15.97:0/3177072464 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b54009490 con 0x7f5b641033c0 2024-03-11T19:58:58.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.432+0000 7f5b63400640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b641033c0 0x7f5b641993f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c00e9c0 tx=0x7f5b5c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.432+0000 7f5b5be00640 1 -- 172.21.15.97:0/3177072464 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c0096e0 con 0x7f5b641033c0 2024-03-11T19:58:58.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.432+0000 7f5b5be00640 1 -- 172.21.15.97:0/3177072464 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b5c00cd80 con 0x7f5b641033c0 2024-03-11T19:58:58.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.432+0000 7f5b5be00640 1 -- 172.21.15.97:0/3177072464 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c010430 con 0x7f5b641033c0 2024-03-11T19:58:58.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.432+0000 7f5b6a000640 1 -- 172.21.15.97:0/3177072464 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b641a14f0 con 0x7f5b641033c0 2024-03-11T19:58:58.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.432+0000 7f5b6a000640 1 -- 172.21.15.97:0/3177072464 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b641a1a00 con 0x7f5b641033c0 2024-03-11T19:58:58.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.435+0000 7f5b5be00640 1 -- 172.21.15.97:0/3177072464 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5b5c003680 con 0x7f5b641033c0 2024-03-11T19:58:58.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.435+0000 7f5b6cc7a740 1 Processor -- start 2024-03-11T19:58:58.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.435+0000 7f5b6cc7a740 1 -- start start 2024-03-11T19:58:58.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.436+0000 7f5b6cc7a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b482580 0x55aa5b4829a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.436+0000 7f5b6cc7a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b487a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.436+0000 7f5b6cc7a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b4880e0 0x55aa5b48a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.436+0000 7f5b6cc7a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa5b3f1b30 con 0x55aa5b482580 2024-03-11T19:58:58.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.436+0000 7f5b6cc7a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa5b3ef060 con 0x55aa5b482f70 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.436+0000 7f5b6cc7a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa5b3ea840 con 0x55aa5b4880e0 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.436+0000 7f5b63e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b482580 0x55aa5b4829a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.436+0000 7f5b62a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b4880e0 0x55aa5b48a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.436+0000 7f5b62a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b4880e0 0x55aa5b48a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:57996/0 (socket says 172.21.15.97:57996) 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.436+0000 7f5b62a00640 1 -- 172.21.15.97:0/3580741342 learned_addr learned my addr 172.21.15.97:0/3580741342 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.436+0000 7f5b63400640 1 --2- 172.21.15.97:0/3580741342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b487a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b62a00640 1 -- 172.21.15.97:0/3580741342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 msgr2=0x55aa5b487a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b62a00640 1 --2- 172.21.15.97:0/3580741342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b487a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b62a00640 1 -- 172.21.15.97:0/3580741342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b482580 msgr2=0x55aa5b4829a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b62a00640 1 --2- 172.21.15.97:0/3580741342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b482580 0x55aa5b4829a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b62a00640 1 -- 172.21.15.97:0/3580741342 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa5b3b24c0 con 0x55aa5b4880e0 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b63400640 1 --2- 172.21.15.97:0/3580741342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b487a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b63e00640 1 --2- 172.21.15.97:0/3580741342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b482580 0x55aa5b4829a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b62a00640 1 --2- 172.21.15.97:0/3580741342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b4880e0 0x55aa5b48a570 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00f430 tx=0x7f5b4c00f8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56a4121a00b1e8fc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b5be00640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5b480a1ab0 0x7f5b480a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b53e00640 1 -- 172.21.15.97:0/3580741342 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c014070 con 0x55aa5b4880e0 2024-03-11T19:58:58.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b53e00640 1 -- 172.21.15.97:0/3580741342 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b4c00cb80 con 0x55aa5b4880e0 2024-03-11T19:58:58.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b53e00640 1 -- 172.21.15.97:0/3580741342 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c018d40 con 0x55aa5b4880e0 2024-03-11T19:58:58.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b5be00640 1 -- 172.21.15.97:0/3177072464 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f5b5c014070 con 0x7f5b641033c0 2024-03-11T19:58:58.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.437+0000 7f5b63e00640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5b480a1ab0 0x7f5b480a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.438+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3580741342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b4880e0 msgr2=0x55aa5b48a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.438+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3580741342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b4880e0 0x55aa5b48a570 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c00f430 tx=0x7f5b4c00f8f0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.438+0000 7f5b63e00640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5b480a1ab0 0x7f5b480a3f70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5b5402c180 tx=0x7f5b540056d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.438+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3580741342 shutdown_connections 2024-03-11T19:58:58.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.438+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3580741342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b4880e0 0x55aa5b48a570 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.438+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3580741342 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b487a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.438+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3580741342 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b482580 0x55aa5b4829a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.438+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3580741342 >> 172.21.15.97:0/3580741342 conn(0x55aa5b47daf0 msgr2=0x55aa5b47def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.438+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3580741342 shutdown_connections 2024-03-11T19:58:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.438+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3580741342 wait complete. 2024-03-11T19:58:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.439+0000 7f5b6cc7a740 1 -- start start 2024-03-11T19:58:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.439+0000 7f5b6cc7a740 1 Processor -- start 2024-03-11T19:58:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.439+0000 7f5b6cc7a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482580 0x55aa5b48c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.439+0000 7f5b6cc7a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482f70 0x55aa5b48c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.439+0000 7f5b6cc7a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b4938f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.439+0000 7f5b6cc7a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa5b38fea0 con 0x55aa5b4880e0 2024-03-11T19:58:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.439+0000 7f5b6cc7a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa5b38fb40 con 0x55aa5b482580 2024-03-11T19:58:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.439+0000 7f5b6cc7a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa5b132f20 con 0x55aa5b482f70 2024-03-11T19:58:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.439+0000 7f5b6cc7a740 10 client.0 _ll_register_callbacks cb 0x55aa5b3a3950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.440+0000 7f5b63e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b4938f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.440+0000 7f5b63400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482580 0x55aa5b48c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.440+0000 7f5b63400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482580 0x55aa5b48c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:51246/0 (socket says 172.21.15.97:51246) 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.440+0000 7f5b63400640 1 -- 172.21.15.97:0/2329817540 learned_addr learned my addr 172.21.15.97:0/2329817540 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.440+0000 7f5b62a00640 1 --2- 172.21.15.97:0/2329817540 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482f70 0x55aa5b48c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.440+0000 7f5b63400640 1 -- 172.21.15.97:0/2329817540 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482f70 msgr2=0x55aa5b48c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.440+0000 7f5b63400640 1 --2- 172.21.15.97:0/2329817540 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482f70 0x55aa5b48c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.440+0000 7f5b63400640 1 -- 172.21.15.97:0/2329817540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 msgr2=0x55aa5b4938f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.440+0000 7f5b63400640 1 --2- 172.21.15.97:0/2329817540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b4938f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.440+0000 7f5b63400640 1 -- 172.21.15.97:0/2329817540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4c009cd0 con 0x55aa5b482580 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.440+0000 7f5b63400640 1 --2- 172.21.15.97:0/2329817540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482580 0x55aa5b48c370 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c097120 tx=0x7f5b5c096390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b65fa62a29b0900c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.440+0000 7f5b50c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.440+0000 7f5b50c00640 1 -- 172.21.15.97:0/2329817540 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c09a0b0 con 0x55aa5b482580 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.441+0000 7f5b50c00640 1 -- 172.21.15.97:0/2329817540 <== mon.2 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f5b5c09ae50 con 0x55aa5b482580 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.441+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/2329817540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aa5b40ec40 con 0x55aa5b482580 2024-03-11T19:58:58.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.441+0000 7f5b50c00640 1 -- 172.21.15.97:0/2329817540 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b5c09a750 con 0x55aa5b482580 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.441+0000 7f5b6cc7a740 10 client.25454 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.441+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/2329817540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55aa5b3e8ec0 con 0x55aa5b482580 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.441+0000 7f5b50c00640 1 -- 172.21.15.97:0/2329817540 <== mon.2 v2:172.21.15.114:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55aa5b40ec40 con 0x55aa5b482580 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.441+0000 7f5b50c00640 1 -- 172.21.15.97:0/2329817540 <== mon.2 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5b5c01f060 con 0x55aa5b482580 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.441+0000 7f5b6cc7a740 10 client.25454 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.441+0000 7f5b6cc7a740 10 client.25454 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.441+0000 7f5b6cc7a740 20 client.25454 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.441+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/2329817540 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x55aa5b4988d0 0x55aa5b49cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.441+0000 7f5b6cc7a740 4 client.25454 mds_command: new command op to 24583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.441+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/2329817540 --> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aa5b0ea6c0 con 0x55aa5b4988d0 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.442+0000 7f5b62a00640 1 --2- 172.21.15.97:0/2329817540 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x55aa5b4988d0 0x55aa5b49cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.442+0000 7f5b62a00640 1 --2- 172.21.15.97:0/2329817540 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x55aa5b4988d0 0x55aa5b49cd50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b73a8598694a70b8 server_cookie=124564cacaa35813 in_seq=0 out_seq=0 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.442+0000 7f5b50c00640 10 client.25454 ms_handle_connect on v2:172.21.15.97:6840/2531110572 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b50c00640 1 -- 172.21.15.97:0/2329817540 <== mds.1 v2:172.21.15.97:6840/2531110572 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55aa5b0ea6c0 con 0x55aa5b4988d0 2024-03-11T19:58:58.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b50c00640 10 client.25454 handle_command_reply: tid=0 2024-03-11T19:58:58.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b6cc7a740 1 client.25454 shutdown 2024-03-11T19:58:58.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/2329817540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482580 msgr2=0x55aa5b48c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/2329817540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482580 0x55aa5b48c370 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c097120 tx=0x7f5b5c096390 comp rx=0 tx=0).stop 2024-03-11T19:58:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/2329817540 shutdown_connections 2024-03-11T19:58:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/2329817540 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x55aa5b4988d0 0x55aa5b49cd50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/2329817540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b4938f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/2329817540 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482f70 0x55aa5b48c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/2329817540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482580 0x55aa5b48c370 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b50c00640 0 client.25454 ms_handle_reset on v2:172.21.15.97:6840/2531110572 2024-03-11T19:58:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/2329817540 >> 172.21.15.97:0/2329817540 conn(0x55aa5b47daf0 msgr2=0x55aa5b47ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/2329817540 shutdown_connections 2024-03-11T19:58:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.443+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/2329817540 wait complete. 2024-03-11T19:58:58.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.444+0000 7f5b6cc7a740 20 client.25454 trim_cache size 0 max 16384 2024-03-11T19:58:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.460+0000 7f5b6cc7a740 1 Processor -- start 2024-03-11T19:58:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.460+0000 7f5b6cc7a740 1 -- start start 2024-03-11T19:58:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.461+0000 7f5b6cc7a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482580 0x55aa5b47c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.461+0000 7f5b6cc7a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b4938b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.461+0000 7f5b6cc7a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b47cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.461+0000 7f5b6cc7a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa5b40ec40 con 0x55aa5b4880e0 2024-03-11T19:58:58.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.461+0000 7f5b6cc7a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa5b3e3540 con 0x55aa5b482f70 2024-03-11T19:58:58.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.461+0000 7f5b6cc7a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa5b3e74a0 con 0x55aa5b482580 2024-03-11T19:58:58.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.461+0000 7f5b63e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b47cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.461+0000 7f5b63e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b47cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49628/0 (socket says 172.21.15.97:49628) 2024-03-11T19:58:58.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.461+0000 7f5b63e00640 1 -- 172.21.15.97:0/3030714676 learned_addr learned my addr 172.21.15.97:0/3030714676 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.461+0000 7f5b62a00640 1 --2- 172.21.15.97:0/3030714676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b4938b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.461+0000 7f5b63400640 1 --2- 172.21.15.97:0/3030714676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482580 0x55aa5b47c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.462+0000 7f5b63e00640 1 -- 172.21.15.97:0/3030714676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482580 msgr2=0x55aa5b47c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.462+0000 7f5b63e00640 1 --2- 172.21.15.97:0/3030714676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482580 0x55aa5b47c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.462+0000 7f5b63e00640 1 -- 172.21.15.97:0/3030714676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 msgr2=0x55aa5b4938b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.462+0000 7f5b63e00640 1 --2- 172.21.15.97:0/3030714676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b4938b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.462+0000 7f5b63e00640 1 -- 172.21.15.97:0/3030714676 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa5b3858b0 con 0x55aa5b4880e0 2024-03-11T19:58:58.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.462+0000 7f5b62a00640 1 --2- 172.21.15.97:0/3030714676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b4938b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:58.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.462+0000 7f5b63e00640 1 --2- 172.21.15.97:0/3030714676 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b47cac0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5b54036000 tx=0x7f5b54040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398b0b0cc8d99715 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.462+0000 7f5b53400640 1 -- 172.21.15.97:0/3030714676 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5402c710 con 0x55aa5b4880e0 2024-03-11T19:58:58.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.462+0000 7f5b53400640 1 -- 172.21.15.97:0/3030714676 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b540044f0 con 0x55aa5b4880e0 2024-03-11T19:58:58.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.462+0000 7f5b53400640 1 -- 172.21.15.97:0/3030714676 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54003850 con 0x55aa5b4880e0 2024-03-11T19:58:58.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.463+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3030714676 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 msgr2=0x55aa5b47cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.463+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3030714676 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b47cac0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5b54036000 tx=0x7f5b54040000 comp rx=0 tx=0).stop 2024-03-11T19:58:58.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.463+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3030714676 shutdown_connections 2024-03-11T19:58:58.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.463+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3030714676 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b47cac0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.463+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3030714676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b4938b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.463+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3030714676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482580 0x55aa5b47c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.463+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3030714676 >> 172.21.15.97:0/3030714676 conn(0x55aa5b47daf0 msgr2=0x55aa5b4a1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.463+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3030714676 shutdown_connections 2024-03-11T19:58:58.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.463+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3030714676 wait complete. 2024-03-11T19:58:58.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.464+0000 7f5b6cc7a740 1 -- start start 2024-03-11T19:58:58.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.464+0000 7f5b6cc7a740 1 Processor -- start 2024-03-11T19:58:58.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.464+0000 7f5b6cc7a740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482580 0x55aa5b490460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.464+0000 7f5b6cc7a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b4909a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.464+0000 7f5b6cc7a740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b4eea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.464+0000 7f5b6cc7a740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa5b4939c0 con 0x55aa5b4880e0 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.464+0000 7f5b6cc7a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa5b493b30 con 0x55aa5b482f70 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.464+0000 7f5b6cc7a740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa5b4eef50 con 0x55aa5b482580 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.464+0000 7f5b6cc7a740 10 client.0 _ll_register_callbacks cb 0x7f5b5c003930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b63e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b4eea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b63e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b4eea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49642/0 (socket says 172.21.15.97:49642) 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b63e00640 1 -- 172.21.15.97:0/3568519167 learned_addr learned my addr 172.21.15.97:0/3568519167 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b63400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482580 0x55aa5b490460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b62a00640 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b4909a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b63e00640 1 -- 172.21.15.97:0/3568519167 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482580 msgr2=0x55aa5b490460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b63e00640 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482580 0x55aa5b490460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b63e00640 1 -- 172.21.15.97:0/3568519167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 msgr2=0x55aa5b4909a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b63e00640 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b4909a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b63e00640 1 -- 172.21.15.97:0/3568519167 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa5b3b24c0 con 0x55aa5b4880e0 2024-03-11T19:58:58.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b62a00640 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b4909a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b63400640 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482580 0x55aa5b490460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b63e00640 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b4eea10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5b5402c0b0 tx=0x7f5b54005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f80e372afbff12 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.465+0000 7f5b50c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b50c00640 1 -- 172.21.15.97:0/3568519167 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5405b070 con 0x55aa5b4880e0 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b50c00640 1 -- 172.21.15.97:0/3568519167 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f5b54032970 con 0x55aa5b4880e0 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3568519167 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b4c00cb80 con 0x55aa5b4880e0 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b50c00640 1 -- 172.21.15.97:0/3568519167 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b5402cad0 con 0x55aa5b4880e0 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b6cc7a740 10 client.15348 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3568519167 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f5b54030070 con 0x55aa5b4880e0 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b50c00640 1 -- 172.21.15.97:0/3568519167 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b4c00cb80 con 0x55aa5b4880e0 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b50c00640 1 -- 172.21.15.97:0/3568519167 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5b54056470 con 0x55aa5b4880e0 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b6cc7a740 10 client.15348 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b6cc7a740 10 client.15348 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b6cc7a740 20 client.15348 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x55aa5b4f3d10 0x55aa5b4f0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b6cc7a740 4 client.15348 mds_command: new command op to 24583 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.466+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3568519167 --> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55aa5b13b270 con 0x55aa5b4f3d10 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.467+0000 7f5b63400640 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x55aa5b4f3d10 0x55aa5b4f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.467+0000 7f5b63400640 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x55aa5b4f3d10 0x55aa5b4f0210 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6391483c9ff7855d server_cookie=c7d1edf42f7601fc in_seq=0 out_seq=0 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.467+0000 7f5b50c00640 10 client.15348 ms_handle_connect on v2:172.21.15.97:6840/2531110572 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.467+0000 7f5b50c00640 1 -- 172.21.15.97:0/3568519167 <== mds.1 v2:172.21.15.97:6840/2531110572 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55aa5b13b270 con 0x55aa5b4f3d10 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.467+0000 7f5b50c00640 10 client.15348 handle_command_reply: tid=0 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.467+0000 7f5b6cc7a740 1 client.15348 shutdown 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.468+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3568519167 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 msgr2=0x55aa5b4eea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.468+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b4eea10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5b5402c0b0 tx=0x7f5b54005fd0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.468+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3568519167 shutdown_connections 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.468+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.97:6840/2531110572,v1:172.21.15.97:6841/2531110572] conn(0x55aa5b4f3d10 0x55aa5b4f0210 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.468+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55aa5b4880e0 0x55aa5b4eea10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.468+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55aa5b482f70 0x55aa5b4909a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.468+0000 7f5b6cc7a740 1 --2- 172.21.15.97:0/3568519167 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55aa5b482580 0x55aa5b490460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.468+0000 7f5b50c00640 0 client.15348 ms_handle_reset on v2:172.21.15.97:6840/2531110572 2024-03-11T19:58:58.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.468+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3568519167 >> 172.21.15.97:0/3568519167 conn(0x55aa5b47daf0 msgr2=0x55aa5b480d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.468+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3568519167 shutdown_connections 2024-03-11T19:58:58.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.468+0000 7f5b6cc7a740 1 -- 172.21.15.97:0/3568519167 wait complete. 2024-03-11T19:58:58.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.468+0000 7f5b6cc7a740 20 client.15348 trim_cache size 0 max 16384 2024-03-11T19:58:58.470 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T19:58:58.470 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:58:58.470 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 1, 2024-03-11T19:58:58.470 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24583, 2024-03-11T19:58:58.470 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-11T19:58:58.471 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-11T19:58:58.471 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-11T19:58:58.471 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 162.83549119899999, 2024-03-11T19:58:58.471 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-03-11T19:58:58.471 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 93, 2024-03-11T19:58:58.471 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 89, 2024-03-11T19:58:58.471 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 202.38872255699999 2024-03-11T19:58:58.471 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T19:58:58.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.469+0000 7f5b6a000640 1 -- 172.21.15.97:0/3177072464 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5b480a1ab0 msgr2=0x7f5b480a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.469+0000 7f5b6a000640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5b480a1ab0 0x7f5b480a3f70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5b5402c180 tx=0x7f5b540056d0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.469+0000 7f5b6a000640 1 -- 172.21.15.97:0/3177072464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b641033c0 msgr2=0x7f5b641993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.469+0000 7f5b6a000640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b641033c0 0x7f5b641993f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c00e9c0 tx=0x7f5b5c00ee80 comp rx=0 tx=0).stop 2024-03-11T19:58:58.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.469+0000 7f5b6a000640 1 -- 172.21.15.97:0/3177072464 shutdown_connections 2024-03-11T19:58:58.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.469+0000 7f5b6a000640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f5b480a1ab0 0x7f5b480a3f70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.469+0000 7f5b6a000640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5b64104f70 0x7f5b641a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.469+0000 7f5b6a000640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b641045b0 0x7f5b64199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.469+0000 7f5b6a000640 1 --2- 172.21.15.97:0/3177072464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b641033c0 0x7f5b641993f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.469+0000 7f5b6a000640 1 -- 172.21.15.97:0/3177072464 >> 172.21.15.97:0/3177072464 conn(0x7f5b640fe930 msgr2=0x7f5b641078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.469+0000 7f5b6a000640 1 -- 172.21.15.97:0/3177072464 shutdown_connections 2024-03-11T19:58:58.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.469+0000 7f5b6a000640 1 -- 172.21.15.97:0/3177072464 wait complete. 2024-03-11T19:58:58.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-11T19:58:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.699+0000 7f8fcaa00640 1 -- 172.21.15.97:0/3921585236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc41033a0 msgr2=0x7f8fc41037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.699+0000 7f8fcaa00640 1 --2- 172.21.15.97:0/3921585236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc41033a0 0x7f8fc41037c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4009900 tx=0x7f8fb402bc30 comp rx=0 tx=0).stop 2024-03-11T19:58:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.699+0000 7f8fcaa00640 1 -- 172.21.15.97:0/3921585236 shutdown_connections 2024-03-11T19:58:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.699+0000 7f8fcaa00640 1 --2- 172.21.15.97:0/3921585236 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8fc4104f50 0x7f8fc410b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.699+0000 7f8fcaa00640 1 --2- 172.21.15.97:0/3921585236 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fc4104590 0x7f8fc4104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.699+0000 7f8fcaa00640 1 --2- 172.21.15.97:0/3921585236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc41033a0 0x7f8fc41037c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.699+0000 7f8fcaa00640 1 -- 172.21.15.97:0/3921585236 >> 172.21.15.97:0/3921585236 conn(0x7f8fc40fe930 msgr2=0x7f8fc4100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.699+0000 7f8fcaa00640 1 -- 172.21.15.97:0/3921585236 shutdown_connections 2024-03-11T19:58:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.699+0000 7f8fcaa00640 1 -- 172.21.15.97:0/3921585236 wait complete. 2024-03-11T19:58:58.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.700+0000 7f8fcaa00640 1 Processor -- start 2024-03-11T19:58:58.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.701+0000 7f8fcaa00640 1 -- start start 2024-03-11T19:58:58.698 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.701+0000 7f8fcaa00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fc41033a0 0x7f8fc4199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.701+0000 7f8fcaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc4104590 0x7f8fc41999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.701+0000 7f8fcaa00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8fc4104f50 0x7f8fc41a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.701+0000 7f8fcaa00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc41a0f70 con 0x7f8fc4104f50 2024-03-11T19:58:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.701+0000 7f8fcaa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc41a10e0 con 0x7f8fc4104590 2024-03-11T19:58:58.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.701+0000 7f8fcaa00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc41a12a0 con 0x7f8fc41033a0 2024-03-11T19:58:58.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.701+0000 7f8fc3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fc41033a0 0x7f8fc4199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.701+0000 7f8fc8a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8fc4104f50 0x7f8fc41a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.701+0000 7f8fc3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fc41033a0 0x7f8fc4199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:58040/0 (socket says 172.21.15.97:58040) 2024-03-11T19:58:58.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.701+0000 7f8fc3e00640 1 -- 172.21.15.97:0/921239536 learned_addr learned my addr 172.21.15.97:0/921239536 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.702+0000 7f8fc3400640 1 --2- 172.21.15.97:0/921239536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc4104590 0x7f8fc41999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.702+0000 7f8fc3e00640 1 -- 172.21.15.97:0/921239536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc4104590 msgr2=0x7f8fc41999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.702+0000 7f8fc3e00640 1 --2- 172.21.15.97:0/921239536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc4104590 0x7f8fc41999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.702+0000 7f8fc3e00640 1 -- 172.21.15.97:0/921239536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8fc4104f50 msgr2=0x7f8fc41a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.702+0000 7f8fc3e00640 1 --2- 172.21.15.97:0/921239536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8fc4104f50 0x7f8fc41a0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.702+0000 7f8fc3e00640 1 -- 172.21.15.97:0/921239536 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc009840 con 0x7f8fc41033a0 2024-03-11T19:58:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.702+0000 7f8fc3e00640 1 --2- 172.21.15.97:0/921239536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fc41033a0 0x7f8fc4199480 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4005ab0 tx=0x7f8fb4005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.703+0000 7f8fc0c00640 1 -- 172.21.15.97:0/921239536 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb40043d0 con 0x7f8fc41033a0 2024-03-11T19:58:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.703+0000 7f8fc0c00640 1 -- 172.21.15.97:0/921239536 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fb40052d0 con 0x7f8fc41033a0 2024-03-11T19:58:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.703+0000 7f8fcaa00640 1 -- 172.21.15.97:0/921239536 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb4009560 con 0x7f8fc41033a0 2024-03-11T19:58:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.703+0000 7f8fc0c00640 1 -- 172.21.15.97:0/921239536 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4003cd0 con 0x7f8fc41033a0 2024-03-11T19:58:58.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.703+0000 7f8fcaa00640 1 -- 172.21.15.97:0/921239536 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc41a18a0 con 0x7f8fc41033a0 2024-03-11T19:58:58.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.706+0000 7f8fcd61b740 1 Processor -- start 2024-03-11T19:58:58.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.706+0000 7f8fcd61b740 1 -- start start 2024-03-11T19:58:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.706+0000 7f8fcd61b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 0x5635519129a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fcd61b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563551912f70 0x563551917a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fcd61b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5635519180e0 0x56355191a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fcd61b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x563551881b30 con 0x563551912580 2024-03-11T19:58:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fcd61b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355187f060 con 0x563551912f70 2024-03-11T19:58:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fcd61b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355187a840 con 0x5635519180e0 2024-03-11T19:58:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fc0c00640 1 -- 172.21.15.97:0/921239536 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8fb4005440 con 0x7f8fc41033a0 2024-03-11T19:58:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fc8a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 0x5635519129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fc8a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 0x5635519129a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49676/0 (socket says 172.21.15.97:49676) 2024-03-11T19:58:58.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fc8a00640 1 -- 172.21.15.97:0/4217840793 learned_addr learned my addr 172.21.15.97:0/4217840793 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fc3400640 1 --2- 172.21.15.97:0/4217840793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5635519180e0 0x56355191a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fc3e00640 1 --2- 172.21.15.97:0/4217840793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563551912f70 0x563551917a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fc8a00640 1 -- 172.21.15.97:0/4217840793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5635519180e0 msgr2=0x56355191a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/4217840793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5635519180e0 0x56355191a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fc8a00640 1 -- 172.21.15.97:0/4217840793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563551912f70 msgr2=0x563551917a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/4217840793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563551912f70 0x563551917a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.707+0000 7f8fc8a00640 1 -- 172.21.15.97:0/4217840793 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635518424c0 con 0x563551912580 2024-03-11T19:58:58.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.708+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/4217840793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 0x5635519129a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0115b0 tx=0x7f8fbc011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=215effb542502ed9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.708+0000 7f8fb9200640 1 -- 172.21.15.97:0/4217840793 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc004d20 con 0x563551912580 2024-03-11T19:58:58.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.708+0000 7f8fb9200640 1 -- 172.21.15.97:0/4217840793 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fbc004e80 con 0x563551912580 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.708+0000 7f8fb9200640 1 -- 172.21.15.97:0/4217840793 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc00f8c0 con 0x563551912580 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.708+0000 7f8fc0c00640 1 --2- 172.21.15.97:0/921239536 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8fa80a1a30 0x7f8fa80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.708+0000 7f8fc0c00640 1 -- 172.21.15.97:0/921239536 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f8fb40ef970 con 0x7f8fc41033a0 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.708+0000 7f8fc3e00640 1 --2- 172.21.15.97:0/921239536 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8fa80a1a30 0x7f8fa80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.709+0000 7f8fcd61b740 1 -- 172.21.15.97:0/4217840793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 msgr2=0x5635519129a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.709+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/4217840793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 0x5635519129a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0115b0 tx=0x7f8fbc011a70 comp rx=0 tx=0).stop 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.709+0000 7f8fcd61b740 1 -- 172.21.15.97:0/4217840793 shutdown_connections 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.709+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/4217840793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5635519180e0 0x56355191a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.709+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/4217840793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563551912f70 0x563551917a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.709+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/4217840793 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 0x5635519129a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.709+0000 7f8fcd61b740 1 -- 172.21.15.97:0/4217840793 >> 172.21.15.97:0/4217840793 conn(0x56355190daf0 msgr2=0x56355190def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.709+0000 7f8fcd61b740 1 -- 172.21.15.97:0/4217840793 shutdown_connections 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.709+0000 7f8fc3e00640 1 --2- 172.21.15.97:0/921239536 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8fa80a1a30 0x7f8fa80a3ef0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4000c00 tx=0x7f8fb40037d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.709+0000 7f8fcd61b740 1 -- 172.21.15.97:0/4217840793 wait complete. 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.710+0000 7f8fcd61b740 1 -- start start 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.710+0000 7f8fcd61b740 1 Processor -- start 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.710+0000 7f8fcd61b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563551912580 0x56355191c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.710+0000 7f8fcd61b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912f70 0x56355191c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.710+0000 7f8fcd61b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5635519180e0 0x563551923950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.710+0000 7f8fcd61b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355181fea0 con 0x5635519180e0 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.710+0000 7f8fcd61b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355181fb40 con 0x563551912580 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.710+0000 7f8fcd61b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635515c2f20 con 0x563551912f70 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.710+0000 7f8fcd61b740 10 client.0 _ll_register_callbacks cb 0x563551833950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.711+0000 7f8fc8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563551912580 0x56355191c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.711+0000 7f8fc3400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912f70 0x56355191c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.711+0000 7f8fc3400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912f70 0x56355191c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:58058/0 (socket says 172.21.15.97:58058) 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.711+0000 7f8fc3400640 1 -- 172.21.15.97:0/2780681201 learned_addr learned my addr 172.21.15.97:0/2780681201 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.711+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/2780681201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5635519180e0 0x563551923950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.711+0000 7f8fc3400640 1 -- 172.21.15.97:0/2780681201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563551912580 msgr2=0x56355191c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.711+0000 7f8fc3400640 1 --2- 172.21.15.97:0/2780681201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563551912580 0x56355191c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.711+0000 7f8fc3400640 1 -- 172.21.15.97:0/2780681201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5635519180e0 msgr2=0x563551923950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.711+0000 7f8fc3400640 1 --2- 172.21.15.97:0/2780681201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5635519180e0 0x563551923950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.711+0000 7f8fc3400640 1 -- 172.21.15.97:0/2780681201 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fbc009e20 con 0x563551912f70 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.711+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/2780681201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5635519180e0 0x563551923950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.711+0000 7f8fc3400640 1 --2- 172.21.15.97:0/2780681201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912f70 0x56355191c910 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00126e0 tx=0x7f8fb0012ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db78441dfba27c48 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.711+0000 7f8fad600640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.712+0000 7f8fad600640 1 -- 172.21.15.97:0/2780681201 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb0017070 con 0x563551912f70 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.712+0000 7f8fcd61b740 1 -- 172.21.15.97:0/2780681201 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56355157a6c0 con 0x563551912f70 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.712+0000 7f8fad600640 1 -- 172.21.15.97:0/2780681201 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f8fb000f8c0 con 0x563551912f70 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.712+0000 7f8fad600640 1 -- 172.21.15.97:0/2780681201 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fb000fa80 con 0x563551912f70 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.712+0000 7f8fcd61b740 10 client.25468 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.712+0000 7f8fcd61b740 1 -- 172.21.15.97:0/2780681201 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563551878ec0 con 0x563551912f70 2024-03-11T19:58:58.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.712+0000 7f8fad600640 1 -- 172.21.15.97:0/2780681201 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56355157a6c0 con 0x563551912f70 2024-03-11T19:58:58.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.712+0000 7f8fad600640 1 -- 172.21.15.97:0/2780681201 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8fb001c420 con 0x563551912f70 2024-03-11T19:58:58.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.713+0000 7f8fcd61b740 10 client.25468 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:58.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.713+0000 7f8fcd61b740 10 client.25468 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T19:58:58.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.713+0000 7f8fcd61b740 20 client.25468 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:58.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.713+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/2780681201 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563551927cf0 0x56355192cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.713+0000 7f8fcd61b740 4 client.25468 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:58:58.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.713+0000 7f8fcd61b740 1 -- 172.21.15.97:0/2780681201 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635515cb270 con 0x563551927cf0 2024-03-11T19:58:58.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.713+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/2780681201 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563551927cf0 0x56355192cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.714+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/2780681201 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563551927cf0 0x56355192cda0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abb4a6ed93a68678 server_cookie=7cb2aef7fb3143d7 in_seq=0 out_seq=0 2024-03-11T19:58:58.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.714+0000 7f8fad600640 10 client.25468 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T19:58:58.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fad600640 1 -- 172.21.15.97:0/2780681201 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5635515cb270 con 0x563551927cf0 2024-03-11T19:58:58.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fad600640 10 client.25468 handle_command_reply: tid=0 2024-03-11T19:58:58.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fcd61b740 1 client.25468 shutdown 2024-03-11T19:58:58.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fcd61b740 1 -- 172.21.15.97:0/2780681201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912f70 msgr2=0x56355191c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/2780681201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912f70 0x56355191c910 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8fb00126e0 tx=0x7f8fb0012ba0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fcd61b740 1 -- 172.21.15.97:0/2780681201 shutdown_connections 2024-03-11T19:58:58.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/2780681201 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563551927cf0 0x56355192cda0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/2780681201 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5635519180e0 0x563551923950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/2780681201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912f70 0x56355191c910 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fad600640 0 client.25468 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T19:58:58.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/2780681201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563551912580 0x56355191c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fcd61b740 1 -- 172.21.15.97:0/2780681201 >> 172.21.15.97:0/2780681201 conn(0x56355190daf0 msgr2=0x56355190ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fcd61b740 1 -- 172.21.15.97:0/2780681201 shutdown_connections 2024-03-11T19:58:58.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fcd61b740 1 -- 172.21.15.97:0/2780681201 wait complete. 2024-03-11T19:58:58.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.715+0000 7f8fcd61b740 20 client.25468 trim_cache size 0 max 16384 2024-03-11T19:58:58.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.732+0000 7f8fcd61b740 1 Processor -- start 2024-03-11T19:58:58.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.732+0000 7f8fcd61b740 1 -- start start 2024-03-11T19:58:58.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.732+0000 7f8fcd61b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 0x56355190ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.732+0000 7f8fcd61b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912f70 0x563551923910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.732+0000 7f8fcd61b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635519180e0 0x56355190c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.732+0000 7f8fcd61b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355191cfe0 con 0x563551912580 2024-03-11T19:58:58.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.732+0000 7f8fcd61b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355191ac30 con 0x5635519180e0 2024-03-11T19:58:58.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.732+0000 7f8fcd61b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635515cb270 con 0x563551912f70 2024-03-11T19:58:58.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.733+0000 7f8fc8a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 0x56355190ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.733+0000 7f8fc8a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 0x56355190ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49694/0 (socket says 172.21.15.97:49694) 2024-03-11T19:58:58.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.733+0000 7f8fc8a00640 1 -- 172.21.15.97:0/3155076791 learned_addr learned my addr 172.21.15.97:0/3155076791 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.733+0000 7f8fc3400640 1 --2- 172.21.15.97:0/3155076791 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912f70 0x563551923910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.733+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/3155076791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635519180e0 0x56355190c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.733+0000 7f8fc8a00640 1 -- 172.21.15.97:0/3155076791 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912f70 msgr2=0x563551923910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.733+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/3155076791 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912f70 0x563551923910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.733+0000 7f8fc8a00640 1 -- 172.21.15.97:0/3155076791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635519180e0 msgr2=0x56355190c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.733+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/3155076791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635519180e0 0x56355190c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.733+0000 7f8fc8a00640 1 -- 172.21.15.97:0/3155076791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635518158b0 con 0x563551912580 2024-03-11T19:58:58.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.734+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/3155076791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 0x56355190ded0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc000f80 tx=0x7f8fbc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348dca233f6d51f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.734+0000 7f8fafe00640 1 -- 172.21.15.97:0/3155076791 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc02cea0 con 0x563551912580 2024-03-11T19:58:58.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.734+0000 7f8fafe00640 1 -- 172.21.15.97:0/3155076791 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fbc004600 con 0x563551912580 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.734+0000 7f8fafe00640 1 -- 172.21.15.97:0/3155076791 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc02c320 con 0x563551912580 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.734+0000 7f8fcd61b740 1 -- 172.21.15.97:0/3155076791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 msgr2=0x56355190ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.735+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/3155076791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 0x56355190ded0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc000f80 tx=0x7f8fbc009500 comp rx=0 tx=0).stop 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.735+0000 7f8fcd61b740 1 -- 172.21.15.97:0/3155076791 shutdown_connections 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.735+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/3155076791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635519180e0 0x56355190c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.735+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/3155076791 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912f70 0x563551923910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.735+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/3155076791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912580 0x56355190ded0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.735+0000 7f8fcd61b740 1 -- 172.21.15.97:0/3155076791 >> 172.21.15.97:0/3155076791 conn(0x56355190daf0 msgr2=0x563551931650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.735+0000 7f8fcd61b740 1 -- 172.21.15.97:0/3155076791 shutdown_connections 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.735+0000 7f8fcd61b740 1 -- 172.21.15.97:0/3155076791 wait complete. 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.735+0000 7f8fcd61b740 1 -- start start 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.735+0000 7f8fcd61b740 1 Processor -- start 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fcd61b740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912580 0x563551923770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fcd61b740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912f70 0x563551920280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fcd61b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635519180e0 0x5635519207c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fcd61b740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x563551920e20 con 0x563551912f70 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fcd61b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355197ea60 con 0x5635519180e0 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fcd61b740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x56355197eba0 con 0x563551912580 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fcd61b740 10 client.0 _ll_register_callbacks cb 0x7f8fb0009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fc3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912f70 0x563551920280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fc3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912f70 0x563551920280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49704/0 (socket says 172.21.15.97:49704) 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fc3400640 1 -- 172.21.15.97:0/3072007874 learned_addr learned my addr 172.21.15.97:0/3072007874 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912580 0x563551923770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635519180e0 0x5635519207c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fc3400640 1 -- 172.21.15.97:0/3072007874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912580 msgr2=0x563551923770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fc3400640 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912580 0x563551923770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fc3400640 1 -- 172.21.15.97:0/3072007874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635519180e0 msgr2=0x5635519207c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fc3400640 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635519180e0 0x5635519207c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fc3400640 1 -- 172.21.15.97:0/3072007874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635518424c0 con 0x563551912f70 2024-03-11T19:58:58.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.736+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912580 0x563551923770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:58:58.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.737+0000 7f8fc3400640 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912f70 0x563551920280 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0012420 tx=0x7f8fb0009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0a0528278ba0c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:58:58.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.737+0000 7f8fad600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:58:58.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.737+0000 7f8fad600640 1 -- 172.21.15.97:0/3072007874 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb0017040 con 0x563551912f70 2024-03-11T19:58:58.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.737+0000 7f8fad600640 1 -- 172.21.15.97:0/3072007874 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f8fb000f990 con 0x563551912f70 2024-03-11T19:58:58.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.737+0000 7f8fcd61b740 1 -- 172.21.15.97:0/3072007874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563551929950 con 0x563551912f70 2024-03-11T19:58:58.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.737+0000 7f8fad600640 1 -- 172.21.15.97:0/3072007874 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fb000fb70 con 0x563551912f70 2024-03-11T19:58:58.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.737+0000 7f8fcd61b740 10 client.15366 fetch_fsmap learned FSMap version 28 2024-03-11T19:58:58.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.737+0000 7f8fcd61b740 1 -- 172.21.15.97:0/3072007874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56355183b510 con 0x563551912f70 2024-03-11T19:58:58.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.738+0000 7f8fad600640 1 -- 172.21.15.97:0/3072007874 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563551929950 con 0x563551912f70 2024-03-11T19:58:58.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.738+0000 7f8fad600640 1 -- 172.21.15.97:0/3072007874 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8fb00047b0 con 0x563551912f70 2024-03-11T19:58:58.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.738+0000 7f8fcd61b740 10 client.15366 fetch_fsmap finished waiting for FSMap version 28 2024-03-11T19:58:58.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.738+0000 7f8fcd61b740 10 client.15366 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T19:58:58.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.738+0000 7f8fcd61b740 20 client.15366 populate_metadata read hostname 'smithi097' 2024-03-11T19:58:58.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.738+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563551983eb0 0x563551980350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:58:58.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.738+0000 7f8fcd61b740 4 client.15366 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T19:58:58.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.738+0000 7f8fcd61b740 1 -- 172.21.15.97:0/3072007874 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56355189ec40 con 0x563551983eb0 2024-03-11T19:58:58.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.738+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563551983eb0 0x563551980350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:58:58.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fc8a00640 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563551983eb0 0x563551980350 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7ce20357c136948 server_cookie=a45113677c604828 in_seq=0 out_seq=0 2024-03-11T19:58:58.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fad600640 10 client.15366 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T19:58:58.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fad600640 1 -- 172.21.15.97:0/3072007874 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x56355189ec40 con 0x563551983eb0 2024-03-11T19:58:58.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fad600640 10 client.15366 handle_command_reply: tid=0 2024-03-11T19:58:58.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fcd61b740 1 client.15366 shutdown 2024-03-11T19:58:58.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fcd61b740 1 -- 172.21.15.97:0/3072007874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912f70 msgr2=0x563551920280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912f70 0x563551920280 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0012420 tx=0x7f8fb0009fd0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fcd61b740 1 -- 172.21.15.97:0/3072007874 shutdown_connections 2024-03-11T19:58:58.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635519180e0 0x5635519207c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563551983eb0 0x563551980350 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563551912f70 0x563551920280 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fcd61b740 1 --2- 172.21.15.97:0/3072007874 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563551912580 0x563551923770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fad600640 0 client.15366 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T19:58:58.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.739+0000 7f8fcd61b740 1 -- 172.21.15.97:0/3072007874 >> 172.21.15.97:0/3072007874 conn(0x56355190daf0 msgr2=0x563551931650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.740+0000 7f8fcd61b740 1 -- 172.21.15.97:0/3072007874 shutdown_connections 2024-03-11T19:58:58.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.740+0000 7f8fcd61b740 1 -- 172.21.15.97:0/3072007874 wait complete. 2024-03-11T19:58:58.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.740+0000 7f8fcd61b740 20 client.15366 trim_cache size 0 max 16384 2024-03-11T19:58:58.742 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T19:58:58.742 INFO:teuthology.orchestra.run.smithi097.stdout: "cluster_fsid": "e81ca0d2-dfdf-11ee-95c8-87774f69a715", 2024-03-11T19:58:58.742 INFO:teuthology.orchestra.run.smithi097.stdout: "whoami": 0, 2024-03-11T19:58:58.742 INFO:teuthology.orchestra.run.smithi097.stdout: "id": 24599, 2024-03-11T19:58:58.742 INFO:teuthology.orchestra.run.smithi097.stdout: "want_state": "up:active", 2024-03-11T19:58:58.742 INFO:teuthology.orchestra.run.smithi097.stdout: "state": "up:active", 2024-03-11T19:58:58.742 INFO:teuthology.orchestra.run.smithi097.stdout: "fs_name": "cephfs", 2024-03-11T19:58:58.742 INFO:teuthology.orchestra.run.smithi097.stdout: "rank_uptime": 192.69999486699999, 2024-03-11T19:58:58.742 INFO:teuthology.orchestra.run.smithi097.stdout: "mdsmap_epoch": 28, 2024-03-11T19:58:58.742 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch": 93, 2024-03-11T19:58:58.742 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_epoch_barrier": 89, 2024-03-11T19:58:58.742 INFO:teuthology.orchestra.run.smithi097.stdout: "uptime": 199.71323926100001 2024-03-11T19:58:58.742 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T19:58:58.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.740+0000 7f8fcaa00640 1 -- 172.21.15.97:0/921239536 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8fa80a1a30 msgr2=0x7f8fa80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.740+0000 7f8fcaa00640 1 --2- 172.21.15.97:0/921239536 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8fa80a1a30 0x7f8fa80a3ef0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4000c00 tx=0x7f8fb40037d0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.740+0000 7f8fcaa00640 1 -- 172.21.15.97:0/921239536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fc41033a0 msgr2=0x7f8fc4199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:58:58.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.740+0000 7f8fcaa00640 1 --2- 172.21.15.97:0/921239536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fc41033a0 0x7f8fc4199480 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4005ab0 tx=0x7f8fb4005e50 comp rx=0 tx=0).stop 2024-03-11T19:58:58.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.740+0000 7f8fcaa00640 1 -- 172.21.15.97:0/921239536 shutdown_connections 2024-03-11T19:58:58.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.740+0000 7f8fcaa00640 1 --2- 172.21.15.97:0/921239536 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f8fa80a1a30 0x7f8fa80a3ef0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.740+0000 7f8fcaa00640 1 --2- 172.21.15.97:0/921239536 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8fc4104f50 0x7f8fc41a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.740+0000 7f8fcaa00640 1 --2- 172.21.15.97:0/921239536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8fc4104590 0x7f8fc41999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.740+0000 7f8fcaa00640 1 --2- 172.21.15.97:0/921239536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fc41033a0 0x7f8fc4199480 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:58:58.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.741+0000 7f8fcaa00640 1 -- 172.21.15.97:0/921239536 >> 172.21.15.97:0/921239536 conn(0x7f8fc40fe930 msgr2=0x7f8fc4107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:58:58.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.741+0000 7f8fcaa00640 1 -- 172.21.15.97:0/921239536 shutdown_connections 2024-03-11T19:58:58.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:58:58.741+0000 7f8fcaa00640 1 -- 172.21.15.97:0/921239536 wait complete. 2024-03-11T19:58:58.755 INFO:teuthology.run_tasks:Running task kclient... 2024-03-11T19:58:58.768 INFO:tasks.kclient:Mounting kernel clients... 2024-03-11T19:58:58.768 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-11T19:58:58.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:58:58.769 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-11T19:58:58.821 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-11T19:58:58.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:58:58.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T19:58:58.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:58:58.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link delete ceph-brx 2024-03-11T19:58:59.038 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-11T19:58:59.038 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-11T19:58:59.039 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-11T19:58:59.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph fs ls 2024-03-11T19:58:59.465 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T19:58:59.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:58:59 smithi097 ceph-mon[31898]: pgmap v420: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:58:59.989 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:58:59 smithi114 ceph-mon[36376]: pgmap v420: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:59:00.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:58:59 smithi161 ceph-mon[36332]: pgmap v420: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-11T19:59:01.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.732+0000 7f4a6d400700 1 -- 172.21.15.97:0/1026017101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a681028d0 msgr2=0x7f4a68106910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:01.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.732+0000 7f4a6d400700 1 --2- 172.21.15.97:0/1026017101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a681028d0 0x7f4a68106910 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4a54009990 tx=0x7f4a54009ca0 comp rx=0 tx=0).stop 2024-03-11T19:59:01.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.732+0000 7f4a6d400700 1 -- 172.21.15.97:0/1026017101 shutdown_connections 2024-03-11T19:59:01.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.732+0000 7f4a6d400700 1 --2- 172.21.15.97:0/1026017101 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68106ff0 0x7f4a6810ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:01.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.732+0000 7f4a6d400700 1 --2- 172.21.15.97:0/1026017101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a68101f20 0x7f4a68102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:01.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.732+0000 7f4a6d400700 1 --2- 172.21.15.97:0/1026017101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a681028d0 0x7f4a68106910 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:01.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.732+0000 7f4a6d400700 1 -- 172.21.15.97:0/1026017101 >> 172.21.15.97:0/1026017101 conn(0x7f4a68078dd0 msgr2=0x7f4a680791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:01.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.732+0000 7f4a6d400700 1 -- 172.21.15.97:0/1026017101 shutdown_connections 2024-03-11T19:59:01.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.732+0000 7f4a6d400700 1 -- 172.21.15.97:0/1026017101 wait complete. 2024-03-11T19:59:01.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.734+0000 7f4a6d400700 1 Processor -- start 2024-03-11T19:59:01.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.734+0000 7f4a6d400700 1 -- start start 2024-03-11T19:59:01.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.734+0000 7f4a6d400700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68101f20 0x7f4a6819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:01.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.734+0000 7f4a6d400700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a681028d0 0x7f4a6819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:01.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.734+0000 7f4a6d400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a68106ff0 0x7f4a6819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:01.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.734+0000 7f4a6d400700 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6819f390 con 0x7f4a68101f20 2024-03-11T19:59:01.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.734+0000 7f4a6d400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6819f500 con 0x7f4a68106ff0 2024-03-11T19:59:01.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.734+0000 7f4a6d400700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6819f6c0 con 0x7f4a681028d0 2024-03-11T19:59:01.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.735+0000 7f4a66800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68101f20 0x7f4a6819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:01.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.735+0000 7f4a66800700 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68101f20 0x7f4a6819a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:49728/0 (socket says 172.21.15.97:49728) 2024-03-11T19:59:01.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.735+0000 7f4a66800700 1 -- 172.21.15.97:0/3007963892 learned_addr learned my addr 172.21.15.97:0/3007963892 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:01.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.735+0000 7f4a67200700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a68106ff0 0x7f4a6819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:01.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.735+0000 7f4a65e00700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a681028d0 0x7f4a6819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:01.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.735+0000 7f4a66800700 1 -- 172.21.15.97:0/3007963892 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a681028d0 msgr2=0x7f4a6819a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:01.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.735+0000 7f4a66800700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a681028d0 0x7f4a6819a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:01.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.735+0000 7f4a66800700 1 -- 172.21.15.97:0/3007963892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a68106ff0 msgr2=0x7f4a6819ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:01.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.735+0000 7f4a66800700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a68106ff0 0x7f4a6819ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:01.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.735+0000 7f4a66800700 1 -- 172.21.15.97:0/3007963892 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a540095f0 con 0x7f4a68101f20 2024-03-11T19:59:01.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.735+0000 7f4a67200700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a68106ff0 0x7f4a6819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:59:01.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.735+0000 7f4a66800700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68101f20 0x7f4a6819a4b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4a5800e9a0 tx=0x7f4a5800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:01.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.736+0000 7f4a5f400700 1 -- 172.21.15.97:0/3007963892 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a58009790 con 0x7f4a68101f20 2024-03-11T19:59:01.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.736+0000 7f4a5f400700 1 -- 172.21.15.97:0/3007963892 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a58010430 con 0x7f4a68101f20 2024-03-11T19:59:01.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.736+0000 7f4a6d400700 1 -- 172.21.15.97:0/3007963892 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6819f9a0 con 0x7f4a68101f20 2024-03-11T19:59:01.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.736+0000 7f4a5f400700 1 -- 172.21.15.97:0/3007963892 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a58018680 con 0x7f4a68101f20 2024-03-11T19:59:01.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.736+0000 7f4a6d400700 1 -- 172.21.15.97:0/3007963892 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a681a7200 con 0x7f4a68101f20 2024-03-11T19:59:01.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.737+0000 7f4a6d400700 1 -- 172.21.15.97:0/3007963892 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a68068930 con 0x7f4a68101f20 2024-03-11T19:59:01.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.739+0000 7f4a5f400700 1 -- 172.21.15.97:0/3007963892 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4a58010910 con 0x7f4a68101f20 2024-03-11T19:59:01.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.740+0000 7f4a5f400700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4a4c0a1b40 0x7f4a4c0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:01.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.740+0000 7f4a5f400700 1 -- 172.21.15.97:0/3007963892 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f4a58014070 con 0x7f4a68101f20 2024-03-11T19:59:01.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.741+0000 7f4a65e00700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4a4c0a1b40 0x7f4a4c0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:01.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.741+0000 7f4a65e00700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4a4c0a1b40 0x7f4a4c0a4000 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4a6810efe0 tx=0x7f4a54005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:01.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.744+0000 7f4a5f400700 1 -- 172.21.15.97:0/3007963892 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4a58052d20 con 0x7f4a68101f20 2024-03-11T19:59:01.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:01 smithi097 ceph-mon[31898]: pgmap v421: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:59:01.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:01 smithi114 ceph-mon[36376]: pgmap v421: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:59:01.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.996+0000 7f4a6d400700 1 -- 172.21.15.97:0/3007963892 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4a68111d70 con 0x7f4a68101f20 2024-03-11T19:59:01.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.997+0000 7f4a5f400700 1 -- 172.21.15.97:0/3007963892 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+88 (secure 0 0 0) 0x7f4a580967a0 con 0x7f4a68101f20 2024-03-11T19:59:01.995 INFO:teuthology.orchestra.run.smithi097.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-11T19:59:01.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.999+0000 7f4a6d400700 1 -- 172.21.15.97:0/3007963892 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4a4c0a1b40 msgr2=0x7f4a4c0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:01.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.999+0000 7f4a6d400700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4a4c0a1b40 0x7f4a4c0a4000 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4a6810efe0 tx=0x7f4a54005fd0 comp rx=0 tx=0).stop 2024-03-11T19:59:01.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.999+0000 7f4a6d400700 1 -- 172.21.15.97:0/3007963892 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68101f20 msgr2=0x7f4a6819a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:01.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.999+0000 7f4a6d400700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68101f20 0x7f4a6819a4b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4a5800e9a0 tx=0x7f4a5800ecb0 comp rx=0 tx=0).stop 2024-03-11T19:59:01.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.999+0000 7f4a6d400700 1 -- 172.21.15.97:0/3007963892 shutdown_connections 2024-03-11T19:59:01.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.999+0000 7f4a6d400700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f4a4c0a1b40 0x7f4a4c0a4000 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:01.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.999+0000 7f4a6d400700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4a68101f20 0x7f4a6819a4b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:01.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.999+0000 7f4a6d400700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a68106ff0 0x7f4a6819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:01.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.999+0000 7f4a6d400700 1 --2- 172.21.15.97:0/3007963892 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a681028d0 0x7f4a6819a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:01.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:01.999+0000 7f4a6d400700 1 -- 172.21.15.97:0/3007963892 >> 172.21.15.97:0/3007963892 conn(0x7f4a68078dd0 msgr2=0x7f4a681081c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:01.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:02.000+0000 7f4a6d400700 1 -- 172.21.15.97:0/3007963892 shutdown_connections 2024-03-11T19:59:01.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:02.000+0000 7f4a6d400700 1 -- 172.21.15.97:0/3007963892 wait complete. 2024-03-11T19:59:02.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:01 smithi161 ceph-mon[36332]: pgmap v421: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T19:59:02.857 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-11T19:59:02.857 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-11T19:59:02.857 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi097.front.sepia.ceph.com 2024-03-11T19:59:02.857 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-11T19:59:02.857 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-11T19:59:02.857 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-11T19:59:02.857 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-11T19:59:02.857 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-11T19:59:02.857 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-11T19:59:02.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:02.857 DEBUG:teuthology.orchestra.run.smithi097:> ip addr 2024-03-11T19:59:02.913 INFO:teuthology.orchestra.run.smithi097.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-11T19:59:02.913 INFO:teuthology.orchestra.run.smithi097.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-11T19:59:02.913 INFO:teuthology.orchestra.run.smithi097.stdout: inet 127.0.0.1/8 scope host lo 2024-03-11T19:59:02.913 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-11T19:59:02.913 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 ::1/128 scope host 2024-03-11T19:59:02.913 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-11T19:59:02.913 INFO:teuthology.orchestra.run.smithi097.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T19:59:02.913 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0e brd ff:ff:ff:ff:ff:ff 2024-03-11T19:59:02.913 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f0 2024-03-11T19:59:02.913 INFO:teuthology.orchestra.run.smithi097.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T19:59:02.913 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:d8:a2:0f brd ff:ff:ff:ff:ff:ff 2024-03-11T19:59:02.913 INFO:teuthology.orchestra.run.smithi097.stdout: altname enp7s0f1 2024-03-11T19:59:02.914 INFO:teuthology.orchestra.run.smithi097.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T19:59:02.914 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:04 brd ff:ff:ff:ff:ff:ff 2024-03-11T19:59:02.914 INFO:teuthology.orchestra.run.smithi097.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-11T19:59:02.914 INFO:teuthology.orchestra.run.smithi097.stdout: link/ether 0c:c4:7a:8f:d8:05 brd ff:ff:ff:ff:ff:ff 2024-03-11T19:59:02.914 INFO:teuthology.orchestra.run.smithi097.stdout: inet 172.21.15.97/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-11T19:59:02.914 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft 41800sec preferred_lft 41800sec 2024-03-11T19:59:02.914 INFO:teuthology.orchestra.run.smithi097.stdout: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link 2024-03-11T19:59:02.914 INFO:teuthology.orchestra.run.smithi097.stdout: valid_lft forever preferred_lft forever 2024-03-11T19:59:02.915 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-11T19:59:02.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:02.915 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-11T19:59:02.915 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-11T19:59:02.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link add name ceph-brx type bridge 2024-03-11T19:59:02.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip addr flush dev ceph-brx 2024-03-11T19:59:02.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set ceph-brx up 2024-03-11T19:59:02.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-11T19:59:02.916 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-11T19:59:02.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:02 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3007963892' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T19:59:03.000 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:02 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3007963892' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T19:59:03.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:02 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3007963892' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T19:59:03.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:03.137 DEBUG:teuthology.orchestra.run.smithi097:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-11T19:59:03.215 INFO:teuthology.orchestra.run.smithi097.stdout:1 2024-03-11T19:59:03.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:03.219 DEBUG:teuthology.orchestra.run.smithi097:> route 2024-03-11T19:59:03.271 INFO:teuthology.orchestra.run.smithi097.stdout:Kernel IP routing table 2024-03-11T19:59:03.271 INFO:teuthology.orchestra.run.smithi097.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-11T19:59:03.271 INFO:teuthology.orchestra.run.smithi097.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-11T19:59:03.271 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-11T19:59:03.271 INFO:teuthology.orchestra.run.smithi097.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-11T19:59:03.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:03.271 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-11T19:59:03.271 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-11T19:59:03.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-11T19:59:03.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-11T19:59:03.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-11T19:59:03.272 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-11T19:59:03.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:03.447 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-11T19:59:03.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:03.496 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list-id 2024-03-11T19:59:03.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:03.540 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-11T19:59:03.540 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-11T19:59:03.540 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-11T19:59:03.540 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-11T19:59:03.540 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-11T19:59:03.721 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-11T19:59:03.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:03.722 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-11T19:59:03.722 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-11T19:59:03.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-11T19:59:03.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-11T19:59:03.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-11T19:59:03.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-11T19:59:03.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-11T19:59:03.722 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-11T19:59:03.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:03 smithi097 ceph-mon[31898]: pgmap v422: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:59:03.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:03.970 DEBUG:teuthology.orchestra.run.smithi097:> (cd / && exec bash -c ' 2024-03-11T19:59:03.970 DEBUG:teuthology.orchestra.run.smithi097:> set -e 2024-03-11T19:59:03.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set brx.0 up 2024-03-11T19:59:03.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link set dev brx.0 master ceph-brx 2024-03-11T19:59:03.970 DEBUG:teuthology.orchestra.run.smithi097:> ') 2024-03-11T19:59:03.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:03 smithi114 ceph-mon[36376]: pgmap v422: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:59:04.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:03 smithi161 ceph-mon[36332]: pgmap v422: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-11T19:59:04.134 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T19:59:04.134 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-11T19:59:04.184 INFO:teuthology.orchestra.run.smithi097.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-11T19:59:04.184 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T19:59:04.184 DEBUG:teuthology.orchestra.run.smithi097:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-11T19:59:04.231 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-11T19:59:04.231 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-11T19:59:04.232 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/326c5357-1c14-4d24-96f1-f194f5b2ba4f 2024-03-11T19:59:04.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:04.232 DEBUG:teuthology.orchestra.run.smithi097:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/326c5357-1c14-4d24-96f1-f194f5b2ba4f /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-03-11T19:59:04.414 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-03-11T19:59:04.414 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-03-11T19:59:04.414 INFO:teuthology.orchestra.run.smithi097.stdout:invalid new device string format 2024-03-11T19:59:05.056 INFO:teuthology.orchestra.run.smithi097.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-03-11T19:59:05.057 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-03-11T19:59:05.057 INFO:teuthology.orchestra.run.smithi097.stdout:invalid new device string format 2024-03-11T19:59:05.057 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: resolved to: "172.21.15.97:3300,172.21.15.114:3300,172.21.15.161:3300" 2024-03-11T19:59:05.057 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.97:3300,172.21.15.114:3300,172.21.15.161:3300:/volumes/qa/sv_1/326c5357-1c14-4d24-96f1-f194f5b2ba4f 2024-03-11T19:59:05.057 INFO:teuthology.orchestra.run.smithi097.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,key=0,fsid=e81ca0d2-dfdf-11ee-95c8-87774f69a715" will pass to kernel 2024-03-11T19:59:05.064 INFO:teuthology.orchestra.run.smithi097.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-11T19:59:05.064 INFO:teuthology.orchestra.run.smithi097.stdout: You just mounted a file system that supports labels which does not 2024-03-11T19:59:05.064 INFO:teuthology.orchestra.run.smithi097.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-11T19:59:05.064 INFO:teuthology.orchestra.run.smithi097.stdout: applications will generate AVC messages and not be allowed access to 2024-03-11T19:59:05.064 INFO:teuthology.orchestra.run.smithi097.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-11T19:59:05.069 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-11T19:59:05.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:05.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-11T19:59:05.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:05.160 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-11T19:59:05.211 INFO:teuthology.orchestra.run.smithi097.stdout:client15375 2024-03-11T19:59:05.212 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T19:59:05.212 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T19:59:05.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:05.265 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-11T19:59:05.320 INFO:teuthology.orchestra.run.smithi097.stdout:client15375 2024-03-11T19:59:05.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:05.321 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-11T19:59:05.374 INFO:teuthology.orchestra.run.smithi097.stdout:client15375 2024-03-11T19:59:05.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T19:59:05.375 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715.client15375/status') 2024-03-11T19:59:05.463 INFO:teuthology.orchestra.run.smithi097.stdout:instance: client.15375 (3)192.168.0.1:0/1694473285 2024-03-11T19:59:05.463 INFO:teuthology.orchestra.run.smithi097.stdout:blocklisted: false 2024-03-11T19:59:05.464 INFO:teuthology.orchestra.run.smithi097.stderr:0+1 records in 2024-03-11T19:59:05.464 INFO:teuthology.orchestra.run.smithi097.stderr:0+1 records out 2024-03-11T19:59:05.464 INFO:teuthology.orchestra.run.smithi097.stderr:70 bytes copied, 6.5588e-05 s, 1.1 MB/s 2024-03-11T19:59:05.468 INFO:teuthology.run_tasks:Running task exec... 2024-03-11T19:59:05.478 INFO:teuthology.task.exec:Executing custom commands... 2024-03-11T19:59:05.478 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-03-11T19:59:05.478 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-11T19:59:05.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.688+0000 7ff1c2400640 1 -- 172.21.15.97:0/3870623117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1bc1089d0 msgr2=0x7ff1bc108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:05.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.688+0000 7ff1c2400640 1 --2- 172.21.15.97:0/3870623117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1bc1089d0 0x7ff1bc108e50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009900 tx=0x7ff1a402bc30 comp rx=0 tx=0).stop 2024-03-11T19:59:05.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.688+0000 7ff1c2400640 1 -- 172.21.15.97:0/3870623117 shutdown_connections 2024-03-11T19:59:05.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.688+0000 7ff1c2400640 1 --2- 172.21.15.97:0/3870623117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1bc109390 0x7ff1bc10fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:05.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.688+0000 7ff1c2400640 1 --2- 172.21.15.97:0/3870623117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1bc1089d0 0x7ff1bc108e50 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:05.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.688+0000 7ff1c2400640 1 --2- 172.21.15.97:0/3870623117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1bc1077e0 0x7ff1bc107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:05.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.688+0000 7ff1c2400640 1 -- 172.21.15.97:0/3870623117 >> 172.21.15.97:0/3870623117 conn(0x7ff1bc076790 msgr2=0x7ff1bc078bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:05.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.688+0000 7ff1c2400640 1 -- 172.21.15.97:0/3870623117 shutdown_connections 2024-03-11T19:59:05.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.688+0000 7ff1c2400640 1 -- 172.21.15.97:0/3870623117 wait complete. 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.689+0000 7ff1c2400640 1 Processor -- start 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.689+0000 7ff1c2400640 1 -- start start 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.689+0000 7ff1c2400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1bc1077e0 0x7ff1bc19d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.689+0000 7ff1c2400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1bc1089d0 0x7ff1bc19dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.689+0000 7ff1c2400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1bc109390 0x7ff1bc1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.689+0000 7ff1c2400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc1a5280 con 0x7ff1bc1077e0 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.689+0000 7ff1c2400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc1a53f0 con 0x7ff1bc1089d0 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.689+0000 7ff1c2400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc1a55b0 con 0x7ff1bc109390 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.690+0000 7ff1bb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1bc1077e0 0x7ff1bc19d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.690+0000 7ff1bb400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1bc1077e0 0x7ff1bc19d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50698/0 (socket says 172.21.15.97:50698) 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.690+0000 7ff1bb400640 1 -- 172.21.15.97:0/701461870 learned_addr learned my addr 172.21.15.97:0/701461870 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.690+0000 7ff1b3e00640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1bc1089d0 0x7ff1bc19dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.690+0000 7ff1bbe00640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1bc109390 0x7ff1bc1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:05.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.690+0000 7ff1bb400640 1 -- 172.21.15.97:0/701461870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1bc109390 msgr2=0x7ff1bc1a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:05.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.690+0000 7ff1bb400640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1bc109390 0x7ff1bc1a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:05.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.690+0000 7ff1bb400640 1 -- 172.21.15.97:0/701461870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1bc1089d0 msgr2=0x7ff1bc19dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:05.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.690+0000 7ff1bb400640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1bc1089d0 0x7ff1bc19dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:05.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.690+0000 7ff1bb400640 1 -- 172.21.15.97:0/701461870 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b4009840 con 0x7ff1bc1077e0 2024-03-11T19:59:05.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.690+0000 7ff1bbe00640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1bc109390 0x7ff1bc1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:59:05.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.691+0000 7ff1bb400640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1bc1077e0 0x7ff1bc19d790 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00e9c0 tx=0x7ff1ac00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:05.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.694+0000 7ff1b8c00640 1 -- 172.21.15.97:0/701461870 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac009700 con 0x7ff1bc1077e0 2024-03-11T19:59:05.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.694+0000 7ff1b8c00640 1 -- 172.21.15.97:0/701461870 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1ac00ccc0 con 0x7ff1bc1077e0 2024-03-11T19:59:05.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.694+0000 7ff1b8c00640 1 -- 172.21.15.97:0/701461870 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac010430 con 0x7ff1bc1077e0 2024-03-11T19:59:05.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.694+0000 7ff1c2400640 1 -- 172.21.15.97:0/701461870 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a4009560 con 0x7ff1bc1077e0 2024-03-11T19:59:05.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.694+0000 7ff1c2400640 1 -- 172.21.15.97:0/701461870 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc1a5c30 con 0x7ff1bc1077e0 2024-03-11T19:59:05.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.695+0000 7ff1c2400640 1 -- 172.21.15.97:0/701461870 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc10eaa0 con 0x7ff1bc1077e0 2024-03-11T19:59:05.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.697+0000 7ff1b8c00640 1 -- 172.21.15.97:0/701461870 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff1ac003f60 con 0x7ff1bc1077e0 2024-03-11T19:59:05.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.699+0000 7ff1b8c00640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff1a00a1b50 0x7ff1a00a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:05.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.699+0000 7ff1b8c00640 1 -- 172.21.15.97:0/701461870 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7ff1ac014070 con 0x7ff1bc1077e0 2024-03-11T19:59:05.696 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.699+0000 7ff1b3e00640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff1a00a1b50 0x7ff1a00a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:05.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.700+0000 7ff1b3e00640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff1a00a1b50 0x7ff1a00a4010 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009230 tx=0x7ff1a4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:05.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.703+0000 7ff1b8c00640 1 -- 172.21.15.97:0/701461870 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff1ac0969b0 con 0x7ff1bc1077e0 2024-03-11T19:59:05.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:05 smithi097 ceph-mon[31898]: pgmap v423: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:59:05.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:05.946+0000 7ff1c2400640 1 -- 172.21.15.97:0/701461870 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) v1 -- 0x7ff1bc02ce40 con 0x7ff1bc1077e0 2024-03-11T19:59:05.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:05 smithi114 ceph-mon[36376]: pgmap v423: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:59:06.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:05 smithi161 ceph-mon[36332]: pgmap v423: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T19:59:06.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.704+0000 7ff1b8c00640 1 -- 172.21.15.97:0/701461870 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v29) v1 ==== 117+0+0 (secure 0 0 0) 0x7ff1ac09a070 con 0x7ff1bc1077e0 2024-03-11T19:59:06.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.709+0000 7ff1c2400640 1 -- 172.21.15.97:0/701461870 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff1a00a1b50 msgr2=0x7ff1a00a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:06.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.709+0000 7ff1c2400640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff1a00a1b50 0x7ff1a00a4010 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009230 tx=0x7ff1a4005fd0 comp rx=0 tx=0).stop 2024-03-11T19:59:06.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.709+0000 7ff1c2400640 1 -- 172.21.15.97:0/701461870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1bc1077e0 msgr2=0x7ff1bc19d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:06.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.709+0000 7ff1c2400640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1bc1077e0 0x7ff1bc19d790 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00e9c0 tx=0x7ff1ac00edc0 comp rx=0 tx=0).stop 2024-03-11T19:59:06.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.709+0000 7ff1c2400640 1 -- 172.21.15.97:0/701461870 shutdown_connections 2024-03-11T19:59:06.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.709+0000 7ff1c2400640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff1bc109390 0x7ff1bc1a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:06.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.709+0000 7ff1c2400640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1bc1089d0 0x7ff1bc19dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:06.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.709+0000 7ff1c2400640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7ff1a00a1b50 0x7ff1a00a4010 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:06.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.709+0000 7ff1c2400640 1 --2- 172.21.15.97:0/701461870 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff1bc1077e0 0x7ff1bc19d790 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:06.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.709+0000 7ff1c2400640 1 -- 172.21.15.97:0/701461870 >> 172.21.15.97:0/701461870 conn(0x7ff1bc076790 msgr2=0x7ff1bc10bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:06.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.709+0000 7ff1c2400640 1 -- 172.21.15.97:0/701461870 shutdown_connections 2024-03-11T19:59:06.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.709+0000 7ff1c2400640 1 -- 172.21.15.97:0/701461870 wait complete. 2024-03-11T19:59:06.729 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-11T19:59:06.740 INFO:teuthology.run_tasks:Running task exec... 2024-03-11T19:59:06.771 INFO:teuthology.task.exec:Executing custom commands... 2024-03-11T19:59:06.772 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-03-11T19:59:06.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-11T19:59:06.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.989+0000 7fda60c00640 1 -- 172.21.15.97:0/947258364 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda5c075b40 msgr2=0x7fda5c075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:06.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.989+0000 7fda60c00640 1 --2- 172.21.15.97:0/947258364 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda5c075b40 0x7fda5c075f60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009900 tx=0x7fda4c02bc30 comp rx=0 tx=0).stop 2024-03-11T19:59:06.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.989+0000 7fda60c00640 1 -- 172.21.15.97:0/947258364 shutdown_connections 2024-03-11T19:59:06.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.989+0000 7fda60c00640 1 --2- 172.21.15.97:0/947258364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c077720 0x7fda5c077fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:06.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.989+0000 7fda60c00640 1 --2- 172.21.15.97:0/947258364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda5c076d80 0x7fda5c0771e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:06.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.989+0000 7fda60c00640 1 --2- 172.21.15.97:0/947258364 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda5c075b40 0x7fda5c075f60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:06.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.989+0000 7fda60c00640 1 -- 172.21.15.97:0/947258364 >> 172.21.15.97:0/947258364 conn(0x7fda5c0fe970 msgr2=0x7fda5c100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:06.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.989+0000 7fda60c00640 1 -- 172.21.15.97:0/947258364 shutdown_connections 2024-03-11T19:59:06.987 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.989+0000 7fda60c00640 1 -- 172.21.15.97:0/947258364 wait complete. 2024-03-11T19:59:06.988 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:06 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/701461870' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-11T19:59:06.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.990+0000 7fda60c00640 1 Processor -- start 2024-03-11T19:59:06.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.990+0000 7fda60c00640 1 -- start start 2024-03-11T19:59:06.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda60c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda5c075b40 0x7fda5c19d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:06.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda60c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda5c076d80 0x7fda5c19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:06.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda60c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c077720 0x7fda5c1a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:06.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda60c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c1a5270 con 0x7fda5c075b40 2024-03-11T19:59:06.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda60c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c1a53e0 con 0x7fda5c077720 2024-03-11T19:59:06.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda60c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c1a55a0 con 0x7fda5c076d80 2024-03-11T19:59:06.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda59e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda5c075b40 0x7fda5c19d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:06.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda59e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda5c075b40 0x7fda5c19d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:50712/0 (socket says 172.21.15.97:50712) 2024-03-11T19:59:06.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda59e00640 1 -- 172.21.15.97:0/3862309752 learned_addr learned my addr 172.21.15.97:0/3862309752 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:06.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda59400640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda5c076d80 0x7fda5c19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:06.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda5a800640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c077720 0x7fda5c1a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:06.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda59e00640 1 -- 172.21.15.97:0/3862309752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda5c076d80 msgr2=0x7fda5c19dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:06.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda59e00640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda5c076d80 0x7fda5c19dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:06.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda59e00640 1 -- 172.21.15.97:0/3862309752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c077720 msgr2=0x7fda5c1a4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:06.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda59e00640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c077720 0x7fda5c1a4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:06.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.991+0000 7fda59e00640 1 -- 172.21.15.97:0/3862309752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c009560 con 0x7fda5c075b40 2024-03-11T19:59:06.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.992+0000 7fda59e00640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda5c075b40 0x7fda5c19d820 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fda4c02c6a0 tx=0x7fda4c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:06.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.992+0000 7fda52a00640 1 -- 172.21.15.97:0/3862309752 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c0044c0 con 0x7fda5c075b40 2024-03-11T19:59:06.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.992+0000 7fda52a00640 1 -- 172.21.15.97:0/3862309752 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda4c0052d0 con 0x7fda5c075b40 2024-03-11T19:59:06.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.992+0000 7fda52a00640 1 -- 172.21.15.97:0/3862309752 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c003e80 con 0x7fda5c075b40 2024-03-11T19:59:06.991 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.992+0000 7fda60c00640 1 -- 172.21.15.97:0/3862309752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5c1a5820 con 0x7fda5c075b40 2024-03-11T19:59:06.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.992+0000 7fda60c00640 1 -- 172.21.15.97:0/3862309752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5c1a5ce0 con 0x7fda5c075b40 2024-03-11T19:59:06.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.993+0000 7fda60c00640 1 -- 172.21.15.97:0/3862309752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5c10fd50 con 0x7fda5c075b40 2024-03-11T19:59:06.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.995+0000 7fda52a00640 1 -- 172.21.15.97:0/3862309752 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fda4c005440 con 0x7fda5c075b40 2024-03-11T19:59:06.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.997+0000 7fda52a00640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fda3c0a1ad0 0x7fda3c0a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:06.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.997+0000 7fda52a00640 1 -- 172.21.15.97:0/3862309752 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fda4c0f0150 con 0x7fda5c075b40 2024-03-11T19:59:06.994 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.997+0000 7fda59400640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fda3c0a1ad0 0x7fda3c0a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:06.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:06.998+0000 7fda59400640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fda3c0a1ad0 0x7fda3c0a3f90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fda5c19ec90 tx=0x7fda44009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:06.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.001+0000 7fda52a00640 1 -- 172.21.15.97:0/3862309752 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fda4c0b9060 con 0x7fda5c075b40 2024-03-11T19:59:07.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:06 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/701461870' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-11T19:59:07.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:06 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/701461870' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-11T19:59:07.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.250+0000 7fda60c00640 1 -- 172.21.15.97:0/3862309752 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fda5c113230 con 0x7fda5c075b40 2024-03-11T19:59:07.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.711+0000 7fda52a00640 1 -- 172.21.15.97:0/3862309752 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7fda4c043020 con 0x7fda5c075b40 2024-03-11T19:59:07.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.715+0000 7fda60c00640 1 -- 172.21.15.97:0/3862309752 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fda3c0a1ad0 msgr2=0x7fda3c0a3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:07.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.715+0000 7fda60c00640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fda3c0a1ad0 0x7fda3c0a3f90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fda5c19ec90 tx=0x7fda44009500 comp rx=0 tx=0).stop 2024-03-11T19:59:07.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.715+0000 7fda60c00640 1 -- 172.21.15.97:0/3862309752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda5c075b40 msgr2=0x7fda5c19d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:07.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.715+0000 7fda60c00640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda5c075b40 0x7fda5c19d820 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fda4c02c6a0 tx=0x7fda4c0040c0 comp rx=0 tx=0).stop 2024-03-11T19:59:07.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.716+0000 7fda60c00640 1 -- 172.21.15.97:0/3862309752 shutdown_connections 2024-03-11T19:59:07.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.716+0000 7fda60c00640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda5c077720 0x7fda5c1a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:07.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.716+0000 7fda60c00640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda5c076d80 0x7fda5c19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:07.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.716+0000 7fda60c00640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fda3c0a1ad0 0x7fda3c0a3f90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:07.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.716+0000 7fda60c00640 1 --2- 172.21.15.97:0/3862309752 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fda5c075b40 0x7fda5c19d820 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:07.714 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.716+0000 7fda60c00640 1 -- 172.21.15.97:0/3862309752 >> 172.21.15.97:0/3862309752 conn(0x7fda5c0fe970 msgr2=0x7fda5c100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:07.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.718+0000 7fda60c00640 1 -- 172.21.15.97:0/3862309752 shutdown_connections 2024-03-11T19:59:07.716 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.719+0000 7fda60c00640 1 -- 172.21.15.97:0/3862309752 wait complete. 2024-03-11T19:59:07.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-11T19:59:07.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.942+0000 7f7f33400640 1 -- 172.21.15.97:0/828915290 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f2c101290 msgr2=0x7f7f2c101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:07.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.942+0000 7f7f33400640 1 --2- 172.21.15.97:0/828915290 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f2c101290 0x7f7f2c101710 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7f14009900 tx=0x7f7f1402bc30 comp rx=0 tx=0).stop 2024-03-11T19:59:07.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.942+0000 7f7f33400640 1 -- 172.21.15.97:0/828915290 shutdown_connections 2024-03-11T19:59:07.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.942+0000 7f7f33400640 1 --2- 172.21.15.97:0/828915290 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f2c101c50 0x7f7f2c1084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:07.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.942+0000 7f7f33400640 1 --2- 172.21.15.97:0/828915290 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f2c101290 0x7f7f2c101710 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:07.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.942+0000 7f7f33400640 1 --2- 172.21.15.97:0/828915290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f2c1000a0 0x7f7f2c1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:07.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.942+0000 7f7f33400640 1 -- 172.21.15.97:0/828915290 >> 172.21.15.97:0/828915290 conn(0x7f7f2c0fb610 msgr2=0x7f7f2c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:07.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.942+0000 7f7f33400640 1 -- 172.21.15.97:0/828915290 shutdown_connections 2024-03-11T19:59:07.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.942+0000 7f7f33400640 1 -- 172.21.15.97:0/828915290 wait complete. 2024-03-11T19:59:07.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.943+0000 7f7f33400640 1 Processor -- start 2024-03-11T19:59:07.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.943+0000 7f7f33400640 1 -- start start 2024-03-11T19:59:07.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.943+0000 7f7f33400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f2c1000a0 0x7f7f2c1993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:07.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.943+0000 7f7f33400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f2c101290 0x7f7f2c199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:07.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.943+0000 7f7f33400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f2c101c50 0x7f7f2c1a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:07.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.943+0000 7f7f33400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c1a0e70 con 0x7f7f2c101290 2024-03-11T19:59:07.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.943+0000 7f7f33400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c1a0fe0 con 0x7f7f2c101c50 2024-03-11T19:59:07.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.943+0000 7f7f33400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c1a11a0 con 0x7f7f2c1000a0 2024-03-11T19:59:07.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.944+0000 7f7f31400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f2c101c50 0x7f7f2c1a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:07.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.944+0000 7f7f30a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f2c1000a0 0x7f7f2c1993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:07.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.944+0000 7f7f23e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f2c101290 0x7f7f2c199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:07.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.944+0000 7f7f31400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f2c101c50 0x7f7f2c1a0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:55676/0 (socket says 172.21.15.97:55676) 2024-03-11T19:59:07.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.944+0000 7f7f31400640 1 -- 172.21.15.97:0/3369916945 learned_addr learned my addr 172.21.15.97:0/3369916945 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:07.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.944+0000 7f7f31400640 1 -- 172.21.15.97:0/3369916945 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f2c1000a0 msgr2=0x7f7f2c1993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:07.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.944+0000 7f7f31400640 1 --2- 172.21.15.97:0/3369916945 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f2c1000a0 0x7f7f2c1993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:07.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.944+0000 7f7f31400640 1 -- 172.21.15.97:0/3369916945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f2c101290 msgr2=0x7f7f2c199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:07.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.944+0000 7f7f31400640 1 --2- 172.21.15.97:0/3369916945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f2c101290 0x7f7f2c199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:07.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.944+0000 7f7f31400640 1 -- 172.21.15.97:0/3369916945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f14009560 con 0x7f7f2c101c50 2024-03-11T19:59:07.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.944+0000 7f7f31400640 1 --2- 172.21.15.97:0/3369916945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f2c101c50 0x7f7f2c1a0930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400eb10 tx=0x7f7f2400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:07.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.945+0000 7f7f2a000640 1 -- 172.21.15.97:0/3369916945 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f24004b20 con 0x7f7f2c101c50 2024-03-11T19:59:07.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.945+0000 7f7f2a000640 1 -- 172.21.15.97:0/3369916945 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f24004c80 con 0x7f7f2c101c50 2024-03-11T19:59:07.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.945+0000 7f7f33400640 1 -- 172.21.15.97:0/3369916945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2c1a1480 con 0x7f7f2c101c50 2024-03-11T19:59:07.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.945+0000 7f7f2a000640 1 -- 172.21.15.97:0/3369916945 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f24010650 con 0x7f7f2c101c50 2024-03-11T19:59:07.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.945+0000 7f7f33400640 1 -- 172.21.15.97:0/3369916945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2c1a1910 con 0x7f7f2c101c50 2024-03-11T19:59:07.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.946+0000 7f7f33400640 1 -- 172.21.15.97:0/3369916945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2c10cbb0 con 0x7f7f2c101c50 2024-03-11T19:59:07.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.948+0000 7f7f2a000640 1 -- 172.21.15.97:0/3369916945 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 136296+0+0 (secure 0 0 0) 0x7f7f24005020 con 0x7f7f2c101c50 2024-03-11T19:59:07.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.949+0000 7f7f2a000640 1 --2- 172.21.15.97:0/3369916945 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7f100a1b20 0x7f7f100a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:07.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.949+0000 7f7f30a00640 1 -- 172.21.15.97:0/3369916945 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7f100a1b20 msgr2=0x7f7f100a3fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3389151509 2024-03-11T19:59:07.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.949+0000 7f7f2a000640 1 -- 172.21.15.97:0/3369916945 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f7f24014070 con 0x7f7f2c101c50 2024-03-11T19:59:07.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.949+0000 7f7f30a00640 1 --2- 172.21.15.97:0/3369916945 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7f100a1b20 0x7f7f100a3fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:59:07.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:07.957+0000 7f7f2a000640 1 -- 172.21.15.97:0/3369916945 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7f24096c30 con 0x7f7f2c101c50 2024-03-11T19:59:07.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:07 smithi114 ceph-mon[36376]: pgmap v424: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-11T19:59:07.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:07 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/701461870' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-11T19:59:07.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:07 smithi114 ceph-mon[36376]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-11T19:59:07.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:07 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3862309752' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-11T19:59:07.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:07 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: ignoring --setuser ceph since I am not root 2024-03-11T19:59:07.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:07 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: ignoring --setgroup ceph since I am not root 2024-03-11T19:59:07.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:07 smithi114 ceph-mgr[37767]: -- 172.21.15.114:0/3363744229 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fb11fe61a0 con 0x55fb11fbcc00 2024-03-11T19:59:08.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:07 smithi161 ceph-mon[36332]: pgmap v424: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-11T19:59:08.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:07 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/701461870' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-11T19:59:08.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:07 smithi161 ceph-mon[36332]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-11T19:59:08.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:07 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3862309752' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-11T19:59:08.033 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:07 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: ignoring --setuser ceph since I am not root 2024-03-11T19:59:08.034 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:07 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: ignoring --setgroup ceph since I am not root 2024-03-11T19:59:08.034 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:07 smithi161 ceph-mgr[38193]: -- 172.21.15.161:0/39164200 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5599a00781a0 con 0x5599a004ec00 2024-03-11T19:59:08.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.150+0000 7f7f30a00640 1 -- 172.21.15.97:0/3369916945 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7f100a1b20 msgr2=0x7f7f100a3fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3389151509 2024-03-11T19:59:08.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.150+0000 7f7f30a00640 1 --2- 172.21.15.97:0/3369916945 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7f100a1b20 0x7f7f100a3fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T19:59:08.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.155+0000 7f7f33400640 1 -- 172.21.15.97:0/3369916945 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f7f2c10cd40 con 0x7f7f2c101c50 2024-03-11T19:59:08.160 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.163+0000 7f7f2a000640 1 -- 172.21.15.97:0/3369916945 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7f2409a2f0 con 0x7f7f2c101c50 2024-03-11T19:59:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.165+0000 7f7f33400640 1 -- 172.21.15.97:0/3369916945 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7f100a1b20 msgr2=0x7f7f100a3fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:59:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.165+0000 7f7f33400640 1 --2- 172.21.15.97:0/3369916945 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7f100a1b20 0x7f7f100a3fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.165+0000 7f7f33400640 1 -- 172.21.15.97:0/3369916945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f2c101c50 msgr2=0x7f7f2c1a0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.165+0000 7f7f33400640 1 --2- 172.21.15.97:0/3369916945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f2c101c50 0x7f7f2c1a0930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400eb10 tx=0x7f7f2400c2d0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.165+0000 7f7f33400640 1 -- 172.21.15.97:0/3369916945 shutdown_connections 2024-03-11T19:59:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.165+0000 7f7f33400640 1 --2- 172.21.15.97:0/3369916945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f2c101c50 0x7f7f2c1a0930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.165+0000 7f7f33400640 1 --2- 172.21.15.97:0/3369916945 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f7f100a1b20 0x7f7f100a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.165+0000 7f7f33400640 1 --2- 172.21.15.97:0/3369916945 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7f2c101290 0x7f7f2c199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.165+0000 7f7f33400640 1 --2- 172.21.15.97:0/3369916945 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f2c1000a0 0x7f7f2c1993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.165+0000 7f7f33400640 1 -- 172.21.15.97:0/3369916945 >> 172.21.15.97:0/3369916945 conn(0x7f7f2c0fb610 msgr2=0x7f7f2c108de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.166+0000 7f7f33400640 1 -- 172.21.15.97:0/3369916945 shutdown_connections 2024-03-11T19:59:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.166+0000 7f7f33400640 1 -- 172.21.15.97:0/3369916945 wait complete. 2024-03-11T19:59:08.170 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:07 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: ignoring --setuser ceph since I am not root 2024-03-11T19:59:08.170 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:07 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: ignoring --setgroup ceph since I am not root 2024-03-11T19:59:08.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:07 smithi097 ceph-mon[31898]: pgmap v424: 129 pgs: 129 active+clean; 644 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-11T19:59:08.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:07 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/701461870' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-11T19:59:08.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:07 smithi097 ceph-mon[31898]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-11T19:59:08.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:07 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3862309752' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-11T19:59:08.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-11T19:59:08.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.369+0000 7f9f6ac00640 1 -- 172.21.15.97:0/326286222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f641033e0 msgr2=0x7f9f64103800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:08.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.369+0000 7f9f6ac00640 1 --2- 172.21.15.97:0/326286222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f641033e0 0x7f9f64103800 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009900 tx=0x7f9f5402bc30 comp rx=0 tx=0).stop 2024-03-11T19:59:08.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.369+0000 7f9f6ac00640 1 -- 172.21.15.97:0/326286222 shutdown_connections 2024-03-11T19:59:08.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.369+0000 7f9f6ac00640 1 --2- 172.21.15.97:0/326286222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f64104f90 0x7f9f6410b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.369+0000 7f9f6ac00640 1 --2- 172.21.15.97:0/326286222 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f641045d0 0x7f9f64104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.369+0000 7f9f6ac00640 1 --2- 172.21.15.97:0/326286222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f641033e0 0x7f9f64103800 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.369+0000 7f9f6ac00640 1 -- 172.21.15.97:0/326286222 >> 172.21.15.97:0/326286222 conn(0x7f9f640fe970 msgr2=0x7f9f64100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:08.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.370+0000 7f9f6ac00640 1 -- 172.21.15.97:0/326286222 shutdown_connections 2024-03-11T19:59:08.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.370+0000 7f9f6ac00640 1 -- 172.21.15.97:0/326286222 wait complete. 2024-03-11T19:59:08.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.370+0000 7f9f6ac00640 1 Processor -- start 2024-03-11T19:59:08.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.371+0000 7f9f6ac00640 1 -- start start 2024-03-11T19:59:08.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.371+0000 7f9f6ac00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f641033e0 0x7f9f64199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:08.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.371+0000 7f9f6ac00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f641045d0 0x7f9f641998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:08.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.371+0000 7f9f6ac00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f64104f90 0x7f9f641a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:08.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.371+0000 7f9f6ac00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f641a0e80 con 0x7f9f641033e0 2024-03-11T19:59:08.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.371+0000 7f9f6ac00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f641a0ff0 con 0x7f9f64104f90 2024-03-11T19:59:08.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.371+0000 7f9f6ac00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f641a11b0 con 0x7f9f641045d0 2024-03-11T19:59:08.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.371+0000 7f9f63e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f641033e0 0x7f9f64199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:08.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.371+0000 7f9f63400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f641045d0 0x7f9f641998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:08.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.371+0000 7f9f63400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f641045d0 0x7f9f641998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:37004/0 (socket says 172.21.15.97:37004) 2024-03-11T19:59:08.373 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.371+0000 7f9f63400640 1 -- 172.21.15.97:0/1941304447 learned_addr learned my addr 172.21.15.97:0/1941304447 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:08.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.371+0000 7f9f68c00640 1 --2- 172.21.15.97:0/1941304447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f64104f90 0x7f9f641a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:08.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.372+0000 7f9f63400640 1 -- 172.21.15.97:0/1941304447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f64104f90 msgr2=0x7f9f641a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:08.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.372+0000 7f9f63400640 1 --2- 172.21.15.97:0/1941304447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f64104f90 0x7f9f641a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.372+0000 7f9f63400640 1 -- 172.21.15.97:0/1941304447 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f641033e0 msgr2=0x7f9f64199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:08.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.372+0000 7f9f63400640 1 --2- 172.21.15.97:0/1941304447 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f641033e0 0x7f9f64199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.372+0000 7f9f63400640 1 -- 172.21.15.97:0/1941304447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f54009560 con 0x7f9f641045d0 2024-03-11T19:59:08.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.372+0000 7f9f63400640 1 --2- 172.21.15.97:0/1941304447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f641045d0 0x7f9f641998d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000e9f0 tx=0x7f9f5000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:08.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.372+0000 7f9f60c00640 1 -- 172.21.15.97:0/1941304447 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50009700 con 0x7f9f641045d0 2024-03-11T19:59:08.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.372+0000 7f9f60c00640 1 -- 172.21.15.97:0/1941304447 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f5000cd80 con 0x7f9f641045d0 2024-03-11T19:59:08.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.372+0000 7f9f6ac00640 1 -- 172.21.15.97:0/1941304447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f641a1490 con 0x7f9f641045d0 2024-03-11T19:59:08.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.372+0000 7f9f60c00640 1 -- 172.21.15.97:0/1941304447 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50018920 con 0x7f9f641045d0 2024-03-11T19:59:08.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.372+0000 7f9f6ac00640 1 -- 172.21.15.97:0/1941304447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f641a19f0 con 0x7f9f641045d0 2024-03-11T19:59:08.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.373+0000 7f9f6ac00640 1 -- 172.21.15.97:0/1941304447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6410a630 con 0x7f9f641045d0 2024-03-11T19:59:08.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.377+0000 7f9f60c00640 1 -- 172.21.15.97:0/1941304447 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 19) v1 ==== 136296+0+0 (secure 0 0 0) 0x7f9f50018a80 con 0x7f9f641045d0 2024-03-11T19:59:08.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.378+0000 7f9f60c00640 1 --2- 172.21.15.97:0/1941304447 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9f480a1ba0 0x7f9f480a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:08.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.378+0000 7f9f63e00640 1 -- 172.21.15.97:0/1941304447 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9f480a1ba0 msgr2=0x7f9f480a4060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3389151509 2024-03-11T19:59:08.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.378+0000 7f9f60c00640 1 -- 172.21.15.97:0/1941304447 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f9f50014070 con 0x7f9f641045d0 2024-03-11T19:59:08.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.378+0000 7f9f63e00640 1 --2- 172.21.15.97:0/1941304447 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9f480a1ba0 0x7f9f480a4060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:59:08.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.382+0000 7f9f60c00640 1 -- 172.21.15.97:0/1941304447 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9f50052d20 con 0x7f9f641045d0 2024-03-11T19:59:08.565 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:08.566+0000 7f80268b3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T19:59:08.576 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.579+0000 7f9f63e00640 1 -- 172.21.15.97:0/1941304447 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9f480a1ba0 msgr2=0x7f9f480a4060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3389151509 2024-03-11T19:59:08.577 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.579+0000 7f9f63e00640 1 --2- 172.21.15.97:0/1941304447 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9f480a1ba0 0x7f9f480a4060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T19:59:08.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.582+0000 7f9f6ac00640 1 -- 172.21.15.97:0/1941304447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f9f6410ee40 con 0x7f9f641045d0 2024-03-11T19:59:08.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.588+0000 7f9f60c00640 1 -- 172.21.15.97:0/1941304447 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9f50096a10 con 0x7f9f641045d0 2024-03-11T19:59:08.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.590+0000 7f9f6ac00640 1 -- 172.21.15.97:0/1941304447 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9f480a1ba0 msgr2=0x7f9f480a4060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:59:08.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.590+0000 7f9f6ac00640 1 --2- 172.21.15.97:0/1941304447 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9f480a1ba0 0x7f9f480a4060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.590+0000 7f9f6ac00640 1 -- 172.21.15.97:0/1941304447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f641045d0 msgr2=0x7f9f641998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:08.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.590+0000 7f9f6ac00640 1 --2- 172.21.15.97:0/1941304447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f641045d0 0x7f9f641998d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000e9f0 tx=0x7f9f5000eeb0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.590+0000 7f9f6ac00640 1 -- 172.21.15.97:0/1941304447 shutdown_connections 2024-03-11T19:59:08.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.590+0000 7f9f6ac00640 1 --2- 172.21.15.97:0/1941304447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f64104f90 0x7f9f641a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.590+0000 7f9f6ac00640 1 --2- 172.21.15.97:0/1941304447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f641045d0 0x7f9f641998d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.590+0000 7f9f6ac00640 1 --2- 172.21.15.97:0/1941304447 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7f9f480a1ba0 0x7f9f480a4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.590+0000 7f9f6ac00640 1 --2- 172.21.15.97:0/1941304447 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9f641033e0 0x7f9f64199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.590+0000 7f9f6ac00640 1 -- 172.21.15.97:0/1941304447 >> 172.21.15.97:0/1941304447 conn(0x7f9f640fe970 msgr2=0x7f9f64107800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:08.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.591+0000 7f9f6ac00640 1 -- 172.21.15.97:0/1941304447 shutdown_connections 2024-03-11T19:59:08.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.591+0000 7f9f6ac00640 1 -- 172.21.15.97:0/1941304447 wait complete. 2024-03-11T19:59:08.606 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-11T19:59:08.729 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:08.286+0000 7fdbc4e91200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T19:59:08.729 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:08.380+0000 7fdbc4e91200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T19:59:08.729 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:08.286+0000 7fb41d290200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T19:59:08.729 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:08.380+0000 7fb41d290200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T19:59:08.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.815+0000 7fc023600640 1 -- 172.21.15.97:0/1873991300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc01c1023c0 msgr2=0x7fc01c102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:08.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.815+0000 7fc023600640 1 --2- 172.21.15.97:0/1873991300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc01c1023c0 0x7fc01c102840 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc008009860 tx=0x7fc00802bb70 comp rx=0 tx=0).stop 2024-03-11T19:59:08.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.815+0000 7fc023600640 1 -- 172.21.15.97:0/1873991300 shutdown_connections 2024-03-11T19:59:08.813 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.815+0000 7fc023600640 1 --2- 172.21.15.97:0/1873991300 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc01c102d80 0x7fc01c109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.815+0000 7fc023600640 1 --2- 172.21.15.97:0/1873991300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc01c1023c0 0x7fc01c102840 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.815+0000 7fc023600640 1 --2- 172.21.15.97:0/1873991300 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc01c1011d0 0x7fc01c1015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.816+0000 7fc023600640 1 -- 172.21.15.97:0/1873991300 >> 172.21.15.97:0/1873991300 conn(0x7fc01c0fc740 msgr2=0x7fc01c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:08.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.816+0000 7fc023600640 1 -- 172.21.15.97:0/1873991300 shutdown_connections 2024-03-11T19:59:08.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.816+0000 7fc023600640 1 -- 172.21.15.97:0/1873991300 wait complete. 2024-03-11T19:59:08.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.817+0000 7fc023600640 1 Processor -- start 2024-03-11T19:59:08.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.817+0000 7fc023600640 1 -- start start 2024-03-11T19:59:08.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.817+0000 7fc023600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc01c1011d0 0x7fc01c074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:08.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.817+0000 7fc023600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc01c1023c0 0x7fc01c073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:08.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.817+0000 7fc023600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc01c102d80 0x7fc01c073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:08.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.817+0000 7fc023600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c074210 con 0x7fc01c1023c0 2024-03-11T19:59:08.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.817+0000 7fc023600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c074380 con 0x7fc01c102d80 2024-03-11T19:59:08.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.817+0000 7fc023600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c1a1250 con 0x7fc01c1011d0 2024-03-11T19:59:08.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.817+0000 7fc021600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc01c102d80 0x7fc01c073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:08.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.817+0000 7fc021600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc01c102d80 0x7fc01c073ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:55708/0 (socket says 172.21.15.97:55708) 2024-03-11T19:59:08.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.817+0000 7fc021600640 1 -- 172.21.15.97:0/1108055453 learned_addr learned my addr 172.21.15.97:0/1108055453 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:08.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.817+0000 7fc020c00640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc01c1011d0 0x7fc01c074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.817+0000 7fc01be00640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc01c1023c0 0x7fc01c073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.818+0000 7fc021600640 1 -- 172.21.15.97:0/1108055453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc01c1011d0 msgr2=0x7fc01c074f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.818+0000 7fc021600640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc01c1011d0 0x7fc01c074f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.818+0000 7fc021600640 1 -- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc01c1023c0 msgr2=0x7fc01c073570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.818+0000 7fc021600640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc01c1023c0 0x7fc01c073570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.818+0000 7fc021600640 1 -- 172.21.15.97:0/1108055453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc010009560 con 0x7fc01c102d80 2024-03-11T19:59:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.818+0000 7fc021600640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc01c102d80 0x7fc01c073ab0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc01400eb40 tx=0x7fc01400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.818+0000 7fc019600640 1 -- 172.21.15.97:0/1108055453 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc014004b20 con 0x7fc01c102d80 2024-03-11T19:59:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.818+0000 7fc019600640 1 -- 172.21.15.97:0/1108055453 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc014004c80 con 0x7fc01c102d80 2024-03-11T19:59:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.818+0000 7fc023600640 1 -- 172.21.15.97:0/1108055453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc008009490 con 0x7fc01c102d80 2024-03-11T19:59:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.818+0000 7fc019600640 1 -- 172.21.15.97:0/1108055453 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc014010670 con 0x7fc01c102d80 2024-03-11T19:59:08.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.818+0000 7fc023600640 1 -- 172.21.15.97:0/1108055453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01c1a1800 con 0x7fc01c102d80 2024-03-11T19:59:08.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.819+0000 7fc023600640 1 -- 172.21.15.97:0/1108055453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01c1084f0 con 0x7fc01c102d80 2024-03-11T19:59:08.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.821+0000 7fc019600640 1 -- 172.21.15.97:0/1108055453 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 136296+0+0 (secure 0 0 0) 0x7fc014005020 con 0x7fc01c102d80 2024-03-11T19:59:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.822+0000 7fc019600640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 0x7fc0040a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.823+0000 7fc020c00640 1 -- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 msgr2=0x7fc0040a3fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3389151509 2024-03-11T19:59:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.823+0000 7fc019600640 1 -- 172.21.15.97:0/1108055453 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fc014014070 con 0x7fc01c102d80 2024-03-11T19:59:08.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.823+0000 7fc020c00640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 0x7fc0040a3fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T19:59:08.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:08.830+0000 7fc019600640 1 -- 172.21.15.97:0/1108055453 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc014096c30 con 0x7fc01c102d80 2024-03-11T19:59:08.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:08 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3862309752' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-11T19:59:08.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:08 smithi097 ceph-mon[31898]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-11T19:59:08.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:08 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' 2024-03-11T19:59:08.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:08 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' 2024-03-11T19:59:08.915 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:08.700+0000 7f80268b3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T19:59:08.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:08 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3862309752' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-11T19:59:08.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:08 smithi114 ceph-mon[36376]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-11T19:59:08.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:08 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' 2024-03-11T19:59:08.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:08 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' 2024-03-11T19:59:09.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:09.024+0000 7fc020c00640 1 -- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 msgr2=0x7fc0040a3fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3389151509 2024-03-11T19:59:09.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:09.024+0000 7fc020c00640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 0x7fc0040a3fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T19:59:09.023 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:08 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3862309752' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-11T19:59:09.023 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:08 smithi161 ceph-mon[36332]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-11T19:59:09.023 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:08 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' 2024-03-11T19:59:09.023 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:08 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' 2024-03-11T19:59:09.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:09.043+0000 7fc023600640 1 -- 172.21.15.97:0/1108055453 --> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc01c02ceb0 con 0x7fc0040a1b20 2024-03-11T19:59:09.283 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:09.023+0000 7fdbc4e91200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T19:59:09.283 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:09.117+0000 7fdbc4e91200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T19:59:09.283 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:09.258+0000 7fdbc4e91200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T19:59:09.349 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:09.022+0000 7fb41d290200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T19:59:09.378 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:09.116+0000 7fb41d290200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T19:59:09.378 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:09.257+0000 7fb41d290200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T19:59:09.378 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:09.350+0000 7fb41d290200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T19:59:09.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:09.425+0000 7fc020c00640 1 -- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 msgr2=0x7fc0040a3fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3389151509 2024-03-11T19:59:09.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:09.425+0000 7fc020c00640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 0x7fc0040a3fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T19:59:09.693 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:09.433+0000 7fb41d290200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T19:59:09.694 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:09.695+0000 7fb41d290200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T19:59:09.699 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:09.355+0000 7fdbc4e91200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T19:59:09.700 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:09.437+0000 7fdbc4e91200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T19:59:09.700 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:09.702+0000 7fdbc4e91200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T19:59:09.805 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:09.506+0000 7f80268b3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T19:59:09.805 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:09.628+0000 7f80268b3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T19:59:09.805 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:09.806+0000 7f80268b3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T19:59:09.986 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:09.805+0000 7fb41d290200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T19:59:09.986 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:09.892+0000 7fb41d290200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T19:59:09.986 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:09.988+0000 7fb41d290200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T19:59:09.992 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:09.813+0000 7fdbc4e91200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T19:59:09.993 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:09.900+0000 7fdbc4e91200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T19:59:09.993 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:09.995+0000 7fdbc4e91200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T19:59:10.165 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:09.931+0000 7f80268b3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T19:59:10.165 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:10 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:10.040+0000 7f80268b3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T19:59:10.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:10.226+0000 7fc020c00640 1 -- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 msgr2=0x7fc0040a3fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3389151509 2024-03-11T19:59:10.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:10.226+0000 7fc020c00640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 0x7fc0040a3fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T19:59:10.236 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:10 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:10.164+0000 7fb41d290200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T19:59:10.283 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:10 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:10.174+0000 7fdbc4e91200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T19:59:10.630 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:10 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:10.381+0000 7f80268b3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T19:59:10.630 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:10 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:10.520+0000 7f80268b3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T19:59:10.631 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:10 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:10.631+0000 7f80268b3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T19:59:10.915 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:10 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:10.754+0000 7f80268b3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T19:59:11.415 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:10 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:10.976+0000 7f80268b3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T19:59:11.487 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:11.122+0000 7fb41d290200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T19:59:11.487 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:11.253+0000 7fb41d290200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T19:59:11.533 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:11.141+0000 7fdbc4e91200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T19:59:11.533 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:11.273+0000 7fdbc4e91200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T19:59:11.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:11.828+0000 7fc020c00640 1 -- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 msgr2=0x7fc0040a3fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3389151509 2024-03-11T19:59:11.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:11.828+0000 7fc020c00640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 0x7fc0040a3fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T19:59:11.864 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:11.583+0000 7fb41d290200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T19:59:11.864 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:11.672+0000 7fb41d290200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T19:59:11.864 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:11.866+0000 7fb41d290200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:59:11.891 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:11.609+0000 7fdbc4e91200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T19:59:11.891 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:11.699+0000 7fdbc4e91200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T19:59:11.892 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:11.893+0000 7fdbc4e91200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:59:12.237 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:12 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:12.049+0000 7fb41d290200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T19:59:12.283 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:12 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:12.077+0000 7fdbc4e91200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T19:59:12.584 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:12 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:12.332+0000 7fb41d290200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:59:12.585 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:12 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:12.586+0000 7fb41d290200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T19:59:12.621 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:12 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:12.360+0000 7fdbc4e91200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:59:12.665 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:12 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:12.258+0000 7f80268b3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T19:59:12.665 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:12 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:12.428+0000 7f80268b3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T19:59:12.986 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:12 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:12.823+0000 7fb41d290200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T19:59:13.033 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:12 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:12.624+0000 7fdbc4e91200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T19:59:13.033 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:12 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:12.859+0000 7fdbc4e91200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T19:59:13.165 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:12 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:12.847+0000 7f80268b3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T19:59:13.165 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:12 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:12.968+0000 7f80268b3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T19:59:13.487 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:13 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:13.229+0000 7f80268b3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:59:13.870 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:13 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:13.489+0000 7f80268b3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T19:59:13.870 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:13 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:13.871+0000 7f80268b3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T19:59:13.986 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:13 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:13.613+0000 7fb41d290200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T19:59:14.033 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:13 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:13.657+0000 7fdbc4e91200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T19:59:14.487 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:14 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:14.185+0000 7fb41d290200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T19:59:14.487 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:14 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:14.282+0000 7fb41d290200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T19:59:14.494 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:14 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:14.189+0000 7f80268b3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T19:59:14.533 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:14 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:14.234+0000 7fdbc4e91200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T19:59:14.533 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:14 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:14.333+0000 7fdbc4e91200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T19:59:14.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:14.582+0000 7fc019600640 1 -- 172.21.15.97:0/1108055453 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 20) v1 ==== 136296+0+0 (secure 0 0 0) 0x7fc014097000 con 0x7fc01c102d80 2024-03-11T19:59:14.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:14 smithi097 ceph-mon[31898]: Standby manager daemon y restarted 2024-03-11T19:59:14.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:14 smithi097 ceph-mon[31898]: Standby manager daemon y started 2024-03-11T19:59:14.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:14 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.114:0/1830193510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T19:59:14.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:14 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.114:0/1830193510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:59:14.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:14 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.114:0/1830193510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T19:59:14.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:14 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.114:0/1830193510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:59:14.916 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:14 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:14.496+0000 7f80268b3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T19:59:14.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:14 smithi114 ceph-mon[36376]: Standby manager daemon y restarted 2024-03-11T19:59:14.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:14 smithi114 ceph-mon[36376]: Standby manager daemon y started 2024-03-11T19:59:14.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:14 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.114:0/1830193510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T19:59:14.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:14 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.114:0/1830193510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:59:14.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:14 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.114:0/1830193510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T19:59:14.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:14 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.114:0/1830193510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:59:14.987 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 19:59:14 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y[37763]: 2024-03-11T19:59:14.498+0000 7fb41d290200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T19:59:15.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:15.032+0000 7fc020c00640 1 -- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 msgr2=0x7fc0040a3fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.97:6800/3389151509 2024-03-11T19:59:15.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:15.032+0000 7fc020c00640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 0x7fc0040a3fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-11T19:59:15.032 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 19:59:14 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z[38189]: 2024-03-11T19:59:14.548+0000 7fdbc4e91200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T19:59:15.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:14 smithi161 ceph-mon[36332]: Standby manager daemon y restarted 2024-03-11T19:59:15.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:14 smithi161 ceph-mon[36332]: Standby manager daemon y started 2024-03-11T19:59:15.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:14 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.114:0/1830193510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T19:59:15.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:14 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.114:0/1830193510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:59:15.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:14 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.114:0/1830193510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T19:59:15.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:14 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.114:0/1830193510' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:59:15.915 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:15.524+0000 7f80268b3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T19:59:15.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:15 smithi097 ceph-mon[31898]: Standby manager daemon z restarted 2024-03-11T19:59:15.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:15 smithi097 ceph-mon[31898]: Standby manager daemon z started 2024-03-11T19:59:15.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:15 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.161:0/341584812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T19:59:15.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:15 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.161:0/341584812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:59:15.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:15 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.161:0/341584812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T19:59:15.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:15 smithi097 ceph-mon[31898]: from='mgr.? 172.21.15.161:0/341584812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:59:15.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:15 smithi097 ceph-mon[31898]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-03-11T19:59:15.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:15 smithi114 ceph-mon[36376]: Standby manager daemon z restarted 2024-03-11T19:59:15.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:15 smithi114 ceph-mon[36376]: Standby manager daemon z started 2024-03-11T19:59:15.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:15 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.161:0/341584812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T19:59:15.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:15 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.161:0/341584812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:59:15.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:15 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.161:0/341584812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T19:59:15.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:15 smithi114 ceph-mon[36376]: from='mgr.? 172.21.15.161:0/341584812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:59:15.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:15 smithi114 ceph-mon[36376]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-03-11T19:59:16.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:15 smithi161 ceph-mon[36332]: Standby manager daemon z restarted 2024-03-11T19:59:16.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:15 smithi161 ceph-mon[36332]: Standby manager daemon z started 2024-03-11T19:59:16.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:15 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.161:0/341584812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T19:59:16.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:15 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.161:0/341584812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T19:59:16.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:15 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.161:0/341584812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T19:59:16.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:15 smithi161 ceph-mon[36332]: from='mgr.? 172.21.15.161:0/341584812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T19:59:16.034 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:15 smithi161 ceph-mon[36332]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-03-11T19:59:16.637 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:16.223+0000 7f80268b3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T19:59:16.638 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:16.354+0000 7f80268b3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T19:59:16.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:16.660+0000 7fc019600640 1 -- 172.21.15.97:0/1108055453 <== mon.2 v2:172.21.15.114:3300/0 8 ==== mgrmap(e 21) v1 ==== 135964+0+0 (secure 0 0 0) 0x7fc0140762b0 con 0x7fc01c102d80 2024-03-11T19:59:16.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:16.662+0000 7fc019600640 1 -- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 msgr2=0x7fc0040a3fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T19:59:16.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:16.662+0000 7fc019600640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/3389151509,v1:172.21.15.97:6801/3389151509] conn(0x7fc0040a1b20 0x7fc0040a3fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:16.915 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:16.639+0000 7f80268b3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T19:59:16.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: Active manager daemon x restarted 2024-03-11T19:59:16.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: Activating manager daemon x 2024-03-11T19:59:16.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: osdmap e94: 12 total, 12 up, 12 in 2024-03-11T19:59:16.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: mgrmap e21: x(active, starting, since 0.0125094s), standbys: y, z 2024-03-11T19:59:16.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:59:16.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:59:16.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:59:16.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T19:59:16.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T19:59:16.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T19:59:16.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T19:59:16.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T19:59:16.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:59:16.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T19:59:16.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T19:59:16.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T19:59:16.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:59:16.917 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:59:16.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:59:16.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T19:59:16.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T19:59:16.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T19:59:16.918 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-mon[31898]: Manager daemon x is now available 2024-03-11T19:59:16.944 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: Active manager daemon x restarted 2024-03-11T19:59:16.944 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: Activating manager daemon x 2024-03-11T19:59:16.945 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: osdmap e94: 12 total, 12 up, 12 in 2024-03-11T19:59:16.945 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: mgrmap e21: x(active, starting, since 0.0125094s), standbys: y, z 2024-03-11T19:59:16.945 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:59:16.945 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:59:16.945 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:59:16.945 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T19:59:16.945 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T19:59:16.945 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T19:59:16.945 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T19:59:16.945 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T19:59:16.945 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:59:16.945 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T19:59:16.945 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T19:59:16.946 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T19:59:16.946 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:59:16.946 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T19:59:16.946 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T19:59:16.946 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T19:59:16.946 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T19:59:16.946 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T19:59:16.946 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:59:16.946 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:59:16.946 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:59:16.946 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:59:16.946 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:59:16.947 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:59:16.947 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:59:16.947 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:59:16.947 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:59:16.947 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:59:16.947 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:59:16.947 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:59:16.947 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T19:59:16.947 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T19:59:16.947 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T19:59:16.947 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:16 smithi114 ceph-mon[36376]: Manager daemon x is now available 2024-03-11T19:59:16.955 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: Active manager daemon x restarted 2024-03-11T19:59:16.955 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: Activating manager daemon x 2024-03-11T19:59:16.955 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: osdmap e94: 12 total, 12 up, 12 in 2024-03-11T19:59:16.955 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: mgrmap e21: x(active, starting, since 0.0125094s), standbys: y, z 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T19:59:16.956 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T19:59:16.957 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T19:59:16.958 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T19:59:16.958 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T19:59:16.958 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T19:59:16.958 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T19:59:16.958 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T19:59:16.958 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:16 smithi161 ceph-mon[36332]: Manager daemon x is now available 2024-03-11T19:59:17.338 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:16.971+0000 7f7f7b000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:59:17.338 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:16.971+0000 7f7f7b000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:59:17.338 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:16.971+0000 7f7f7b000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:59:17.338 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:16.971+0000 7f7f7b000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:59:17.338 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:16.971+0000 7f7f7b000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:59:17.338 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:16.971+0000 7f7f78800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:59:17.338 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:16.972+0000 7f7f78800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:59:17.339 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:16.972+0000 7f7f78800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:59:17.339 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:16.972+0000 7f7f78800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:59:17.339 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 19:59:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T19:59:16.972+0000 7f7f78800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T19:59:17.665 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.667+0000 7fc019600640 1 -- 172.21.15.97:0/1108055453 <== mon.2 v2:172.21.15.114:3300/0 9 ==== mgrmap(e 22) v1 ==== 136091+0+0 (secure 0 0 0) 0x7fc01404fa40 con 0x7fc01c102d80 2024-03-11T19:59:17.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.668+0000 7fc019600640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc004034e20 0x7fc004035200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:17.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.669+0000 7fc019600640 1 -- 172.21.15.97:0/1108055453 --> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc01c02ceb0 con 0x7fc004034e20 2024-03-11T19:59:17.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.670+0000 7fc020c00640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc004034e20 0x7fc004035200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:17.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.677+0000 7fc020c00640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc004034e20 0x7fc004035200 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc010009530 tx=0x7fc010009260 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:17.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.762+0000 7fc019600640 1 -- 172.21.15.97:0/1108055453 <== mgr.15396 v2:172.21.15.97:6800/1085760108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc01c02ceb0 con 0x7fc004034e20 2024-03-11T19:59:17.760 INFO:teuthology.orchestra.run.smithi097.stdout:Schedule set for path / 2024-03-11T19:59:17.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.765+0000 7fc023600640 1 -- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc004034e20 msgr2=0x7fc004035200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:17.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.765+0000 7fc023600640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc004034e20 0x7fc004035200 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc010009530 tx=0x7fc010009260 comp rx=0 tx=0).stop 2024-03-11T19:59:17.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.765+0000 7fc023600640 1 -- 172.21.15.97:0/1108055453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc01c102d80 msgr2=0x7fc01c073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:17.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.765+0000 7fc023600640 1 --2- 172.21.15.97:0/1108055453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc01c102d80 0x7fc01c073ab0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc01400eb40 tx=0x7fc01400c2d0 comp rx=0 tx=0).stop 2024-03-11T19:59:17.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.765+0000 7fc021600640 1 -- 172.21.15.97:0/1108055453 reap_dead start 2024-03-11T19:59:17.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.765+0000 7fc023600640 1 -- 172.21.15.97:0/1108055453 shutdown_connections 2024-03-11T19:59:17.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.765+0000 7fc023600640 1 -- 172.21.15.97:0/1108055453 >> 172.21.15.97:0/1108055453 conn(0x7fc01c0fc740 msgr2=0x7fc01c1056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:17.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.765+0000 7fc023600640 1 -- 172.21.15.97:0/1108055453 shutdown_connections 2024-03-11T19:59:17.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.765+0000 7fc023600640 1 -- 172.21.15.97:0/1108055453 wait complete. 2024-03-11T19:59:17.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-11T19:59:17.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.963+0000 7f798c800640 1 -- 172.21.15.97:0/1555375032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79841045d0 msgr2=0x7f7984104a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:17.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.963+0000 7f798c800640 1 --2- 172.21.15.97:0/1555375032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79841045d0 0x7f7984104a50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7974009900 tx=0x7f797402bc30 comp rx=0 tx=0).stop 2024-03-11T19:59:17.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.963+0000 7f798c800640 1 -- 172.21.15.97:0/1555375032 shutdown_connections 2024-03-11T19:59:17.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.963+0000 7f798c800640 1 --2- 172.21.15.97:0/1555375032 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7984104f90 0x7f798410b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:17.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.963+0000 7f798c800640 1 --2- 172.21.15.97:0/1555375032 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79841045d0 0x7f7984104a50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:17.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.963+0000 7f798c800640 1 --2- 172.21.15.97:0/1555375032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79841033e0 0x7f7984103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:17.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.963+0000 7f798c800640 1 -- 172.21.15.97:0/1555375032 >> 172.21.15.97:0/1555375032 conn(0x7f79840fe970 msgr2=0x7f7984100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:17.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.963+0000 7f798c800640 1 -- 172.21.15.97:0/1555375032 shutdown_connections 2024-03-11T19:59:17.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.963+0000 7f798c800640 1 -- 172.21.15.97:0/1555375032 wait complete. 2024-03-11T19:59:17.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.964+0000 7f798c800640 1 Processor -- start 2024-03-11T19:59:17.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.964+0000 7f798c800640 1 -- start start 2024-03-11T19:59:17.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f798c800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79841033e0 0x7f7984199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:17.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f798c800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79841045d0 0x7f79841998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:17.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f798c800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7984104f90 0x7f79841a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:17.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f798c800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79841a0e70 con 0x7f79841033e0 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f798c800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79841a0fe0 con 0x7f7984104f90 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f798c800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79841a11a0 con 0x7f79841045d0 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f7989400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79841045d0 0x7f79841998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f7989400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79841045d0 0x7f79841998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:55748/0 (socket says 172.21.15.97:55748) 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f7989400640 1 -- 172.21.15.97:0/3553737943 learned_addr learned my addr 172.21.15.97:0/3553737943 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f7989e00640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79841033e0 0x7f7984199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f798a800640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7984104f90 0x7f79841a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f7989400640 1 -- 172.21.15.97:0/3553737943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7984104f90 msgr2=0x7f79841a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f7989400640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7984104f90 0x7f79841a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f7989400640 1 -- 172.21.15.97:0/3553737943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79841033e0 msgr2=0x7f7984199380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f7989400640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79841033e0 0x7f7984199380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f7989400640 1 -- 172.21.15.97:0/3553737943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7974009560 con 0x7f79841045d0 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.965+0000 7f7989e00640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79841033e0 0x7f7984199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.966+0000 7f7989400640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79841045d0 0x7f79841998c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7974006010 tx=0x7f7974005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:17.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.966+0000 7f797ea00640 1 -- 172.21.15.97:0/3553737943 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79740043d0 con 0x7f79841045d0 2024-03-11T19:59:17.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.966+0000 7f797ea00640 1 -- 172.21.15.97:0/3553737943 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79740052d0 con 0x7f79841045d0 2024-03-11T19:59:17.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.966+0000 7f797ea00640 1 -- 172.21.15.97:0/3553737943 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974003cc0 con 0x7f79841045d0 2024-03-11T19:59:17.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.966+0000 7f798c800640 1 -- 172.21.15.97:0/3553737943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79841a1420 con 0x7f79841045d0 2024-03-11T19:59:17.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.966+0000 7f798c800640 1 -- 172.21.15.97:0/3553737943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79841a1980 con 0x7f79841045d0 2024-03-11T19:59:17.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.967+0000 7f798c800640 1 -- 172.21.15.97:0/3553737943 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798410a6a0 con 0x7f79841045d0 2024-03-11T19:59:17.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.970+0000 7f797ea00640 1 -- 172.21.15.97:0/3553737943 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 22) v1 ==== 136091+0+0 (secure 0 0 0) 0x7f7974005440 con 0x7f79841045d0 2024-03-11T19:59:17.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.971+0000 7f797ea00640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f796c0a1960 0x7f796c0a3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:17.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.972+0000 7f797ea00640 1 -- 172.21.15.97:0/3553737943 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f79740ef700 con 0x7f79841045d0 2024-03-11T19:59:17.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.972+0000 7f7989e00640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f796c0a1960 0x7f796c0a3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:17.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.973+0000 7f7989e00640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f796c0a1960 0x7f796c0a3e20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7984104430 tx=0x7f7978005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:17.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:17.976+0000 7f797ea00640 1 -- 172.21.15.97:0/3553737943 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f79740b8740 con 0x7f79841045d0 2024-03-11T19:59:17.986 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:17 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:59:17.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:17 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:59:17.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:17 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T19:59:17.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:17 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T19:59:17.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:17 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T19:59:17.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:17 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T19:59:17.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:17 smithi114 ceph-mon[36376]: mgrmap e22: x(active, since 1.02468s), standbys: y, z 2024-03-11T19:59:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:17 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:59:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:17 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:59:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:17 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T19:59:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:17 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T19:59:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:17 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T19:59:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:17 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T19:59:18.009 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:17 smithi161 ceph-mon[36332]: mgrmap e22: x(active, since 1.02468s), standbys: y, z 2024-03-11T19:59:18.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:17 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:59:18.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:17 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:59:18.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:17 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T19:59:18.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:17 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T19:59:18.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:17 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T19:59:18.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:17 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T19:59:18.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:17 smithi097 ceph-mon[31898]: mgrmap e22: x(active, since 1.02468s), standbys: y, z 2024-03-11T19:59:18.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.218+0000 7f798c800640 1 -- 172.21.15.97:0/3553737943 --> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f798402ceb0 con 0x7f796c0a1960 2024-03-11T19:59:18.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.315+0000 7f797ea00640 1 -- 172.21.15.97:0/3553737943 <== mgr.15396 v2:172.21.15.97:6800/1085760108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f798402ceb0 con 0x7f796c0a1960 2024-03-11T19:59:18.313 INFO:teuthology.orchestra.run.smithi097.stdout:Retention added to path / 2024-03-11T19:59:18.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.320+0000 7f798c800640 1 -- 172.21.15.97:0/3553737943 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f796c0a1960 msgr2=0x7f796c0a3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:18.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.320+0000 7f798c800640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f796c0a1960 0x7f796c0a3e20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7984104430 tx=0x7f7978005f90 comp rx=0 tx=0).stop 2024-03-11T19:59:18.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.320+0000 7f798c800640 1 -- 172.21.15.97:0/3553737943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79841045d0 msgr2=0x7f79841998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:18.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.320+0000 7f798c800640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79841045d0 0x7f79841998c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7974006010 tx=0x7f7974005e80 comp rx=0 tx=0).stop 2024-03-11T19:59:18.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.320+0000 7f798c800640 1 -- 172.21.15.97:0/3553737943 shutdown_connections 2024-03-11T19:59:18.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.320+0000 7f798c800640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f796c0a1960 0x7f796c0a3e20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.320+0000 7f798c800640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7984104f90 0x7f79841a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.320+0000 7f798c800640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79841045d0 0x7f79841998c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.320+0000 7f798c800640 1 --2- 172.21.15.97:0/3553737943 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f79841033e0 0x7f7984199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.320+0000 7f798c800640 1 -- 172.21.15.97:0/3553737943 >> 172.21.15.97:0/3553737943 conn(0x7f79840fe970 msgr2=0x7f7984107870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:18.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.320+0000 7f798c800640 1 -- 172.21.15.97:0/3553737943 shutdown_connections 2024-03-11T19:59:18.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.320+0000 7f798c800640 1 -- 172.21.15.97:0/3553737943 wait complete. 2024-03-11T19:59:18.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-11T19:59:18.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.522+0000 7fc3b6e00640 1 -- 172.21.15.97:0/798270212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3b00751c0 msgr2=0x7fc3b0073610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:18.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.522+0000 7fc3b6e00640 1 --2- 172.21.15.97:0/798270212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3b00751c0 0x7fc3b0073610 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009900 tx=0x7fc3a002bc30 comp rx=0 tx=0).stop 2024-03-11T19:59:18.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.522+0000 7fc3b6e00640 1 -- 172.21.15.97:0/798270212 shutdown_connections 2024-03-11T19:59:18.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.522+0000 7fc3b6e00640 1 --2- 172.21.15.97:0/798270212 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3b0109070 0x7fc3b010b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.522+0000 7fc3b6e00640 1 --2- 172.21.15.97:0/798270212 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3b0073be0 0x7fc3b0074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.522+0000 7fc3b6e00640 1 --2- 172.21.15.97:0/798270212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3b00751c0 0x7fc3b0073610 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.522+0000 7fc3b6e00640 1 -- 172.21.15.97:0/798270212 >> 172.21.15.97:0/798270212 conn(0x7fc3b00fc740 msgr2=0x7fc3b00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:18.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.522+0000 7fc3b6e00640 1 -- 172.21.15.97:0/798270212 shutdown_connections 2024-03-11T19:59:18.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.522+0000 7fc3b6e00640 1 -- 172.21.15.97:0/798270212 wait complete. 2024-03-11T19:59:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.523+0000 7fc3b6e00640 1 Processor -- start 2024-03-11T19:59:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.523+0000 7fc3b6e00640 1 -- start start 2024-03-11T19:59:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3b6e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3b0073be0 0x7fc3b01993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3b6e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3b00751c0 0x7fc3b01998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3b6e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3b0109070 0x7fc3b01a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3b6e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b01a0e90 con 0x7fc3b0073be0 2024-03-11T19:59:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3b6e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b01a1000 con 0x7fc3b0109070 2024-03-11T19:59:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3b6e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b01a11c0 con 0x7fc3b00751c0 2024-03-11T19:59:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3afe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3b0073be0 0x7fc3b01993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3b4e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3b0109070 0x7fc3b01a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3af400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3b00751c0 0x7fc3b01998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3b4e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3b0109070 0x7fc3b01a0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:48294/0 (socket says 172.21.15.97:48294) 2024-03-11T19:59:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3af400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3b00751c0 0x7fc3b01998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:55758/0 (socket says 172.21.15.97:55758) 2024-03-11T19:59:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3b4e00640 1 -- 172.21.15.97:0/2637097640 learned_addr learned my addr 172.21.15.97:0/2637097640 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3afe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3b0073be0 0x7fc3b01993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37210/0 (socket says 172.21.15.97:37210) 2024-03-11T19:59:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3af400640 1 -- 172.21.15.97:0/2637097640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3b0109070 msgr2=0x7fc3b01a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.524+0000 7fc3af400640 1 --2- 172.21.15.97:0/2637097640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3b0109070 0x7fc3b01a0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.525+0000 7fc3af400640 1 -- 172.21.15.97:0/2637097640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3b0073be0 msgr2=0x7fc3b01993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.525+0000 7fc3af400640 1 --2- 172.21.15.97:0/2637097640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3b0073be0 0x7fc3b01993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.525+0000 7fc3af400640 1 -- 172.21.15.97:0/2637097640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a0009560 con 0x7fc3b00751c0 2024-03-11T19:59:18.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.525+0000 7fc3af400640 1 --2- 172.21.15.97:0/2637097640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3b00751c0 0x7fc3b01998e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00e9f0 tx=0x7fc39c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:18.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.525+0000 7fc3acc00640 1 -- 172.21.15.97:0/2637097640 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c00ccc0 con 0x7fc3b00751c0 2024-03-11T19:59:18.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.525+0000 7fc3b6e00640 1 -- 172.21.15.97:0/2637097640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b01a14a0 con 0x7fc3b00751c0 2024-03-11T19:59:18.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.525+0000 7fc3acc00640 1 -- 172.21.15.97:0/2637097640 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc39c00ce20 con 0x7fc3b00751c0 2024-03-11T19:59:18.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.525+0000 7fc3acc00640 1 -- 172.21.15.97:0/2637097640 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c018920 con 0x7fc3b00751c0 2024-03-11T19:59:18.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.525+0000 7fc3b6e00640 1 -- 172.21.15.97:0/2637097640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b01a19b0 con 0x7fc3b00751c0 2024-03-11T19:59:18.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.527+0000 7fc3b6e00640 1 -- 172.21.15.97:0/2637097640 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b0107da0 con 0x7fc3b00751c0 2024-03-11T19:59:18.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.529+0000 7fc3acc00640 1 -- 172.21.15.97:0/2637097640 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 22) v1 ==== 136091+0+0 (secure 0 0 0) 0x7fc39c018a80 con 0x7fc3b00751c0 2024-03-11T19:59:18.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.530+0000 7fc3acc00640 1 --2- 172.21.15.97:0/2637097640 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc3940a18e0 0x7fc3940a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:18.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.531+0000 7fc3acc00640 1 -- 172.21.15.97:0/2637097640 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7fc39c014070 con 0x7fc3b00751c0 2024-03-11T19:59:18.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.531+0000 7fc3afe00640 1 --2- 172.21.15.97:0/2637097640 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc3940a18e0 0x7fc3940a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:18.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.531+0000 7fc3afe00640 1 --2- 172.21.15.97:0/2637097640 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc3940a18e0 0x7fc3940a3da0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009230 tx=0x7fc3a0005fd0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:18.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.535+0000 7fc3acc00640 1 -- 172.21.15.97:0/2637097640 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc39c052d20 con 0x7fc3b00751c0 2024-03-11T19:59:18.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.725+0000 7fc3acc00640 1 -- 172.21.15.97:0/2637097640 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mgrmap(e 23) v1 ==== 136399+0+0 (secure 0 0 0) 0x7fc39c096c00 con 0x7fc3b00751c0 2024-03-11T19:59:18.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.754+0000 7fc3b6e00640 1 -- 172.21.15.97:0/2637097640 --> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3b002ce40 con 0x7fc3940a18e0 2024-03-11T19:59:18.766 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.769+0000 7fc3acc00640 1 -- 172.21.15.97:0/2637097640 <== mgr.15396 v2:172.21.15.97:6800/1085760108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7fc3b002ce40 con 0x7fc3940a18e0 2024-03-11T19:59:18.767 INFO:teuthology.orchestra.run.smithi097.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-11T00:00:00", "created": "2024-03-11T19:59:17", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-11T19:59:18.769 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.772+0000 7fc3b6e00640 1 -- 172.21.15.97:0/2637097640 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc3940a18e0 msgr2=0x7fc3940a3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.772+0000 7fc3b6e00640 1 --2- 172.21.15.97:0/2637097640 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc3940a18e0 0x7fc3940a3da0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009230 tx=0x7fc3a0005fd0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.772+0000 7fc3b6e00640 1 -- 172.21.15.97:0/2637097640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3b00751c0 msgr2=0x7fc3b01998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.772+0000 7fc3b6e00640 1 --2- 172.21.15.97:0/2637097640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3b00751c0 0x7fc3b01998e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00e9f0 tx=0x7fc39c00eeb0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.772+0000 7fc3b6e00640 1 -- 172.21.15.97:0/2637097640 shutdown_connections 2024-03-11T19:59:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.772+0000 7fc3b6e00640 1 --2- 172.21.15.97:0/2637097640 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc3940a18e0 0x7fc3940a3da0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.772+0000 7fc3b6e00640 1 --2- 172.21.15.97:0/2637097640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc3b0109070 0x7fc3b01a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.772+0000 7fc3b6e00640 1 --2- 172.21.15.97:0/2637097640 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3b00751c0 0x7fc3b01998e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.772+0000 7fc3b6e00640 1 --2- 172.21.15.97:0/2637097640 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3b0073be0 0x7fc3b01993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.772+0000 7fc3b6e00640 1 -- 172.21.15.97:0/2637097640 >> 172.21.15.97:0/2637097640 conn(0x7fc3b00fc740 msgr2=0x7fc3b0104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.772+0000 7fc3b6e00640 1 -- 172.21.15.97:0/2637097640 shutdown_connections 2024-03-11T19:59:18.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.772+0000 7fc3b6e00640 1 -- 172.21.15.97:0/2637097640 wait complete. 2024-03-11T19:59:18.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-11T19:59:18.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.970+0000 7f96f4a00640 1 -- 172.21.15.97:0/1449198533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96f0101290 msgr2=0x7f96f0101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:18.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.970+0000 7f96f4a00640 1 --2- 172.21.15.97:0/1449198533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96f0101290 0x7f96f0101710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f96dc009900 tx=0x7f96dc02bc30 comp rx=0 tx=0).stop 2024-03-11T19:59:18.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.970+0000 7f96f4a00640 1 -- 172.21.15.97:0/1449198533 shutdown_connections 2024-03-11T19:59:18.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.970+0000 7f96f4a00640 1 --2- 172.21.15.97:0/1449198533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96f0101c50 0x7f96f01084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.970+0000 7f96f4a00640 1 --2- 172.21.15.97:0/1449198533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96f0101290 0x7f96f0101710 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.970+0000 7f96f4a00640 1 --2- 172.21.15.97:0/1449198533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f01000a0 0x7f96f01004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.970+0000 7f96f4a00640 1 -- 172.21.15.97:0/1449198533 >> 172.21.15.97:0/1449198533 conn(0x7f96f00fb610 msgr2=0x7f96f00fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:18.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.971+0000 7f96f4a00640 1 -- 172.21.15.97:0/1449198533 shutdown_connections 2024-03-11T19:59:18.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.971+0000 7f96f4a00640 1 -- 172.21.15.97:0/1449198533 wait complete. 2024-03-11T19:59:18.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.971+0000 7f96f4a00640 1 Processor -- start 2024-03-11T19:59:18.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.972+0000 7f96f4a00640 1 -- start start 2024-03-11T19:59:18.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.972+0000 7f96f4a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96f01000a0 0x7f96f0199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:18.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.972+0000 7f96f4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f0101290 0x7f96f01999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.972+0000 7f96f4a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96f0101c50 0x7f96f01a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.972+0000 7f96f4a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f01a0f50 con 0x7f96f01000a0 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.972+0000 7f96f4a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f01a10c0 con 0x7f96f0101290 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.972+0000 7f96f4a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f01a1280 con 0x7f96f0101c50 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.972+0000 7f96ede00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96f01000a0 0x7f96f0199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.972+0000 7f96ede00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96f01000a0 0x7f96f0199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37234/0 (socket says 172.21.15.97:37234) 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.972+0000 7f96ede00640 1 -- 172.21.15.97:0/1399271568 learned_addr learned my addr 172.21.15.97:0/1399271568 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.973+0000 7f96ee800640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96f0101c50 0x7f96f01a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.973+0000 7f96ed400640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f0101290 0x7f96f01999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.973+0000 7f96ede00640 1 -- 172.21.15.97:0/1399271568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96f0101c50 msgr2=0x7f96f01a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.973+0000 7f96ede00640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96f0101c50 0x7f96f01a0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.973+0000 7f96ede00640 1 -- 172.21.15.97:0/1399271568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f0101290 msgr2=0x7f96f01999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.973+0000 7f96ede00640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f0101290 0x7f96f01999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.973+0000 7f96ede00640 1 -- 172.21.15.97:0/1399271568 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc009560 con 0x7f96f01000a0 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.973+0000 7f96ee800640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96f0101c50 0x7f96f01a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:59:18.971 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.973+0000 7f96ede00640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96f01000a0 0x7f96f0199460 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f96e000e9c0 tx=0x7f96e000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:18.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.974+0000 7f96e6a00640 1 -- 172.21.15.97:0/1399271568 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e00096e0 con 0x7f96f01000a0 2024-03-11T19:59:18.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.974+0000 7f96f4a00640 1 -- 172.21.15.97:0/1399271568 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f01a1560 con 0x7f96f01000a0 2024-03-11T19:59:18.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.974+0000 7f96e6a00640 1 -- 172.21.15.97:0/1399271568 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96e000cd80 con 0x7f96f01000a0 2024-03-11T19:59:18.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.974+0000 7f96e6a00640 1 -- 172.21.15.97:0/1399271568 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0010430 con 0x7f96f01000a0 2024-03-11T19:59:18.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.974+0000 7f96f4a00640 1 -- 172.21.15.97:0/1399271568 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f01a1ac0 con 0x7f96f01000a0 2024-03-11T19:59:18.973 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.975+0000 7f96f4a00640 1 -- 172.21.15.97:0/1399271568 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f010bb60 con 0x7f96f01000a0 2024-03-11T19:59:18.976 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.980+0000 7f96e6a00640 1 -- 172.21.15.97:0/1399271568 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 23) v1 ==== 136399+0+0 (secure 0 0 0) 0x7f96e0003680 con 0x7f96f01000a0 2024-03-11T19:59:18.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.981+0000 7f96e6a00640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f96d40a1cc0 0x7f96d40a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:18.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.982+0000 7f96e6a00640 1 -- 172.21.15.97:0/1399271568 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f96e0014070 con 0x7f96f01000a0 2024-03-11T19:59:18.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.982+0000 7f96ed400640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f96d40a1cc0 0x7f96d40a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:18.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.983+0000 7f96ed400640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f96d40a1cc0 0x7f96d40a4180 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f96dc000c00 tx=0x7f96dc02c7e0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:18.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:18.987+0000 7f96e6a00640 1 -- 172.21.15.97:0/1399271568 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f96e0096a00 con 0x7f96f01000a0 2024-03-11T19:59:18.986 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:18 smithi114 ceph-mon[36376]: [11/Mar/2024:19:59:17] ENGINE Bus STARTING 2024-03-11T19:59:18.986 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:18 smithi114 ceph-mon[36376]: [11/Mar/2024:19:59:17] ENGINE Serving on https://172.21.15.97:7150 2024-03-11T19:59:18.986 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:18 smithi114 ceph-mon[36376]: [11/Mar/2024:19:59:17] ENGINE Serving on http://172.21.15.97:8765 2024-03-11T19:59:18.986 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:18 smithi114 ceph-mon[36376]: [11/Mar/2024:19:59:17] ENGINE Bus STARTED 2024-03-11T19:59:18.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:18 smithi114 ceph-mon[36376]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:59:18.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:18 smithi114 ceph-mon[36376]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:59:19.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:18 smithi161 ceph-mon[36332]: [11/Mar/2024:19:59:17] ENGINE Bus STARTING 2024-03-11T19:59:19.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:18 smithi161 ceph-mon[36332]: [11/Mar/2024:19:59:17] ENGINE Serving on https://172.21.15.97:7150 2024-03-11T19:59:19.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:18 smithi161 ceph-mon[36332]: [11/Mar/2024:19:59:17] ENGINE Serving on http://172.21.15.97:8765 2024-03-11T19:59:19.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:18 smithi161 ceph-mon[36332]: [11/Mar/2024:19:59:17] ENGINE Bus STARTED 2024-03-11T19:59:19.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:18 smithi161 ceph-mon[36332]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:59:19.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:18 smithi161 ceph-mon[36332]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:59:19.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:18 smithi097 ceph-mon[31898]: [11/Mar/2024:19:59:17] ENGINE Bus STARTING 2024-03-11T19:59:19.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:18 smithi097 ceph-mon[31898]: [11/Mar/2024:19:59:17] ENGINE Serving on https://172.21.15.97:7150 2024-03-11T19:59:19.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:18 smithi097 ceph-mon[31898]: [11/Mar/2024:19:59:17] ENGINE Serving on http://172.21.15.97:8765 2024-03-11T19:59:19.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:18 smithi097 ceph-mon[31898]: [11/Mar/2024:19:59:17] ENGINE Bus STARTED 2024-03-11T19:59:19.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:18 smithi097 ceph-mon[31898]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:59:19.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:18 smithi097 ceph-mon[31898]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:59:19.202 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.205+0000 7f96f4a00640 1 -- 172.21.15.97:0/1399271568 --> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f96f00fd6e0 con 0x7f96d40a1cc0 2024-03-11T19:59:19.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.214+0000 7f96e6a00640 1 -- 172.21.15.97:0/1399271568 <== mgr.15396 v2:172.21.15.97:6800/1085760108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f96f00fd6e0 con 0x7f96d40a1cc0 2024-03-11T19:59:19.211 INFO:teuthology.orchestra.run.smithi097.stdout:/ 1m 6m3h 2024-03-11T19:59:19.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.217+0000 7f96f4a00640 1 -- 172.21.15.97:0/1399271568 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f96d40a1cc0 msgr2=0x7f96d40a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:19.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.217+0000 7f96f4a00640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f96d40a1cc0 0x7f96d40a4180 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f96dc000c00 tx=0x7f96dc02c7e0 comp rx=0 tx=0).stop 2024-03-11T19:59:19.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.217+0000 7f96f4a00640 1 -- 172.21.15.97:0/1399271568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96f01000a0 msgr2=0x7f96f0199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:19.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.217+0000 7f96f4a00640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96f01000a0 0x7f96f0199460 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f96e000e9c0 tx=0x7f96e000ee80 comp rx=0 tx=0).stop 2024-03-11T19:59:19.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.217+0000 7f96f4a00640 1 -- 172.21.15.97:0/1399271568 shutdown_connections 2024-03-11T19:59:19.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.217+0000 7f96f4a00640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f96d40a1cc0 0x7f96d40a4180 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:19.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.217+0000 7f96f4a00640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96f0101c50 0x7f96f01a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:19.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.217+0000 7f96f4a00640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f96f0101290 0x7f96f01999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:19.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.217+0000 7f96f4a00640 1 --2- 172.21.15.97:0/1399271568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f96f01000a0 0x7f96f0199460 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:19.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.217+0000 7f96f4a00640 1 -- 172.21.15.97:0/1399271568 >> 172.21.15.97:0/1399271568 conn(0x7f96f00fb610 msgr2=0x7f96f00fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:19.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.217+0000 7f96f4a00640 1 -- 172.21.15.97:0/1399271568 shutdown_connections 2024-03-11T19:59:19.215 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T19:59:19.217+0000 7f96f4a00640 1 -- 172.21.15.97:0/1399271568 wait complete. 2024-03-11T19:59:19.231 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-11T19:59:19.305 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-11T19:59:19.322 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-11T19:59:19.333 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-03-11T19:59:19.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-11T19:59:19.338 INFO:teuthology.run_tasks:Running task workunit... 2024-03-11T19:59:19.359 INFO:tasks.workunit:Pulling workunits from ref b6a4997be2b8bb74411268cc42753f29ae9a78ad 2024-03-11T19:59:19.359 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-11T19:59:19.359 INFO:tasks.workunit:timeout=3h 2024-03-11T19:59:19.359 INFO:tasks.workunit:cleanup=True 2024-03-11T19:59:19.360 DEBUG:teuthology.orchestra.run.smithi097:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-11T19:59:19.398 INFO:teuthology.orchestra.run.smithi097.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-11T19:59:19.398 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-11T19:59:19.398 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-11T19:59:19.398 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T19:59:19.398 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-11T19:59:19.398 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-11 19:56:20.076654224 +0000 2024-03-11T19:59:19.398 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-11 19:56:20.076654224 +0000 2024-03-11T19:59:19.399 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-11 19:59:05.153357217 +0000 2024-03-11T19:59:19.399 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-11 19:56:20.076654224 +0000 2024-03-11T19:59:19.399 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-11T19:59:19.399 DEBUG:teuthology.orchestra.run.smithi097:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-11T19:59:19.536 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b6a4997be2b8bb74411268cc42753f29ae9a78ad 2024-03-11T19:59:19.583 INFO:tasks.workunit.client.0.smithi097.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-11T19:59:19.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:19 smithi097 ceph-mon[31898]: from='client.25558 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:59:19.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:19 smithi097 ceph-mon[31898]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:59:19.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:19 smithi097 ceph-mon[31898]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-03-11T19:59:19.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:19 smithi097 ceph-mon[31898]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:59:19.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:19 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:19.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:19 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:19.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:19 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:19.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:19 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:19.986 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:19 smithi114 ceph-mon[36376]: from='client.25558 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:59:19.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:19 smithi114 ceph-mon[36376]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:59:19.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:19 smithi114 ceph-mon[36376]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-03-11T19:59:19.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:19 smithi114 ceph-mon[36376]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:59:19.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:19 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:19.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:19 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:19.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:19 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:19.987 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:19 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:20.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:19 smithi161 ceph-mon[36332]: from='client.25558 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:59:20.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:19 smithi161 ceph-mon[36332]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:59:20.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:19 smithi161 ceph-mon[36332]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-03-11T19:59:20.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:19 smithi161 ceph-mon[36332]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:59:20.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:19 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:20.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:19 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:20.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:19 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:20.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:19 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:59:21.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:20 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:59:21.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:20 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T19:59:21.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:21.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.285 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:21.285 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:20 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:22.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:21 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi161 to 174.1M 2024-03-11T19:59:22.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:21 smithi097 ceph-mon[31898]: Unable to set osd_memory_target on smithi161 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-11T19:59:22.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:21 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi114 to 174.1M 2024-03-11T19:59:22.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:21 smithi097 ceph-mon[31898]: Unable to set osd_memory_target on smithi114 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-11T19:59:22.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:21 smithi097 ceph-mon[31898]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:59:22.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:21 smithi097 ceph-mon[31898]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-03-11T19:59:22.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:21 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi161 to 174.1M 2024-03-11T19:59:22.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:21 smithi114 ceph-mon[36376]: Unable to set osd_memory_target on smithi161 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-11T19:59:22.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:21 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi114 to 174.1M 2024-03-11T19:59:22.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:21 smithi114 ceph-mon[36376]: Unable to set osd_memory_target on smithi114 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-11T19:59:22.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:21 smithi114 ceph-mon[36376]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:59:22.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:21 smithi114 ceph-mon[36376]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-03-11T19:59:22.282 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:21 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi161 to 174.1M 2024-03-11T19:59:22.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:21 smithi161 ceph-mon[36332]: Unable to set osd_memory_target on smithi161 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-11T19:59:22.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:21 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi114 to 174.1M 2024-03-11T19:59:22.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:21 smithi161 ceph-mon[36332]: Unable to set osd_memory_target on smithi114 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-11T19:59:22.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:21 smithi161 ceph-mon[36332]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:59:22.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:21 smithi161 ceph-mon[36332]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-03-11T19:59:23.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:23.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:23.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi097 to 176.7M 2024-03-11T19:59:23.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: Unable to set osd_memory_target on smithi097 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-11T19:59:23.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:59:23.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:22 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:59:23.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:23.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:23.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi097 to 176.7M 2024-03-11T19:59:23.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: Unable to set osd_memory_target on smithi097 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-11T19:59:23.239 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:59:23.239 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:22 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:59:23.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:23.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:23.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T19:59:23.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi097 to 176.7M 2024-03-11T19:59:23.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: Unable to set osd_memory_target on smithi097 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-11T19:59:23.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:59:23.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:22 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:59:24.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: Updating smithi097:/etc/ceph/ceph.conf 2024-03-11T19:59:24.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-11T19:59:24.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: Updating smithi161:/etc/ceph/ceph.conf 2024-03-11T19:59:24.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: Updating smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:59:24.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: Updating smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:59:24.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:59:24.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:59:24.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: Updating smithi097:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:59:24.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:59:24.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: Updating smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:59:24.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: Updating smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:59:24.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:24.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:24.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:24.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:23 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:24.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: Updating smithi097:/etc/ceph/ceph.conf 2024-03-11T19:59:24.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-11T19:59:24.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: Updating smithi161:/etc/ceph/ceph.conf 2024-03-11T19:59:24.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: Updating smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:59:24.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: Updating smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:59:24.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:59:24.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:59:24.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: Updating smithi097:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:59:24.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:59:24.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: Updating smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:59:24.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: Updating smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:59:24.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:24.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:24.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:24.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:23 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:24.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: Updating smithi097:/etc/ceph/ceph.conf 2024-03-11T19:59:24.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: Updating smithi114:/etc/ceph/ceph.conf 2024-03-11T19:59:24.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: Updating smithi161:/etc/ceph/ceph.conf 2024-03-11T19:59:24.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: Updating smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:59:24.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: Updating smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:59:24.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:59:24.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:59:24.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: Updating smithi097:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T19:59:24.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T19:59:24.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: Updating smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:59:24.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: Updating smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:59:24.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:24.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:24.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:24.284 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:23 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:24 smithi097 ceph-mon[31898]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:59:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:24 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:24 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:24 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:25.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:24 smithi114 ceph-mon[36376]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:59:25.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:24 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:25.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:24 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:25.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:24 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:25.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:24 smithi161 ceph-mon[36332]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-03-11T19:59:25.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:24 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:25.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:24 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:25.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:24 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:26.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:25 smithi097 ceph-mon[31898]: Updating smithi097:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:59:26.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:25 smithi097 ceph-mon[31898]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 133 KiB/s wr, 80 op/s 2024-03-11T19:59:26.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:25 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:59:26.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:25 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:59:26.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:25 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:59:26.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:25 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:26.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:25 smithi114 ceph-mon[36376]: Updating smithi097:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:59:26.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:25 smithi114 ceph-mon[36376]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 133 KiB/s wr, 80 op/s 2024-03-11T19:59:26.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:25 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:59:26.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:25 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:59:26.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:25 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:59:26.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:25 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:26.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:25 smithi161 ceph-mon[36332]: Updating smithi097:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T19:59:26.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:25 smithi161 ceph-mon[36332]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 133 KiB/s wr, 80 op/s 2024-03-11T19:59:26.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:25 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T19:59:26.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:25 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T19:59:26.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:25 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T19:59:26.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:25 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T19:59:28.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:27 smithi097 ceph-mon[31898]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-11T19:59:28.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:27 smithi114 ceph-mon[36376]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-11T19:59:28.282 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:27 smithi161 ceph-mon[36332]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-11T19:59:30.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:29 smithi097 ceph-mon[31898]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-11T19:59:30.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:29 smithi114 ceph-mon[36376]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-11T19:59:30.282 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:29 smithi161 ceph-mon[36332]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-11T19:59:32.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:31 smithi097 ceph-mon[31898]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-11T19:59:32.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:31 smithi114 ceph-mon[36376]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-11T19:59:32.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:31 smithi161 ceph-mon[36332]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-11T19:59:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.572+0000 7fa002a00640 1 -- 172.21.15.97:0/3139626977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ffc102c80 msgr2=0x7f9ffc10b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.572+0000 7fa002a00640 1 --2- 172.21.15.97:0/3139626977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ffc102c80 0x7f9ffc10b160 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4009860 tx=0x7f9fe402bb70 comp rx=0 tx=0).stop 2024-03-11T19:59:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.573+0000 7fa002a00640 1 -- 172.21.15.97:0/3139626977 shutdown_connections 2024-03-11T19:59:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.573+0000 7fa002a00640 1 --2- 172.21.15.97:0/3139626977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ffc10b7d0 0x7f9ffc10dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.573+0000 7fa002a00640 1 --2- 172.21.15.97:0/3139626977 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ffc102c80 0x7f9ffc10b160 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.573+0000 7fa002a00640 1 --2- 172.21.15.97:0/3139626977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ffc102360 0x7f9ffc102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.573+0000 7fa002a00640 1 -- 172.21.15.97:0/3139626977 >> 172.21.15.97:0/3139626977 conn(0x7f9ffc0771b0 msgr2=0x7f9ffc0775c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.573+0000 7fa002a00640 1 -- 172.21.15.97:0/3139626977 shutdown_connections 2024-03-11T19:59:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.573+0000 7fa002a00640 1 -- 172.21.15.97:0/3139626977 wait complete. 2024-03-11T19:59:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.574+0000 7fa002a00640 1 Processor -- start 2024-03-11T19:59:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.574+0000 7fa002a00640 1 -- start start 2024-03-11T19:59:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.574+0000 7fa002a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ffc102360 0x7f9ffc197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.574+0000 7fa002a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ffc102c80 0x7f9ffc197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.574+0000 7fa002a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ffc10b7d0 0x7f9ffc19bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.574+0000 7fa002a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc19c5d0 con 0x7f9ffc102360 2024-03-11T19:59:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.574+0000 7fa002a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc19c740 con 0x7f9ffc102c80 2024-03-11T19:59:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.574+0000 7fa002a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc19c900 con 0x7f9ffc10b7d0 2024-03-11T19:59:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.574+0000 7f9ffbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ffc102360 0x7f9ffc197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.574+0000 7f9ffbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ffc102360 0x7f9ffc197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52418/0 (socket says 172.21.15.97:52418) 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.574+0000 7f9ffbe00640 1 -- 172.21.15.97:0/4235374535 learned_addr learned my addr 172.21.15.97:0/4235374535 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.574+0000 7f9ff3e00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ffc102c80 0x7f9ffc197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.575+0000 7fa000a00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ffc10b7d0 0x7f9ffc19bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.575+0000 7f9ff3e00640 1 -- 172.21.15.97:0/4235374535 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ffc10b7d0 msgr2=0x7f9ffc19bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.575+0000 7f9ff3e00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ffc10b7d0 0x7f9ffc19bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.575+0000 7f9ff3e00640 1 -- 172.21.15.97:0/4235374535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ffc102360 msgr2=0x7f9ffc197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.575+0000 7f9ff3e00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ffc102360 0x7f9ffc197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.575+0000 7f9ff3e00640 1 -- 172.21.15.97:0/4235374535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff4009900 con 0x7f9ffc102c80 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.575+0000 7f9ffbe00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ffc102360 0x7f9ffc197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.575+0000 7f9ff3e00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ffc102c80 0x7f9ffc197ca0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9fe402c0b0 tx=0x7f9fe40045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.575+0000 7f9ff9600640 1 -- 172.21.15.97:0/4235374535 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe40349d0 con 0x7f9ffc102c80 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.576+0000 7f9ff9600640 1 -- 172.21.15.97:0/4235374535 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fe402c9f0 con 0x7f9ffc102c80 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.576+0000 7fa002a00640 1 -- 172.21.15.97:0/4235374535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe4009490 con 0x7f9ffc102c80 2024-03-11T19:59:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.576+0000 7f9ff9600640 1 -- 172.21.15.97:0/4235374535 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe40035f0 con 0x7f9ffc102c80 2024-03-11T19:59:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.576+0000 7fa002a00640 1 -- 172.21.15.97:0/4235374535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ffc19cdb0 con 0x7f9ffc102c80 2024-03-11T19:59:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.579+0000 7fa0057ec740 1 Processor -- start 2024-03-11T19:59:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.579+0000 7f9ff9600640 1 -- 172.21.15.97:0/4235374535 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) v1 ==== 136455+0+0 (secure 0 0 0) 0x7f9fe403b040 con 0x7f9ffc102c80 2024-03-11T19:59:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.579+0000 7fa0057ec740 1 -- start start 2024-03-11T19:59:32.579 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.579+0000 7fa0057ec740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab885bb20 0x564ab885a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.579 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.579+0000 7fa0057ec740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885adb0 0x564ab885b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.579 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.579+0000 7fa0057ec740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab8868740 0x564ab886ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.579+0000 7fa0057ec740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ab8816620 con 0x564ab885bb20 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.579+0000 7fa0057ec740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ab87c88c0 con 0x564ab885adb0 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.579+0000 7fa0057ec740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ab87c5df0 con 0x564ab8868740 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.579+0000 7fa000a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab885bb20 0x564ab885a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7f9ffbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885adb0 0x564ab885b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7f9ff3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab8868740 0x564ab886ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7fa000a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab885bb20 0x564ab885a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52430/0 (socket says 172.21.15.97:52430) 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7fa000a00640 1 -- 172.21.15.97:0/1249629194 learned_addr learned my addr 172.21.15.97:0/1249629194 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7f9ffbe00640 1 -- 172.21.15.97:0/1249629194 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab8868740 msgr2=0x564ab886ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7f9ffbe00640 1 --2- 172.21.15.97:0/1249629194 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab8868740 0x564ab886ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7f9ffbe00640 1 -- 172.21.15.97:0/1249629194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab885bb20 msgr2=0x564ab885a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7f9ffbe00640 1 --2- 172.21.15.97:0/1249629194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab885bb20 0x564ab885a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7f9ffbe00640 1 -- 172.21.15.97:0/1249629194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ab8742b00 con 0x564ab885adb0 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7f9ffbe00640 1 --2- 172.21.15.97:0/1249629194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885adb0 0x564ab885b230 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0114f0 tx=0x7f9fec0119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4ed06d49706844b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7f9ff1200640 1 -- 172.21.15.97:0/1249629194 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec004030 con 0x564ab885adb0 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7f9ff9600640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9fe00a1d60 0x7f9fe00a4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7f9ff1200640 1 -- 172.21.15.97:0/1249629194 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fec0035f0 con 0x564ab885adb0 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.580+0000 7f9ff1200640 1 -- 172.21.15.97:0/1249629194 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec00f940 con 0x564ab885adb0 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.581+0000 7f9ff9600640 1 -- 172.21.15.97:0/4235374535 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f9fe40ef6c0 con 0x7f9ffc102c80 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.581+0000 7fa000a00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9fe00a1d60 0x7f9fe00a4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.581+0000 7fa0057ec740 1 -- 172.21.15.97:0/1249629194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885adb0 msgr2=0x564ab885b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.581+0000 7fa0057ec740 1 --2- 172.21.15.97:0/1249629194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885adb0 0x564ab885b230 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0114f0 tx=0x7f9fec0119b0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.581+0000 7fa0057ec740 1 -- 172.21.15.97:0/1249629194 shutdown_connections 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.581+0000 7fa0057ec740 1 --2- 172.21.15.97:0/1249629194 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab8868740 0x564ab886ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.581+0000 7fa0057ec740 1 --2- 172.21.15.97:0/1249629194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885adb0 0x564ab885b230 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.581+0000 7fa0057ec740 1 --2- 172.21.15.97:0/1249629194 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab885bb20 0x564ab885a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.581+0000 7fa0057ec740 1 -- 172.21.15.97:0/1249629194 >> 172.21.15.97:0/1249629194 conn(0x564ab88224b0 msgr2=0x564ab8855aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.581+0000 7fa0057ec740 1 -- 172.21.15.97:0/1249629194 shutdown_connections 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.581+0000 7fa000a00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9fe00a1d60 0x7f9fe00a4220 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4005c40 tx=0x7f9ff4010040 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.581+0000 7fa0057ec740 1 -- 172.21.15.97:0/1249629194 wait complete. 2024-03-11T19:59:32.582 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.582+0000 7fa0057ec740 1 -- start start 2024-03-11T19:59:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.582+0000 7fa0057ec740 1 Processor -- start 2024-03-11T19:59:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7fa0057ec740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885adb0 0x564ab8865760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7fa0057ec740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885bb20 0x564ab8865ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7fa0057ec740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab8868740 0x564ab88661e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7fa0057ec740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ab87c15d0 con 0x564ab8868740 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7fa0057ec740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ab8766b50 con 0x564ab885adb0 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7fa0057ec740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ab87667f0 con 0x564ab885bb20 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7fa0057ec740 10 client.0 _ll_register_callbacks cb 0x564ab877a550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7f9ffbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885adb0 0x564ab8865760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7fa000a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab8868740 0x564ab88661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7f9ffbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885adb0 0x564ab8865760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:49104/0 (socket says 172.21.15.97:49104) 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7f9ffbe00640 1 -- 172.21.15.97:0/2674645879 learned_addr learned my addr 172.21.15.97:0/2674645879 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7f9ff3e00640 1 --2- 172.21.15.97:0/2674645879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885bb20 0x564ab8865ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7f9ffbe00640 1 -- 172.21.15.97:0/2674645879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885bb20 msgr2=0x564ab8865ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7f9ffbe00640 1 --2- 172.21.15.97:0/2674645879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885bb20 0x564ab8865ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7f9ffbe00640 1 -- 172.21.15.97:0/2674645879 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab8868740 msgr2=0x564ab88661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7f9ffbe00640 1 --2- 172.21.15.97:0/2674645879 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab8868740 0x564ab88661e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.583+0000 7f9ffbe00640 1 -- 172.21.15.97:0/2674645879 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fec009cd0 con 0x564ab885adb0 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.584+0000 7f9ffbe00640 1 --2- 172.21.15.97:0/2674645879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885adb0 0x564ab8865760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00f620 tx=0x7f9fec0035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27bff58db8250f79 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.584+0000 7f9fe9600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.584+0000 7f9fe9600640 1 -- 172.21.15.97:0/2674645879 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec003d80 con 0x564ab885adb0 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.584+0000 7fa0057ec740 1 -- 172.21.15.97:0/2674645879 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564ab8793cd0 con 0x564ab885adb0 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.584+0000 7f9fe9600640 1 -- 172.21.15.97:0/2674645879 <== mon.2 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f9fec0039f0 con 0x564ab885adb0 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.584+0000 7f9fe9600640 1 -- 172.21.15.97:0/2674645879 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fec003bb0 con 0x564ab885adb0 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.584+0000 7fa0057ec740 10 client.25598 fetch_fsmap learned FSMap version 29 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.584+0000 7fa0057ec740 1 -- 172.21.15.97:0/2674645879 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564ab87bfc50 con 0x564ab885adb0 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.585+0000 7f9fe9600640 1 -- 172.21.15.97:0/2674645879 <== mon.2 v2:172.21.15.114:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564ab8793cd0 con 0x564ab885adb0 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.585+0000 7f9fe9600640 1 -- 172.21.15.97:0/2674645879 <== mon.2 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9fec00f6c0 con 0x564ab885adb0 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.585+0000 7fa0057ec740 10 client.25598 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.585+0000 7fa0057ec740 10 client.25598 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.585+0000 7fa0057ec740 20 client.25598 populate_metadata read hostname 'smithi097' 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.585+0000 7fa0057ec740 1 --2- 172.21.15.97:0/2674645879 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x564ab8878670 0x564ab887aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.585+0000 7fa0057ec740 4 client.25598 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:59:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.585+0000 7fa0057ec740 1 -- 172.21.15.97:0/2674645879 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564ab87b7ce0 con 0x564ab8878670 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.585+0000 7f9ff3e00640 1 --2- 172.21.15.97:0/2674645879 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x564ab8878670 0x564ab887aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.586+0000 7f9ff3e00640 1 --2- 172.21.15.97:0/2674645879 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x564ab8878670 0x564ab887aa60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1c313837563149 server_cookie=78f0f67a43c9b336 in_seq=0 out_seq=0 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.586+0000 7f9fe9600640 10 client.25598 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.586+0000 7f9fe9600640 1 -- 172.21.15.97:0/2674645879 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564ab87b7ce0 con 0x564ab8878670 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.586+0000 7f9fe9600640 10 client.25598 handle_command_reply: tid=0 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.586+0000 7fa0057ec740 1 client.25598 shutdown 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.587+0000 7fa0057ec740 1 -- 172.21.15.97:0/2674645879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885adb0 msgr2=0x564ab8865760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.587+0000 7fa0057ec740 1 --2- 172.21.15.97:0/2674645879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885adb0 0x564ab8865760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00f620 tx=0x7f9fec0035f0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.587+0000 7fa0057ec740 1 -- 172.21.15.97:0/2674645879 shutdown_connections 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.587+0000 7fa0057ec740 1 --2- 172.21.15.97:0/2674645879 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x564ab8878670 0x564ab887aa60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.587+0000 7fa0057ec740 1 --2- 172.21.15.97:0/2674645879 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab8868740 0x564ab88661e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.587+0000 7fa0057ec740 1 --2- 172.21.15.97:0/2674645879 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885bb20 0x564ab8865ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.587+0000 7fa0057ec740 1 --2- 172.21.15.97:0/2674645879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885adb0 0x564ab8865760 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.587+0000 7f9fe9600640 0 client.25598 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T19:59:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.587+0000 7fa0057ec740 1 -- 172.21.15.97:0/2674645879 >> 172.21.15.97:0/2674645879 conn(0x564ab88224b0 msgr2=0x564ab88577f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.587+0000 7fa0057ec740 1 -- 172.21.15.97:0/2674645879 shutdown_connections 2024-03-11T19:59:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.587+0000 7fa0057ec740 1 -- 172.21.15.97:0/2674645879 wait complete. 2024-03-11T19:59:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.587+0000 7fa0057ec740 20 client.25598 trim_cache size 0 max 16384 2024-03-11T19:59:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.601+0000 7fa0057ec740 1 Processor -- start 2024-03-11T19:59:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.601+0000 7fa0057ec740 1 -- start start 2024-03-11T19:59:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.601+0000 7fa0057ec740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab885adb0 0x564ab8867e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.601+0000 7fa0057ec740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885bb20 0x564ab8866000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.601+0000 7fa0057ec740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab8868740 0x564ab88629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.601+0000 7fa0057ec740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ab8509e60 con 0x564ab885adb0 2024-03-11T19:59:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.601+0000 7fa0057ec740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ab8878470 con 0x564ab8868740 2024-03-11T19:59:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.601+0000 7fa0057ec740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ab885c110 con 0x564ab885bb20 2024-03-11T19:59:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.602+0000 7f9ffbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab885adb0 0x564ab8867e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.602+0000 7fa000a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab8868740 0x564ab88629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.602+0000 7f9ffbe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab885adb0 0x564ab8867e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52440/0 (socket says 172.21.15.97:52440) 2024-03-11T19:59:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.602+0000 7f9ffbe00640 1 -- 172.21.15.97:0/197888449 learned_addr learned my addr 172.21.15.97:0/197888449 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.602+0000 7f9ff3e00640 1 --2- 172.21.15.97:0/197888449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885bb20 0x564ab8866000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.602+0000 7f9ff3e00640 1 -- 172.21.15.97:0/197888449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab8868740 msgr2=0x564ab88629c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.602+0000 7f9ff3e00640 1 --2- 172.21.15.97:0/197888449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab8868740 0x564ab88629c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.602+0000 7f9ff3e00640 1 -- 172.21.15.97:0/197888449 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab885adb0 msgr2=0x564ab8867e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.602+0000 7f9ff3e00640 1 --2- 172.21.15.97:0/197888449 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab885adb0 0x564ab8867e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.602+0000 7f9ff3e00640 1 -- 172.21.15.97:0/197888449 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ab875c530 con 0x564ab885bb20 2024-03-11T19:59:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.602+0000 7f9ffbe00640 1 --2- 172.21.15.97:0/197888449 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab885adb0 0x564ab8867e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:59:32.601 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.602+0000 7f9ff3e00640 1 --2- 172.21.15.97:0/197888449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885bb20 0x564ab8866000 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9fe40eca60 tx=0x7f9fe40f5480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18226fa29914c6e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:32.604 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.603+0000 7f9febe00640 1 -- 172.21.15.97:0/197888449 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe40bb8a0 con 0x564ab885bb20 2024-03-11T19:59:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.603+0000 7f9febe00640 1 -- 172.21.15.97:0/197888449 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fe40bbf40 con 0x564ab885bb20 2024-03-11T19:59:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.603+0000 7f9febe00640 1 -- 172.21.15.97:0/197888449 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe40ba2d0 con 0x564ab885bb20 2024-03-11T19:59:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.603+0000 7fa0057ec740 1 -- 172.21.15.97:0/197888449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885bb20 msgr2=0x564ab8866000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.603+0000 7fa0057ec740 1 --2- 172.21.15.97:0/197888449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885bb20 0x564ab8866000 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9fe40eca60 tx=0x7f9fe40f5480 comp rx=0 tx=0).stop 2024-03-11T19:59:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.603+0000 7fa0057ec740 1 -- 172.21.15.97:0/197888449 shutdown_connections 2024-03-11T19:59:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.603+0000 7fa0057ec740 1 --2- 172.21.15.97:0/197888449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab8868740 0x564ab88629c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.603+0000 7fa0057ec740 1 --2- 172.21.15.97:0/197888449 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885bb20 0x564ab8866000 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.603+0000 7fa0057ec740 1 --2- 172.21.15.97:0/197888449 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab885adb0 0x564ab8867e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.603+0000 7fa0057ec740 1 -- 172.21.15.97:0/197888449 >> 172.21.15.97:0/197888449 conn(0x564ab88224b0 msgr2=0x564ab8864030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.603+0000 7fa0057ec740 1 -- 172.21.15.97:0/197888449 shutdown_connections 2024-03-11T19:59:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.604+0000 7fa0057ec740 1 -- 172.21.15.97:0/197888449 wait complete. 2024-03-11T19:59:32.605 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.604+0000 7fa0057ec740 1 -- start start 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.604+0000 7fa0057ec740 1 Processor -- start 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.604+0000 7fa0057ec740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885adb0 0x564ab886f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.604+0000 7fa0057ec740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885bb20 0x564ab886da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.604+0000 7fa0057ec740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab8868740 0x564ab886df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.604+0000 7fa0057ec740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ab886e4b0 con 0x564ab8868740 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.604+0000 7fa0057ec740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ab886e620 con 0x564ab885bb20 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.604+0000 7fa0057ec740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ab886e7e0 con 0x564ab885adb0 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.605+0000 7fa0057ec740 10 client.0 _ll_register_callbacks cb 0x7f9fec01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.605+0000 7fa000a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab8868740 0x564ab886df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.605+0000 7f9ffbe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885adb0 0x564ab886f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.605+0000 7fa000a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab8868740 0x564ab886df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:52450/0 (socket says 172.21.15.97:52450) 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.605+0000 7f9ff3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885bb20 0x564ab886da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.605+0000 7fa000a00640 1 -- 172.21.15.97:0/1619438373 learned_addr learned my addr 172.21.15.97:0/1619438373 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:32.606 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.605+0000 7fa000a00640 1 -- 172.21.15.97:0/1619438373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885adb0 msgr2=0x564ab886f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.605+0000 7fa000a00640 1 --2- 172.21.15.97:0/1619438373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885adb0 0x564ab886f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.605+0000 7fa000a00640 1 -- 172.21.15.97:0/1619438373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885bb20 msgr2=0x564ab886da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.605+0000 7fa000a00640 1 --2- 172.21.15.97:0/1619438373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885bb20 0x564ab886da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.605+0000 7fa000a00640 1 -- 172.21.15.97:0/1619438373 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ab87bfc50 con 0x564ab8868740 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.605+0000 7fa000a00640 1 --2- 172.21.15.97:0/1619438373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab8868740 0x564ab886df70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4003ce0 tx=0x7f9ff4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb0192287c7b07 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.605+0000 7f9fe9600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.606+0000 7f9fe9600640 1 -- 172.21.15.97:0/1619438373 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff4013040 con 0x564ab8868740 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.606+0000 7f9fe9600640 1 -- 172.21.15.97:0/1619438373 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f9ff40047c0 con 0x564ab8868740 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.606+0000 7fa0057ec740 1 -- 172.21.15.97:0/1619438373 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564ab87e6d30 con 0x564ab8868740 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.606+0000 7f9fe9600640 1 -- 172.21.15.97:0/1619438373 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ff4004b20 con 0x564ab8868740 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.606+0000 7fa0057ec740 10 client.15486 fetch_fsmap learned FSMap version 29 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.606+0000 7fa0057ec740 1 -- 172.21.15.97:0/1619438373 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f9ffc19cdb0 con 0x564ab8868740 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.606+0000 7f9fe9600640 1 -- 172.21.15.97:0/1619438373 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564ab87e6d30 con 0x564ab8868740 2024-03-11T19:59:32.607 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.607+0000 7f9fe9600640 1 -- 172.21.15.97:0/1619438373 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9ff4027460 con 0x564ab8868740 2024-03-11T19:59:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.607+0000 7fa0057ec740 10 client.15486 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T19:59:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.607+0000 7fa0057ec740 10 client.15486 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T19:59:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.607+0000 7fa0057ec740 20 client.15486 populate_metadata read hostname 'smithi097' 2024-03-11T19:59:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.607+0000 7fa0057ec740 1 --2- 172.21.15.97:0/1619438373 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x564ab88d0070 0x564ab88d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.607+0000 7fa0057ec740 4 client.15486 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T19:59:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.607+0000 7fa0057ec740 1 -- 172.21.15.97:0/1619438373 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564ab84c1600 con 0x564ab88d0070 2024-03-11T19:59:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.607+0000 7f9ffbe00640 1 --2- 172.21.15.97:0/1619438373 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x564ab88d0070 0x564ab88d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.607+0000 7f9ffbe00640 1 --2- 172.21.15.97:0/1619438373 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x564ab88d0070 0x564ab88d2490 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75966bdd76af4763 server_cookie=989a30a003765653 in_seq=0 out_seq=0 2024-03-11T19:59:32.608 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.607+0000 7f9fe9600640 10 client.15486 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T19:59:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.621+0000 7f9fe9600640 1 -- 172.21.15.97:0/1619438373 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564ab84c1600 con 0x564ab88d0070 2024-03-11T19:59:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7f9fe9600640 10 client.15486 handle_command_reply: tid=0 2024-03-11T19:59:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7fa0057ec740 1 client.15486 shutdown 2024-03-11T19:59:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7fa0057ec740 1 -- 172.21.15.97:0/1619438373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab8868740 msgr2=0x564ab886df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7fa0057ec740 1 --2- 172.21.15.97:0/1619438373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab8868740 0x564ab886df70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4003ce0 tx=0x7f9ff4004ab0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7fa0057ec740 1 -- 172.21.15.97:0/1619438373 shutdown_connections 2024-03-11T19:59:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7fa0057ec740 1 --2- 172.21.15.97:0/1619438373 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x564ab88d0070 0x564ab88d2490 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7fa0057ec740 1 --2- 172.21.15.97:0/1619438373 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564ab8868740 0x564ab886df70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.620 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7fa0057ec740 1 --2- 172.21.15.97:0/1619438373 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564ab885bb20 0x564ab886da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7fa0057ec740 1 --2- 172.21.15.97:0/1619438373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564ab885adb0 0x564ab886f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7fa0057ec740 1 -- 172.21.15.97:0/1619438373 >> 172.21.15.97:0/1619438373 conn(0x564ab88224b0 msgr2=0x564ab88544b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7f9fe9600640 0 client.15486 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T19:59:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7fa0057ec740 1 -- 172.21.15.97:0/1619438373 shutdown_connections 2024-03-11T19:59:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7fa0057ec740 1 -- 172.21.15.97:0/1619438373 wait complete. 2024-03-11T19:59:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.622+0000 7fa0057ec740 20 client.15486 trim_cache size 0 max 16384 2024-03-11T19:59:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-03-11T19:59:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-03-11T19:59:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-03-11T19:59:32.621 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-03-11T19:59:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.623+0000 7fa002a00640 1 -- 172.21.15.97:0/4235374535 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9fe00a1d60 msgr2=0x7f9fe00a4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.623+0000 7fa002a00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9fe00a1d60 0x7f9fe00a4220 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4005c40 tx=0x7f9ff4010040 comp rx=0 tx=0).stop 2024-03-11T19:59:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.623+0000 7fa002a00640 1 -- 172.21.15.97:0/4235374535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ffc102c80 msgr2=0x7f9ffc197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.623+0000 7fa002a00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ffc102c80 0x7f9ffc197ca0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9fe402c0b0 tx=0x7f9fe40045b0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.623+0000 7fa002a00640 1 -- 172.21.15.97:0/4235374535 shutdown_connections 2024-03-11T19:59:32.622 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.623+0000 7fa002a00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9fe00a1d60 0x7f9fe00a4220 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.623+0000 7fa002a00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ffc10b7d0 0x7f9ffc19bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.623+0000 7fa002a00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ffc102c80 0x7f9ffc197ca0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.623+0000 7fa002a00640 1 --2- 172.21.15.97:0/4235374535 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ffc102360 0x7f9ffc197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.623+0000 7fa002a00640 1 -- 172.21.15.97:0/4235374535 >> 172.21.15.97:0/4235374535 conn(0x7f9ffc0771b0 msgr2=0x7f9ffc0fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.623+0000 7fa002a00640 1 -- 172.21.15.97:0/4235374535 shutdown_connections 2024-03-11T19:59:32.623 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:32.623+0000 7fa002a00640 1 -- 172.21.15.97:0/4235374535 wait complete. 2024-03-11T19:59:33.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:32 smithi114 ceph-mon[36376]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-11T19:59:33.282 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:32 smithi161 ceph-mon[36332]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-11T19:59:33.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:32 smithi097 ceph-mon[31898]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-11T19:59:35.986 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:35 smithi114 ceph-mon[36376]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 79 KiB/s wr, 59 op/s 2024-03-11T19:59:36.032 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:35 smithi161 ceph-mon[36332]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 79 KiB/s wr, 59 op/s 2024-03-11T19:59:36.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:35 smithi097 ceph-mon[31898]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 79 KiB/s wr, 59 op/s 2024-03-11T19:59:38.032 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:37 smithi161 ceph-mon[36332]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-11T19:59:38.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:37 smithi097 ceph-mon[31898]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-11T19:59:38.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:37 smithi114 ceph-mon[36376]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-11T19:59:40.000 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:39 smithi114 ceph-mon[36376]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.7 KiB/s wr, 17 op/s 2024-03-11T19:59:40.032 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:39 smithi161 ceph-mon[36332]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.7 KiB/s wr, 17 op/s 2024-03-11T19:59:40.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:39 smithi097 ceph-mon[31898]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.7 KiB/s wr, 17 op/s 2024-03-11T19:59:42.032 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:41 smithi161 ceph-mon[36332]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-11T19:59:42.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:41 smithi097 ceph-mon[31898]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-11T19:59:42.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:41 smithi114 ceph-mon[36376]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-11T19:59:44.032 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:43 smithi161 ceph-mon[36332]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-11T19:59:44.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:43 smithi097 ceph-mon[31898]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-11T19:59:44.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:43 smithi114 ceph-mon[36376]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-11T19:59:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.774+0000 7f3a52c00640 1 -- 172.21.15.97:0/539994179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a4c1000d0 msgr2=0x7f3a4c1004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.774+0000 7f3a52c00640 1 --2- 172.21.15.97:0/539994179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a4c1000d0 0x7f3a4c1004b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009900 tx=0x7f3a4002bc50 comp rx=0 tx=0).stop 2024-03-11T19:59:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.774+0000 7f3a52c00640 1 -- 172.21.15.97:0/539994179 shutdown_connections 2024-03-11T19:59:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.774+0000 7f3a52c00640 1 --2- 172.21.15.97:0/539994179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c1051a0 0x7f3a4c10cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.774+0000 7f3a52c00640 1 --2- 172.21.15.97:0/539994179 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a4c100a80 0x7f3a4c104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.774+0000 7f3a52c00640 1 --2- 172.21.15.97:0/539994179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a4c1000d0 0x7f3a4c1004b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.774+0000 7f3a52c00640 1 -- 172.21.15.97:0/539994179 >> 172.21.15.97:0/539994179 conn(0x7f3a4c0fb990 msgr2=0x7f3a4c0fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.774+0000 7f3a52c00640 1 -- 172.21.15.97:0/539994179 shutdown_connections 2024-03-11T19:59:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.774+0000 7f3a52c00640 1 -- 172.21.15.97:0/539994179 wait complete. 2024-03-11T19:59:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.775+0000 7f3a52c00640 1 Processor -- start 2024-03-11T19:59:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.775+0000 7f3a52c00640 1 -- start start 2024-03-11T19:59:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.775+0000 7f3a52c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a4c1000d0 0x7f3a4c19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.775+0000 7f3a52c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a4c1051a0 0x7f3a4c19fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.775+0000 7f3a52c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c1997a0 0x7f3a4c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.775+0000 7f3a52c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c19a270 con 0x7f3a4c1051a0 2024-03-11T19:59:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.775+0000 7f3a52c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c19a3e0 con 0x7f3a4c1997a0 2024-03-11T19:59:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.775+0000 7f3a52c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c19a5a0 con 0x7f3a4c1000d0 2024-03-11T19:59:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.776+0000 7f3a51800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a4c1000d0 0x7f3a4c19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.776+0000 7f3a50e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a4c1051a0 0x7f3a4c19fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.776+0000 7f3a51800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a4c1000d0 0x7f3a4c19f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:37434/0 (socket says 172.21.15.97:37434) 2024-03-11T19:59:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.776+0000 7f3a51800640 1 -- 172.21.15.97:0/2352625894 learned_addr learned my addr 172.21.15.97:0/2352625894 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.776+0000 7f3a52200640 1 --2- 172.21.15.97:0/2352625894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c1997a0 0x7f3a4c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.776+0000 7f3a50e00640 1 -- 172.21.15.97:0/2352625894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a4c1000d0 msgr2=0x7f3a4c19f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.776+0000 7f3a50e00640 1 --2- 172.21.15.97:0/2352625894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a4c1000d0 0x7f3a4c19f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.776+0000 7f3a50e00640 1 -- 172.21.15.97:0/2352625894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c1997a0 msgr2=0x7f3a4c199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.776+0000 7f3a50e00640 1 --2- 172.21.15.97:0/2352625894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c1997a0 0x7f3a4c199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.776+0000 7f3a50e00640 1 -- 172.21.15.97:0/2352625894 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a38009490 con 0x7f3a4c1051a0 2024-03-11T19:59:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.776+0000 7f3a50e00640 1 --2- 172.21.15.97:0/2352625894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a4c1051a0 0x7f3a4c19fbd0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3a3800e9c0 tx=0x7f3a3800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.777+0000 7f3a4a000640 1 -- 172.21.15.97:0/2352625894 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3800cd80 con 0x7f3a4c1051a0 2024-03-11T19:59:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.777+0000 7f3a4a000640 1 -- 172.21.15.97:0/2352625894 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a380035f0 con 0x7f3a4c1051a0 2024-03-11T19:59:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.777+0000 7f3a52c00640 1 -- 172.21.15.97:0/2352625894 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a40009560 con 0x7f3a4c1051a0 2024-03-11T19:59:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.777+0000 7f3a4a000640 1 -- 172.21.15.97:0/2352625894 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38010670 con 0x7f3a4c1051a0 2024-03-11T19:59:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.777+0000 7f3a52c00640 1 -- 172.21.15.97:0/2352625894 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4c1a6840 con 0x7f3a4c1051a0 2024-03-11T19:59:45.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.780+0000 7f3a56e6e740 1 Processor -- start 2024-03-11T19:59:45.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.780+0000 7f3a4a000640 1 -- 172.21.15.97:0/2352625894 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 24) v1 ==== 136455+0+0 (secure 0 0 0) 0x7f3a38003af0 con 0x7f3a4c1051a0 2024-03-11T19:59:45.777 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.780+0000 7f3a56e6e740 1 -- start start 2024-03-11T19:59:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.781+0000 7f3a56e6e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 0x557c59cc6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.781+0000 7f3a56e6e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cc6ff0 0x557c59cc7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.781+0000 7f3a56e6e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cd0570 0x557c59cd2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.781+0000 7f3a56e6e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c59c82620 con 0x557c59cd0570 2024-03-11T19:59:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.781+0000 7f3a56e6e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c59c348c0 con 0x557c59cc6ff0 2024-03-11T19:59:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.781+0000 7f3a56e6e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c59c31df0 con 0x557c59cc7d60 2024-03-11T19:59:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.781+0000 7f3a52200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 0x557c59cc6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.781+0000 7f3a50e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cd0570 0x557c59cd2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.781+0000 7f3a52200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 0x557c59cc6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:37440/0 (socket says 172.21.15.97:37440) 2024-03-11T19:59:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.781+0000 7f3a52200640 1 -- 172.21.15.97:0/1139408192 learned_addr learned my addr 172.21.15.97:0/1139408192 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.781+0000 7f3a51800640 1 --2- 172.21.15.97:0/1139408192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cc6ff0 0x557c59cc7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.782+0000 7f3a52200640 1 -- 172.21.15.97:0/1139408192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cc6ff0 msgr2=0x557c59cc7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.782+0000 7f3a52200640 1 --2- 172.21.15.97:0/1139408192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cc6ff0 0x557c59cc7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.782+0000 7f3a52200640 1 -- 172.21.15.97:0/1139408192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cd0570 msgr2=0x557c59cd2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.782+0000 7f3a52200640 1 --2- 172.21.15.97:0/1139408192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cd0570 0x557c59cd2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.782+0000 7f3a52200640 1 -- 172.21.15.97:0/1139408192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c59baeb00 con 0x557c59cc7d60 2024-03-11T19:59:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.782+0000 7f3a52200640 1 --2- 172.21.15.97:0/1139408192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 0x557c59cc6ab0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009f50 tx=0x7f3a440118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5016d3ca54fb30c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.782+0000 7f3a4a000640 1 --2- 172.21.15.97:0/2352625894 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f3a340a1c20 0x7f3a340a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.782+0000 7f3a3ea00640 1 -- 172.21.15.97:0/1139408192 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44004d20 con 0x557c59cc7d60 2024-03-11T19:59:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.782+0000 7f3a3ea00640 1 -- 172.21.15.97:0/1139408192 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a44004e80 con 0x557c59cc7d60 2024-03-11T19:59:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.782+0000 7f3a3ea00640 1 -- 172.21.15.97:0/1139408192 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44005280 con 0x557c59cc7d60 2024-03-11T19:59:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.782+0000 7f3a4a000640 1 -- 172.21.15.97:0/2352625894 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f3a38014070 con 0x7f3a4c1051a0 2024-03-11T19:59:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.782+0000 7f3a51800640 1 --2- 172.21.15.97:0/2352625894 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f3a340a1c20 0x7f3a340a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.783+0000 7f3a56e6e740 1 -- 172.21.15.97:0/1139408192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 msgr2=0x557c59cc6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.783+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/1139408192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 0x557c59cc6ab0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009f50 tx=0x7f3a440118d0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.783+0000 7f3a51800640 1 --2- 172.21.15.97:0/2352625894 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f3a340a1c20 0x7f3a340a40e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3a40000c00 tx=0x7f3a400058d0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.783+0000 7f3a56e6e740 1 -- 172.21.15.97:0/1139408192 shutdown_connections 2024-03-11T19:59:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.783+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/1139408192 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cd0570 0x557c59cd2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.783+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/1139408192 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cc6ff0 0x557c59cc7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.783+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/1139408192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 0x557c59cc6ab0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.783+0000 7f3a56e6e740 1 -- 172.21.15.97:0/1139408192 >> 172.21.15.97:0/1139408192 conn(0x557c59c8e4b0 msgr2=0x557c59cc1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.783+0000 7f3a56e6e740 1 -- 172.21.15.97:0/1139408192 shutdown_connections 2024-03-11T19:59:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.783+0000 7f3a56e6e740 1 -- 172.21.15.97:0/1139408192 wait complete. 2024-03-11T19:59:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.784+0000 7f3a56e6e740 1 -- start start 2024-03-11T19:59:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.784+0000 7f3a56e6e740 1 Processor -- start 2024-03-11T19:59:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.784+0000 7f3a56e6e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cc6ff0 0x557c59ccd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.784+0000 7f3a56e6e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 0x557c59ccdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.784+0000 7f3a56e6e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cd0570 0x557c59cce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.784+0000 7f3a56e6e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c59c2d5d0 con 0x557c59cc7d60 2024-03-11T19:59:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.784+0000 7f3a56e6e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c59bd2b50 con 0x557c59cc6ff0 2024-03-11T19:59:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.784+0000 7f3a56e6e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c59bd27f0 con 0x557c59cd0570 2024-03-11T19:59:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.785+0000 7f3a56e6e740 10 client.0 _ll_register_callbacks cb 0x557c59be6550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:59:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.785+0000 7f3a50e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 0x557c59ccdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.785+0000 7f3a50e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 0x557c59ccdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60140/0 (socket says 172.21.15.97:60140) 2024-03-11T19:59:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.785+0000 7f3a50e00640 1 -- 172.21.15.97:0/1109410760 learned_addr learned my addr 172.21.15.97:0/1109410760 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.785+0000 7f3a51800640 1 --2- 172.21.15.97:0/1109410760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cc6ff0 0x557c59ccd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.785+0000 7f3a52200640 1 --2- 172.21.15.97:0/1109410760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cd0570 0x557c59cce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.785+0000 7f3a50e00640 1 -- 172.21.15.97:0/1109410760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cd0570 msgr2=0x557c59cce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.785+0000 7f3a50e00640 1 --2- 172.21.15.97:0/1109410760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cd0570 0x557c59cce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.785+0000 7f3a50e00640 1 -- 172.21.15.97:0/1109410760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cc6ff0 msgr2=0x557c59ccd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.785+0000 7f3a50e00640 1 --2- 172.21.15.97:0/1109410760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cc6ff0 0x557c59ccd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.785+0000 7f3a50e00640 1 -- 172.21.15.97:0/1109410760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a44009c00 con 0x557c59cc7d60 2024-03-11T19:59:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.785+0000 7f3a50e00640 1 --2- 172.21.15.97:0/1109410760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 0x557c59ccdad0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3a3800c6a0 tx=0x7f3a3800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22552fa083ff8c69 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.786+0000 7f3a33400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:59:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.786+0000 7f3a33400640 1 -- 172.21.15.97:0/1109410760 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38003960 con 0x557c59cc7d60 2024-03-11T19:59:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.786+0000 7f3a33400640 1 -- 172.21.15.97:0/1109410760 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f3a3809aec0 con 0x557c59cc7d60 2024-03-11T19:59:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.786+0000 7f3a56e6e740 1 -- 172.21.15.97:0/1109410760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557c59bffcd0 con 0x557c59cc7d60 2024-03-11T19:59:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.786+0000 7f3a33400640 1 -- 172.21.15.97:0/1109410760 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a3809a590 con 0x557c59cc7d60 2024-03-11T19:59:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.786+0000 7f3a56e6e740 10 client.15498 fetch_fsmap learned FSMap version 29 2024-03-11T19:59:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.786+0000 7f3a56e6e740 1 -- 172.21.15.97:0/1109410760 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557c59c2bc50 con 0x557c59cc7d60 2024-03-11T19:59:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.786+0000 7f3a33400640 1 -- 172.21.15.97:0/1109410760 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557c59bffcd0 con 0x557c59cc7d60 2024-03-11T19:59:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.786+0000 7f3a33400640 1 -- 172.21.15.97:0/1109410760 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3a380cb640 con 0x557c59cc7d60 2024-03-11T19:59:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.787+0000 7f3a56e6e740 10 client.15498 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T19:59:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.787+0000 7f3a56e6e740 10 client.15498 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T19:59:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.787+0000 7f3a56e6e740 20 client.15498 populate_metadata read hostname 'smithi097' 2024-03-11T19:59:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.787+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/1109410760 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557c59ce05b0 0x557c59ce29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.787+0000 7f3a56e6e740 4 client.15498 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:59:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.787+0000 7f3a56e6e740 1 -- 172.21.15.97:0/1109410760 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557c59c23ce0 con 0x557c59ce05b0 2024-03-11T19:59:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.787+0000 7f3a51800640 1 --2- 172.21.15.97:0/1109410760 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557c59ce05b0 0x557c59ce29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.788+0000 7f3a51800640 1 --2- 172.21.15.97:0/1109410760 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557c59ce05b0 0x557c59ce29d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25589787a126cd46 server_cookie=7b9033930214e87b in_seq=0 out_seq=0 2024-03-11T19:59:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.788+0000 7f3a33400640 10 client.15498 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T19:59:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.788+0000 7f3a33400640 1 -- 172.21.15.97:0/1109410760 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557c59c23ce0 con 0x557c59ce05b0 2024-03-11T19:59:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.788+0000 7f3a33400640 10 client.15498 handle_command_reply: tid=0 2024-03-11T19:59:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.788+0000 7f3a56e6e740 1 client.15498 shutdown 2024-03-11T19:59:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.789+0000 7f3a56e6e740 1 -- 172.21.15.97:0/1109410760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 msgr2=0x557c59ccdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.789+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/1109410760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 0x557c59ccdad0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3a3800c6a0 tx=0x7f3a3800cf60 comp rx=0 tx=0).stop 2024-03-11T19:59:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.789+0000 7f3a56e6e740 1 -- 172.21.15.97:0/1109410760 shutdown_connections 2024-03-11T19:59:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.789+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/1109410760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cd0570 0x557c59cce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.789+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/1109410760 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557c59ce05b0 0x557c59ce29d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.789+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/1109410760 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 0x557c59ccdad0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.789+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/1109410760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cc6ff0 0x557c59ccd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.789+0000 7f3a33400640 0 client.15498 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T19:59:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.789+0000 7f3a56e6e740 1 -- 172.21.15.97:0/1109410760 >> 172.21.15.97:0/1109410760 conn(0x557c59c8e4b0 msgr2=0x557c59cc3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.789+0000 7f3a56e6e740 1 -- 172.21.15.97:0/1109410760 shutdown_connections 2024-03-11T19:59:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.789+0000 7f3a56e6e740 1 -- 172.21.15.97:0/1109410760 wait complete. 2024-03-11T19:59:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.789+0000 7f3a56e6e740 20 client.15498 trim_cache size 0 max 16384 2024-03-11T19:59:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.805+0000 7f3a56e6e740 1 Processor -- start 2024-03-11T19:59:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.805+0000 7f3a56e6e740 1 -- start start 2024-03-11T19:59:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.805+0000 7f3a56e6e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc6ff0 0x557c59cc8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.805+0000 7f3a56e6e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 0x557c59cc8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.805+0000 7f3a56e6e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cd0570 0x557c59ccbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.805+0000 7f3a56e6e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c59975e60 con 0x557c59cc6ff0 2024-03-11T19:59:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.805+0000 7f3a56e6e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c59cc1550 con 0x557c59cd0570 2024-03-11T19:59:45.802 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.805+0000 7f3a56e6e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c59ccfff0 con 0x557c59cc7d60 2024-03-11T19:59:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.805+0000 7f3a50e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 0x557c59cc8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.805+0000 7f3a51800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc6ff0 0x557c59cc8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.805+0000 7f3a50e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 0x557c59cc8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:37458/0 (socket says 172.21.15.97:37458) 2024-03-11T19:59:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.805+0000 7f3a50e00640 1 -- 172.21.15.97:0/2529654503 learned_addr learned my addr 172.21.15.97:0/2529654503 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.805+0000 7f3a52200640 1 --2- 172.21.15.97:0/2529654503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cd0570 0x557c59ccbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.806+0000 7f3a50e00640 1 -- 172.21.15.97:0/2529654503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cd0570 msgr2=0x557c59ccbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.806+0000 7f3a50e00640 1 --2- 172.21.15.97:0/2529654503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cd0570 0x557c59ccbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.806+0000 7f3a50e00640 1 -- 172.21.15.97:0/2529654503 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc6ff0 msgr2=0x557c59cc8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.806+0000 7f3a50e00640 1 --2- 172.21.15.97:0/2529654503 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc6ff0 0x557c59cc8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.806+0000 7f3a50e00640 1 -- 172.21.15.97:0/2529654503 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c59bc8530 con 0x557c59cc7d60 2024-03-11T19:59:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.806+0000 7f3a50e00640 1 --2- 172.21.15.97:0/2529654503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 0x557c59cc8d90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3a38000c00 tx=0x7f3a38005bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec6a363d3078a945 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.806+0000 7f3a3e000640 1 -- 172.21.15.97:0/2529654503 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a380cb2d0 con 0x557c59cc7d60 2024-03-11T19:59:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.806+0000 7f3a3e000640 1 -- 172.21.15.97:0/2529654503 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a3809a590 con 0x557c59cc7d60 2024-03-11T19:59:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.806+0000 7f3a3e000640 1 -- 172.21.15.97:0/2529654503 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a380d16c0 con 0x557c59cc7d60 2024-03-11T19:59:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.807+0000 7f3a56e6e740 1 -- 172.21.15.97:0/2529654503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 msgr2=0x557c59cc8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.807+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/2529654503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 0x557c59cc8d90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3a38000c00 tx=0x7f3a38005bb0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.807+0000 7f3a56e6e740 1 -- 172.21.15.97:0/2529654503 shutdown_connections 2024-03-11T19:59:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.807+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/2529654503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cd0570 0x557c59ccbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.807+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/2529654503 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc7d60 0x557c59cc8d90 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.807+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/2529654503 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc6ff0 0x557c59cc8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.807+0000 7f3a56e6e740 1 -- 172.21.15.97:0/2529654503 >> 172.21.15.97:0/2529654503 conn(0x557c59c8e4b0 msgr2=0x557c59ce7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:45.807 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.807+0000 7f3a56e6e740 1 -- 172.21.15.97:0/2529654503 shutdown_connections 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.807+0000 7f3a56e6e740 1 -- 172.21.15.97:0/2529654503 wait complete. 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.808+0000 7f3a56e6e740 1 -- start start 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.808+0000 7f3a56e6e740 1 Processor -- start 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.808+0000 7f3a56e6e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc6ff0 0x557c59cd5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.808+0000 7f3a56e6e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 0x557c59cd55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.808+0000 7f3a56e6e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cd0570 0x557c59cd5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.808+0000 7f3a56e6e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c59d35c50 con 0x557c59cc7d60 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.808+0000 7f3a56e6e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c59d35dc0 con 0x557c59cd0570 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.808+0000 7f3a56e6e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c59d35f80 con 0x557c59cc6ff0 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.808+0000 7f3a56e6e740 10 client.0 _ll_register_callbacks cb 0x7f3a38003df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a50e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 0x557c59cd55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a52200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cd0570 0x557c59cd5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a50e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 0x557c59cd55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:60164/0 (socket says 172.21.15.97:60164) 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a50e00640 1 -- 172.21.15.97:0/2679629489 learned_addr learned my addr 172.21.15.97:0/2679629489 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a51800640 1 --2- 172.21.15.97:0/2679629489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc6ff0 0x557c59cd5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a50e00640 1 -- 172.21.15.97:0/2679629489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc6ff0 msgr2=0x557c59cd5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a50e00640 1 --2- 172.21.15.97:0/2679629489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc6ff0 0x557c59cd5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a50e00640 1 -- 172.21.15.97:0/2679629489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cd0570 msgr2=0x557c59cd5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a50e00640 1 --2- 172.21.15.97:0/2679629489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cd0570 0x557c59cd5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a50e00640 1 -- 172.21.15.97:0/2679629489 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c59c2bc50 con 0x557c59cc7d60 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a52200640 1 --2- 172.21.15.97:0/2679629489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cd0570 0x557c59cd5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a50e00640 1 --2- 172.21.15.97:0/2679629489 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 0x557c59cd55a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3a380cba50 tx=0x7f3a38003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2481e9e2883d2bb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a33400640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.809+0000 7f3a33400640 1 -- 172.21.15.97:0/2679629489 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a380cb2d0 con 0x557c59cc7d60 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.810+0000 7f3a33400640 1 -- 172.21.15.97:0/2679629489 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f3a380a2f60 con 0x557c59cc7d60 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.810+0000 7f3a56e6e740 1 -- 172.21.15.97:0/2679629489 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557c59c52d30 con 0x557c59cc7d60 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.810+0000 7f3a33400640 1 -- 172.21.15.97:0/2679629489 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a3809a120 con 0x557c59cc7d60 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.810+0000 7f3a56e6e740 10 client.15504 fetch_fsmap learned FSMap version 29 2024-03-11T19:59:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.810+0000 7f3a56e6e740 1 -- 172.21.15.97:0/2679629489 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557c59bee2c0 con 0x557c59cc7d60 2024-03-11T19:59:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.810+0000 7f3a33400640 1 -- 172.21.15.97:0/2679629489 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557c59c52d30 con 0x557c59cc7d60 2024-03-11T19:59:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.810+0000 7f3a33400640 1 -- 172.21.15.97:0/2679629489 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3a3807bd20 con 0x557c59cc7d60 2024-03-11T19:59:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.810+0000 7f3a56e6e740 10 client.15504 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T19:59:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.810+0000 7f3a56e6e740 10 client.15504 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T19:59:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.810+0000 7f3a56e6e740 20 client.15504 populate_metadata read hostname 'smithi097' 2024-03-11T19:59:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.810+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/2679629489 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557c59d37280 0x557c59d39670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.810+0000 7f3a56e6e740 4 client.15504 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T19:59:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.810+0000 7f3a56e6e740 1 -- 172.21.15.97:0/2679629489 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557c5992d600 con 0x557c59d37280 2024-03-11T19:59:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.811+0000 7f3a51800640 1 --2- 172.21.15.97:0/2679629489 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557c59d37280 0x557c59d39670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.811+0000 7f3a51800640 1 --2- 172.21.15.97:0/2679629489 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557c59d37280 0x557c59d39670 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b2086cc3c625c90 server_cookie=2d20216dfc65a562 in_seq=0 out_seq=0 2024-03-11T19:59:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.811+0000 7f3a33400640 10 client.15504 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T19:59:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.817+0000 7f3a33400640 1 -- 172.21.15.97:0/2679629489 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557c5992d600 con 0x557c59d37280 2024-03-11T19:59:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.817+0000 7f3a33400640 10 client.15504 handle_command_reply: tid=0 2024-03-11T19:59:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.817+0000 7f3a56e6e740 1 client.15504 shutdown 2024-03-11T19:59:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.817+0000 7f3a56e6e740 1 -- 172.21.15.97:0/2679629489 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 msgr2=0x557c59cd55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.817+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/2679629489 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 0x557c59cd55a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3a380cba50 tx=0x7f3a38003970 comp rx=0 tx=0).stop 2024-03-11T19:59:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.818+0000 7f3a56e6e740 1 -- 172.21.15.97:0/2679629489 shutdown_connections 2024-03-11T19:59:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.818+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/2679629489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c59cd0570 0x557c59cd5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.818+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/2679629489 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557c59d37280 0x557c59d39670 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.818+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/2679629489 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557c59cc7d60 0x557c59cd55a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.818+0000 7f3a56e6e740 1 --2- 172.21.15.97:0/2679629489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557c59cc6ff0 0x557c59cd5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.818+0000 7f3a33400640 0 client.15504 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T19:59:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.818+0000 7f3a56e6e740 1 -- 172.21.15.97:0/2679629489 >> 172.21.15.97:0/2679629489 conn(0x557c59c8e4b0 msgr2=0x557c59cc8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.818+0000 7f3a56e6e740 1 -- 172.21.15.97:0/2679629489 shutdown_connections 2024-03-11T19:59:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.818+0000 7f3a56e6e740 1 -- 172.21.15.97:0/2679629489 wait complete. 2024-03-11T19:59:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.818+0000 7f3a56e6e740 20 client.15504 trim_cache size 0 max 16384 2024-03-11T19:59:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-03-11T19:59:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-03-11T19:59:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-03-11T19:59:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-03-11T19:59:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.819+0000 7f3a52c00640 1 -- 172.21.15.97:0/2352625894 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f3a340a1c20 msgr2=0x7f3a340a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.819+0000 7f3a52c00640 1 --2- 172.21.15.97:0/2352625894 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f3a340a1c20 0x7f3a340a40e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3a40000c00 tx=0x7f3a400058d0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.819+0000 7f3a52c00640 1 -- 172.21.15.97:0/2352625894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a4c1051a0 msgr2=0x7f3a4c19fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.819+0000 7f3a52c00640 1 --2- 172.21.15.97:0/2352625894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a4c1051a0 0x7f3a4c19fbd0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3a3800e9c0 tx=0x7f3a3800ee80 comp rx=0 tx=0).stop 2024-03-11T19:59:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.819+0000 7f3a52c00640 1 -- 172.21.15.97:0/2352625894 shutdown_connections 2024-03-11T19:59:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.819+0000 7f3a52c00640 1 --2- 172.21.15.97:0/2352625894 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f3a340a1c20 0x7f3a340a40e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.819+0000 7f3a52c00640 1 --2- 172.21.15.97:0/2352625894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a4c1997a0 0x7f3a4c199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.819+0000 7f3a52c00640 1 --2- 172.21.15.97:0/2352625894 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3a4c1051a0 0x7f3a4c19fbd0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.819+0000 7f3a52c00640 1 --2- 172.21.15.97:0/2352625894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a4c1000d0 0x7f3a4c19f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.819+0000 7f3a52c00640 1 -- 172.21.15.97:0/2352625894 >> 172.21.15.97:0/2352625894 conn(0x7f3a4c0fb990 msgr2=0x7f3a4c068ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.819+0000 7f3a52c00640 1 -- 172.21.15.97:0/2352625894 shutdown_connections 2024-03-11T19:59:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:45.819+0000 7f3a52c00640 1 -- 172.21.15.97:0/2352625894 wait complete. 2024-03-11T19:59:46.032 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:45 smithi161 ceph-mon[36332]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.7 KiB/s wr, 17 op/s 2024-03-11T19:59:46.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:45 smithi097 ceph-mon[31898]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.7 KiB/s wr, 17 op/s 2024-03-11T19:59:46.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:45 smithi114 ceph-mon[36376]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.7 KiB/s wr, 17 op/s 2024-03-11T19:59:48.032 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:47 smithi161 ceph-mon[36332]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-11T19:59:48.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:47 smithi097 ceph-mon[31898]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-11T19:59:48.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:47 smithi114 ceph-mon[36376]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-11T19:59:49.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:49 smithi097 ceph-mon[31898]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 426 B/s wr, 13 op/s 2024-03-11T19:59:50.033 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:49 smithi161 ceph-mon[36332]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 426 B/s wr, 13 op/s 2024-03-11T19:59:50.056 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:49 smithi114 ceph-mon[36376]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 426 B/s wr, 13 op/s 2024-03-11T19:59:52.032 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:51 smithi161 ceph-mon[36332]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-11T19:59:52.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:51 smithi097 ceph-mon[31898]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-11T19:59:52.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:51 smithi114 ceph-mon[36376]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-11T19:59:54.032 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:53 smithi161 ceph-mon[36332]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-11T19:59:54.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:53 smithi097 ceph-mon[31898]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-11T19:59:54.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:53 smithi114 ceph-mon[36376]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-11T19:59:56.032 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:55 smithi161 ceph-mon[36332]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-11T19:59:56.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:55 smithi097 ceph-mon[31898]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-11T19:59:56.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:55 smithi114 ceph-mon[36376]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-11T19:59:58.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:57 smithi097 ceph-mon[31898]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-11T19:59:58.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:57 smithi114 ceph-mon[36376]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-11T19:59:58.282 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:57 smithi161 ceph-mon[36332]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-11T19:59:58.951 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.954+0000 7fade3e00640 1 -- 172.21.15.97:0/2590968310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fade41011d0 msgr2=0x7fade41015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:58.951 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.954+0000 7fade3e00640 1 --2- 172.21.15.97:0/2590968310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fade41011d0 0x7fade41015b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009900 tx=0x7fadd402bc30 comp rx=0 tx=0).stop 2024-03-11T19:59:58.951 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.954+0000 7fade3e00640 1 -- 172.21.15.97:0/2590968310 shutdown_connections 2024-03-11T19:59:58.951 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.954+0000 7fade3e00640 1 --2- 172.21.15.97:0/2590968310 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fade41062a0 0x7fade410de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.951 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.954+0000 7fade3e00640 1 --2- 172.21.15.97:0/2590968310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade4101b80 0x7fade4105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.951 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.954+0000 7fade3e00640 1 --2- 172.21.15.97:0/2590968310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fade41011d0 0x7fade41015b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.954+0000 7fade3e00640 1 -- 172.21.15.97:0/2590968310 >> 172.21.15.97:0/2590968310 conn(0x7fade40fcad0 msgr2=0x7fade40feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.955+0000 7fade3e00640 1 -- 172.21.15.97:0/2590968310 shutdown_connections 2024-03-11T19:59:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.955+0000 7fade3e00640 1 -- 172.21.15.97:0/2590968310 wait complete. 2024-03-11T19:59:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.956+0000 7fade3e00640 1 Processor -- start 2024-03-11T19:59:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.956+0000 7fade3e00640 1 -- start start 2024-03-11T19:59:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.957+0000 7fade3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade41011d0 0x7fade4197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.957+0000 7fade3e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fade41062a0 0x7fade4197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.957+0000 7fade3e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fade419bf20 0x7fade419c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.957+0000 7fade3e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade419c910 con 0x7fade419bf20 2024-03-11T19:59:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.957+0000 7fade3e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade419ca80 con 0x7fade41011d0 2024-03-11T19:59:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.957+0000 7fade3e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade419cbf0 con 0x7fade41062a0 2024-03-11T19:59:58.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.957+0000 7fade2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade41011d0 0x7fade4197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.957+0000 7fade2000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fade41062a0 0x7fade4197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.957+0000 7fade3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fade419bf20 0x7fade419c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.957+0000 7fade2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade41011d0 0x7fade4197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:33626/0 (socket says 172.21.15.97:33626) 2024-03-11T19:59:58.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.957+0000 7fade2000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fade41062a0 0x7fade4197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:40404/0 (socket says 172.21.15.97:40404) 2024-03-11T19:59:58.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.957+0000 7fade2a00640 1 -- 172.21.15.97:0/731413394 learned_addr learned my addr 172.21.15.97:0/731413394 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:58.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.958+0000 7fade2a00640 1 -- 172.21.15.97:0/731413394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fade41062a0 msgr2=0x7fade4197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:58.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.958+0000 7fade2a00640 1 --2- 172.21.15.97:0/731413394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fade41062a0 0x7fade4197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.958+0000 7fade2a00640 1 -- 172.21.15.97:0/731413394 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fade419bf20 msgr2=0x7fade419c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:58.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.958+0000 7fade2a00640 1 --2- 172.21.15.97:0/731413394 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fade419bf20 0x7fade419c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.958+0000 7fade2a00640 1 -- 172.21.15.97:0/731413394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd4009560 con 0x7fade41011d0 2024-03-11T19:59:58.955 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.958+0000 7fade2000640 1 --2- 172.21.15.97:0/731413394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fade41062a0 0x7fade4197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T19:59:58.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.958+0000 7fade2a00640 1 --2- 172.21.15.97:0/731413394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade41011d0 0x7fade4197660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fadd4005940 tx=0x7fadd402c700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:58.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.958+0000 7fade8a00640 1 -- 172.21.15.97:0/731413394 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd40043d0 con 0x7fade41011d0 2024-03-11T19:59:58.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.958+0000 7fade8a00640 1 -- 172.21.15.97:0/731413394 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fadd40052d0 con 0x7fade41011d0 2024-03-11T19:59:58.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.958+0000 7fade3e00640 1 -- 172.21.15.97:0/731413394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade41a41d0 con 0x7fade41011d0 2024-03-11T19:59:58.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.958+0000 7fade8a00640 1 -- 172.21.15.97:0/731413394 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd4003cc0 con 0x7fade41011d0 2024-03-11T19:59:58.956 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.958+0000 7fade3e00640 1 -- 172.21.15.97:0/731413394 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade41a4650 con 0x7fade41011d0 2024-03-11T19:59:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.962+0000 7fadecb69740 1 Processor -- start 2024-03-11T19:59:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.962+0000 7fade8a00640 1 -- 172.21.15.97:0/731413394 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) v1 ==== 136455+0+0 (secure 0 0 0) 0x7fadd4005440 con 0x7fade41011d0 2024-03-11T19:59:58.958 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.962+0000 7fadecb69740 1 -- start start 2024-03-11T19:59:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.962+0000 7fadecb69740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb862cd60 0x55cfb862bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.962+0000 7fadecb69740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb862c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.962+0000 7fadecb69740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb8639980 0x55cfb863bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.962+0000 7fadecb69740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb85e7620 con 0x55cfb862bff0 2024-03-11T19:59:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.962+0000 7fadecb69740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb85998c0 con 0x55cfb8639980 2024-03-11T19:59:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.962+0000 7fadecb69740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb8596df0 con 0x55cfb862cd60 2024-03-11T19:59:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.962+0000 7fade2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb862c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.962+0000 7fade3400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb862cd60 0x55cfb862bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.962+0000 7fade2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb862c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57850/0 (socket says 172.21.15.97:57850) 2024-03-11T19:59:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.962+0000 7fade2a00640 1 -- 172.21.15.97:0/631501596 learned_addr learned my addr 172.21.15.97:0/631501596 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.963+0000 7fade2000640 1 --2- 172.21.15.97:0/631501596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb8639980 0x55cfb863bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.963+0000 7fade3400640 1 -- 172.21.15.97:0/631501596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb8639980 msgr2=0x55cfb863bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.963+0000 7fade3400640 1 --2- 172.21.15.97:0/631501596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb8639980 0x55cfb863bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.963+0000 7fade3400640 1 -- 172.21.15.97:0/631501596 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 msgr2=0x55cfb862c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.963+0000 7fade3400640 1 --2- 172.21.15.97:0/631501596 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb862c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.963+0000 7fade3400640 1 -- 172.21.15.97:0/631501596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cfb8513b00 con 0x55cfb862cd60 2024-03-11T19:59:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.963+0000 7fade3400640 1 --2- 172.21.15.97:0/631501596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb862cd60 0x55cfb862bab0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faddc009eb0 tx=0x7faddc0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9cec725e5b0be5b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.964+0000 7fadcfe00640 1 -- 172.21.15.97:0/631501596 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faddc003b70 con 0x55cfb862cd60 2024-03-11T19:59:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.964+0000 7fadcfe00640 1 -- 172.21.15.97:0/631501596 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faddc003cd0 con 0x55cfb862cd60 2024-03-11T19:59:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.964+0000 7fadcfe00640 1 -- 172.21.15.97:0/631501596 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faddc01aa10 con 0x55cfb862cd60 2024-03-11T19:59:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.964+0000 7fade8a00640 1 --2- 172.21.15.97:0/731413394 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fadc80a1c20 0x7fadc80a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.964+0000 7fade8a00640 1 -- 172.21.15.97:0/731413394 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7fadd40efe50 con 0x7fade41011d0 2024-03-11T19:59:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.964+0000 7fade2a00640 1 --2- 172.21.15.97:0/731413394 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fadc80a1c20 0x7fadc80a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.964+0000 7fadecb69740 1 -- 172.21.15.97:0/631501596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb862cd60 msgr2=0x55cfb862bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.964+0000 7fadecb69740 1 --2- 172.21.15.97:0/631501596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb862cd60 0x55cfb862bab0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faddc009eb0 tx=0x7faddc0107a0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.964+0000 7fadecb69740 1 -- 172.21.15.97:0/631501596 shutdown_connections 2024-03-11T19:59:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.964+0000 7fadecb69740 1 --2- 172.21.15.97:0/631501596 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb8639980 0x55cfb863bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.964+0000 7fadecb69740 1 --2- 172.21.15.97:0/631501596 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb862c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.964+0000 7fadecb69740 1 --2- 172.21.15.97:0/631501596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb862cd60 0x55cfb862bab0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.964+0000 7fadecb69740 1 -- 172.21.15.97:0/631501596 >> 172.21.15.97:0/631501596 conn(0x55cfb85f34b0 msgr2=0x55cfb8626ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.965+0000 7fadecb69740 1 -- 172.21.15.97:0/631501596 shutdown_connections 2024-03-11T19:59:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.965+0000 7fade2a00640 1 --2- 172.21.15.97:0/731413394 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fadc80a1c20 0x7fadc80a40e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fadd40b8a80 tx=0x7fadd40f3810 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.965+0000 7fadecb69740 1 -- 172.21.15.97:0/631501596 wait complete. 2024-03-11T19:59:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.965+0000 7fadecb69740 1 -- start start 2024-03-11T19:59:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.965+0000 7fadecb69740 1 Processor -- start 2024-03-11T19:59:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fadecb69740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb86369d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fadecb69740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 0x55cfb8636f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fadecb69740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 0x55cfb8637450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fadecb69740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb85925d0 con 0x55cfb862bff0 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fadecb69740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb8537b50 con 0x55cfb862cd60 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fadecb69740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb85377f0 con 0x55cfb8639980 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fadecb69740 10 client.0 _ll_register_callbacks cb 0x55cfb854b550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fade3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb86369d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fade2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 0x55cfb8636f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fade2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 0x55cfb8636f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:33644/0 (socket says 172.21.15.97:33644) 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fade2000640 1 -- 172.21.15.97:0/3461810287 learned_addr learned my addr 172.21.15.97:0/3461810287 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fade3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb86369d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57854/0 (socket says 172.21.15.97:57854) 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fade3400640 1 --2- 172.21.15.97:0/3461810287 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 0x55cfb8637450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.966+0000 7fade2000640 1 -- 172.21.15.97:0/3461810287 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 msgr2=0x55cfb8637450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.967+0000 7fade2000640 1 --2- 172.21.15.97:0/3461810287 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 0x55cfb8637450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.967+0000 7fade2000640 1 -- 172.21.15.97:0/3461810287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 msgr2=0x55cfb86369d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.967+0000 7fade2000640 1 --2- 172.21.15.97:0/3461810287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb86369d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.967+0000 7fade2000640 1 -- 172.21.15.97:0/3461810287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faddc009b10 con 0x55cfb862cd60 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.967+0000 7fade2000640 1 --2- 172.21.15.97:0/3461810287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 0x55cfb8636f10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fadd0012d20 tx=0x7fadd000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=237849cece8aa8df server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.967+0000 7fadccc00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.967+0000 7fadccc00640 1 -- 172.21.15.97:0/3461810287 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0017070 con 0x55cfb862cd60 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.967+0000 7fadecb69740 1 -- 172.21.15.97:0/3461810287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cfb8564cd0 con 0x55cfb862cd60 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.967+0000 7fadccc00640 1 -- 172.21.15.97:0/3461810287 <== mon.2 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7fadd0012110 con 0x55cfb862cd60 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.968+0000 7fadccc00640 1 -- 172.21.15.97:0/3461810287 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fadd00122d0 con 0x55cfb862cd60 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.968+0000 7fadecb69740 10 client.25634 fetch_fsmap learned FSMap version 29 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.968+0000 7fadecb69740 1 -- 172.21.15.97:0/3461810287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55cfb8590c50 con 0x55cfb862cd60 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.968+0000 7fadccc00640 1 -- 172.21.15.97:0/3461810287 <== mon.2 v2:172.21.15.114:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55cfb8564cd0 con 0x55cfb862cd60 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.968+0000 7fadccc00640 1 -- 172.21.15.97:0/3461810287 <== mon.2 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fadd0003ca0 con 0x55cfb862cd60 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.968+0000 7fadecb69740 10 client.25634 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.968+0000 7fadecb69740 10 client.25634 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.968+0000 7fadecb69740 20 client.25634 populate_metadata read hostname 'smithi097' 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.968+0000 7fadecb69740 1 --2- 172.21.15.97:0/3461810287 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55cfb8649a20 0x55cfb864be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.968+0000 7fadecb69740 4 client.25634 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T19:59:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.968+0000 7fadecb69740 1 -- 172.21.15.97:0/3461810287 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cfb8588ce0 con 0x55cfb8649a20 2024-03-11T19:59:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.968+0000 7fade3400640 1 --2- 172.21.15.97:0/3461810287 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55cfb8649a20 0x55cfb864be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.969+0000 7fadccc00640 10 client.25634 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T19:59:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.969+0000 7fade3400640 1 --2- 172.21.15.97:0/3461810287 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55cfb8649a20 0x55cfb864be40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e74fb04924420f81 server_cookie=ff24c50211751213 in_seq=0 out_seq=0 2024-03-11T19:59:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadccc00640 1 -- 172.21.15.97:0/3461810287 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cfb8588ce0 con 0x55cfb8649a20 2024-03-11T19:59:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadccc00640 10 client.25634 handle_command_reply: tid=0 2024-03-11T19:59:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadecb69740 1 client.25634 shutdown 2024-03-11T19:59:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadecb69740 1 -- 172.21.15.97:0/3461810287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 msgr2=0x55cfb8636f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadecb69740 1 --2- 172.21.15.97:0/3461810287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 0x55cfb8636f10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fadd0012d20 tx=0x7fadd000f790 comp rx=0 tx=0).stop 2024-03-11T19:59:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadecb69740 1 -- 172.21.15.97:0/3461810287 shutdown_connections 2024-03-11T19:59:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadecb69740 1 --2- 172.21.15.97:0/3461810287 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 0x55cfb8637450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadecb69740 1 --2- 172.21.15.97:0/3461810287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 0x55cfb8636f10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadecb69740 1 --2- 172.21.15.97:0/3461810287 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55cfb8649a20 0x55cfb864be40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadecb69740 1 --2- 172.21.15.97:0/3461810287 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb86369d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadecb69740 1 -- 172.21.15.97:0/3461810287 >> 172.21.15.97:0/3461810287 conn(0x55cfb85f34b0 msgr2=0x55cfb8628a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadccc00640 0 client.25634 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T19:59:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadecb69740 1 -- 172.21.15.97:0/3461810287 shutdown_connections 2024-03-11T19:59:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadecb69740 1 -- 172.21.15.97:0/3461810287 wait complete. 2024-03-11T19:59:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.970+0000 7fadecb69740 20 client.25634 trim_cache size 0 max 16384 2024-03-11T19:59:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.985+0000 7fadecb69740 1 Processor -- start 2024-03-11T19:59:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.985+0000 7fadecb69740 1 -- start start 2024-03-11T19:59:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.985+0000 7fadecb69740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb8639100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.985+0000 7fadecb69740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 0x55cfb86352a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.985+0000 7fadecb69740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 0x55cfb86357e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.985+0000 7fadecb69740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb82dae60 con 0x55cfb862bff0 2024-03-11T19:59:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.985+0000 7fadecb69740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb8627ae0 con 0x55cfb862cd60 2024-03-11T19:59:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.985+0000 7fadecb69740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb86457f0 con 0x55cfb8639980 2024-03-11T19:59:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.985+0000 7fade3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb8639100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.985+0000 7fade2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 0x55cfb86352a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.985+0000 7fade2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 0x55cfb86352a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:33654/0 (socket says 172.21.15.97:33654) 2024-03-11T19:59:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.985+0000 7fade2000640 1 -- 172.21.15.97:0/2570602985 learned_addr learned my addr 172.21.15.97:0/2570602985 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.985+0000 7fade3400640 1 --2- 172.21.15.97:0/2570602985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 0x55cfb86357e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.986+0000 7fade2000640 1 -- 172.21.15.97:0/2570602985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 msgr2=0x55cfb86357e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.986+0000 7fade2000640 1 --2- 172.21.15.97:0/2570602985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 0x55cfb86357e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.986+0000 7fade2000640 1 -- 172.21.15.97:0/2570602985 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 msgr2=0x55cfb8639100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.986+0000 7fade2000640 1 --2- 172.21.15.97:0/2570602985 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb8639100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.986+0000 7fade2000640 1 -- 172.21.15.97:0/2570602985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cfb852d530 con 0x55cfb862cd60 2024-03-11T19:59:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.986+0000 7fade2000640 1 --2- 172.21.15.97:0/2570602985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 0x55cfb86352a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009710 tx=0x7fadd000fee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=597593828679cd95 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.986+0000 7fadcf400640 1 -- 172.21.15.97:0/2570602985 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0017040 con 0x55cfb862cd60 2024-03-11T19:59:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.986+0000 7fadcf400640 1 -- 172.21.15.97:0/2570602985 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fadd0010e90 con 0x55cfb862cd60 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.986+0000 7fadcf400640 1 -- 172.21.15.97:0/2570602985 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0005030 con 0x55cfb862cd60 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.987+0000 7fadecb69740 1 -- 172.21.15.97:0/2570602985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 msgr2=0x55cfb86352a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.987+0000 7fadecb69740 1 --2- 172.21.15.97:0/2570602985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 0x55cfb86352a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009710 tx=0x7fadd000fee0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.987+0000 7fadecb69740 1 -- 172.21.15.97:0/2570602985 shutdown_connections 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.987+0000 7fadecb69740 1 --2- 172.21.15.97:0/2570602985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 0x55cfb86357e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.987+0000 7fadecb69740 1 --2- 172.21.15.97:0/2570602985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862cd60 0x55cfb86352a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.987+0000 7fadecb69740 1 --2- 172.21.15.97:0/2570602985 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862bff0 0x55cfb8639100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.987+0000 7fadecb69740 1 -- 172.21.15.97:0/2570602985 >> 172.21.15.97:0/2570602985 conn(0x55cfb85f34b0 msgr2=0x55cfb8626810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.987+0000 7fadecb69740 1 -- 172.21.15.97:0/2570602985 shutdown_connections 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.987+0000 7fadecb69740 1 -- 172.21.15.97:0/2570602985 wait complete. 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.987+0000 7fadecb69740 1 -- start start 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.988+0000 7fadecb69740 1 Processor -- start 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.988+0000 7fadecb69740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862bff0 0x55cfb863f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.988+0000 7fadecb69740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862cd60 0x55cfb863f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.988+0000 7fadecb69740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 0x55cfb863fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.988+0000 7fadecb69740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb869ff70 con 0x55cfb862cd60 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.988+0000 7fadecb69740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb86a00b0 con 0x55cfb862bff0 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.988+0000 7fadecb69740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb86a0270 con 0x55cfb8639980 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.988+0000 7fadecb69740 10 client.0 _ll_register_callbacks cb 0x7fadd0009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.988+0000 7fade3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862bff0 0x55cfb863f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.988+0000 7fade3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862bff0 0x55cfb863f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:33666/0 (socket says 172.21.15.97:33666) 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.988+0000 7fade3400640 1 -- 172.21.15.97:0/344426207 learned_addr learned my addr 172.21.15.97:0/344426207 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.988+0000 7fade2000640 1 --2- 172.21.15.97:0/344426207 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862cd60 0x55cfb863f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.988+0000 7fade3400640 1 --2- 172.21.15.97:0/344426207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 0x55cfb863fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.989+0000 7fade3400640 1 -- 172.21.15.97:0/344426207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 msgr2=0x55cfb863fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.989+0000 7fade3400640 1 --2- 172.21.15.97:0/344426207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 0x55cfb863fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.989+0000 7fade3400640 1 -- 172.21.15.97:0/344426207 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862cd60 msgr2=0x55cfb863f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.989+0000 7fade3400640 1 --2- 172.21.15.97:0/344426207 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862cd60 0x55cfb863f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.989+0000 7fade3400640 1 -- 172.21.15.97:0/344426207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cfb8590c50 con 0x55cfb862bff0 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.989+0000 7fade3400640 1 --2- 172.21.15.97:0/344426207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862bff0 0x55cfb863f310 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faddc0097c0 tx=0x7faddc01a670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adf3a188f4188c39 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.989+0000 7fadccc00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.989+0000 7fadccc00640 1 -- 172.21.15.97:0/344426207 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faddc01e040 con 0x55cfb862bff0 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.989+0000 7fadccc00640 1 -- 172.21.15.97:0/344426207 <== mon.2 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7faddc02a860 con 0x55cfb862bff0 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.989+0000 7fadecb69740 1 -- 172.21.15.97:0/344426207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cfb85b7d30 con 0x55cfb862bff0 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.989+0000 7fadccc00640 1 -- 172.21.15.97:0/344426207 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7faddc004a00 con 0x55cfb862bff0 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.989+0000 7fadecb69740 10 client.25640 fetch_fsmap learned FSMap version 29 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.989+0000 7fadecb69740 1 -- 172.21.15.97:0/344426207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55cfb85532c0 con 0x55cfb862bff0 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.990+0000 7fadccc00640 1 -- 172.21.15.97:0/344426207 <== mon.2 v2:172.21.15.114:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55cfb85b7d30 con 0x55cfb862bff0 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.990+0000 7fadccc00640 1 -- 172.21.15.97:0/344426207 <== mon.2 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7faddc034070 con 0x55cfb862bff0 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.990+0000 7fadecb69740 10 client.25640 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.990+0000 7fadecb69740 10 client.25640 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.990+0000 7fadecb69740 20 client.25640 populate_metadata read hostname 'smithi097' 2024-03-11T19:59:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.990+0000 7fadecb69740 1 --2- 172.21.15.97:0/344426207 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55cfb86a1a60 0x55cfb86a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T19:59:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.990+0000 7fadecb69740 4 client.25640 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T19:59:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.990+0000 7fadecb69740 1 -- 172.21.15.97:0/344426207 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cfb8292600 con 0x55cfb86a1a60 2024-03-11T19:59:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.990+0000 7fade2000640 1 --2- 172.21.15.97:0/344426207 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55cfb86a1a60 0x55cfb86a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T19:59:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.991+0000 7fade2000640 1 --2- 172.21.15.97:0/344426207 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55cfb86a1a60 0x55cfb86a3e80 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2aab61a588829968 server_cookie=940e16a681ac230a in_seq=0 out_seq=0 2024-03-11T19:59:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.991+0000 7fadccc00640 10 client.25640 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T19:59:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadccc00640 1 -- 172.21.15.97:0/344426207 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cfb8292600 con 0x55cfb86a1a60 2024-03-11T19:59:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadccc00640 10 client.25640 handle_command_reply: tid=0 2024-03-11T19:59:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadecb69740 1 client.25640 shutdown 2024-03-11T19:59:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadecb69740 1 -- 172.21.15.97:0/344426207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862bff0 msgr2=0x55cfb863f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadecb69740 1 --2- 172.21.15.97:0/344426207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862bff0 0x55cfb863f310 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faddc0097c0 tx=0x7faddc01a670 comp rx=0 tx=0).stop 2024-03-11T19:59:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadecb69740 1 -- 172.21.15.97:0/344426207 shutdown_connections 2024-03-11T19:59:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadecb69740 1 --2- 172.21.15.97:0/344426207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55cfb8639980 0x55cfb863fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadecb69740 1 --2- 172.21.15.97:0/344426207 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55cfb86a1a60 0x55cfb86a3e80 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadecb69740 1 --2- 172.21.15.97:0/344426207 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55cfb862cd60 0x55cfb863f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadecb69740 1 --2- 172.21.15.97:0/344426207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cfb862bff0 0x55cfb863f310 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadccc00640 0 client.25640 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T19:59:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadecb69740 1 -- 172.21.15.97:0/344426207 >> 172.21.15.97:0/344426207 conn(0x55cfb85f34b0 msgr2=0x55cfb8626810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:58.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadecb69740 1 -- 172.21.15.97:0/344426207 shutdown_connections 2024-03-11T19:59:58.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.998+0000 7fadecb69740 1 -- 172.21.15.97:0/344426207 wait complete. 2024-03-11T19:59:58.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.999+0000 7fadecb69740 20 client.25640 trim_cache size 0 max 16384 2024-03-11T19:59:58.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-03-11T19:59:58.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-03-11T19:59:58.996 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-03-11T19:59:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-03-11T19:59:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.999+0000 7fade3e00640 1 -- 172.21.15.97:0/731413394 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fadc80a1c20 msgr2=0x7fadc80a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.999+0000 7fade3e00640 1 --2- 172.21.15.97:0/731413394 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fadc80a1c20 0x7fadc80a40e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fadd40b8a80 tx=0x7fadd40f3810 comp rx=0 tx=0).stop 2024-03-11T19:59:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.999+0000 7fade3e00640 1 -- 172.21.15.97:0/731413394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade41011d0 msgr2=0x7fade4197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T19:59:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.999+0000 7fade3e00640 1 --2- 172.21.15.97:0/731413394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade41011d0 0x7fade4197660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fadd4005940 tx=0x7fadd402c700 comp rx=0 tx=0).stop 2024-03-11T19:59:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.999+0000 7fade3e00640 1 -- 172.21.15.97:0/731413394 shutdown_connections 2024-03-11T19:59:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.999+0000 7fade3e00640 1 --2- 172.21.15.97:0/731413394 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fadc80a1c20 0x7fadc80a40e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.999+0000 7fade3e00640 1 --2- 172.21.15.97:0/731413394 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fade419bf20 0x7fade419c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.999+0000 7fade3e00640 1 --2- 172.21.15.97:0/731413394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fade41062a0 0x7fade4197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.999+0000 7fade3e00640 1 --2- 172.21.15.97:0/731413394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fade41011d0 0x7fade4197660 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T19:59:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:58.999+0000 7fade3e00640 1 -- 172.21.15.97:0/731413394 >> 172.21.15.97:0/731413394 conn(0x7fade40fcad0 msgr2=0x7fade40fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T19:59:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:59.000+0000 7fade3e00640 1 -- 172.21.15.97:0/731413394 shutdown_connections 2024-03-11T19:59:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T19:59:59.000+0000 7fade3e00640 1 -- 172.21.15.97:0/731413394 wait complete. 2024-03-11T20:00:00.037 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 19:59:59 smithi097 ceph-mon[31898]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-03-11T20:00:00.084 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 19:59:59 smithi161 ceph-mon[36332]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-03-11T20:00:00.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 19:59:59 smithi114 ceph-mon[36376]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-03-11T20:00:00.665 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 20:00:00 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T20:00:00.333+0000 7f7f52600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T20:00:00.665 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 20:00:00 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T20:00:00.333+0000 7f7f52600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T20:00:00.665 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 20:00:00 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T20:00:00.333+0000 7f7f52600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T20:00:00.665 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 20:00:00 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T20:00:00.333+0000 7f7f52600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T20:00:00.665 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 20:00:00 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T20:00:00.333+0000 7f7f52600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T20:00:01.105 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:00 smithi161 ceph-mon[36332]: overall HEALTH_OK 2024-03-11T20:00:01.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:00 smithi097 ceph-mon[31898]: overall HEALTH_OK 2024-03-11T20:00:01.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:00 smithi114 ceph-mon[36376]: overall HEALTH_OK 2024-03-11T20:00:02.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:01 smithi097 ceph-mon[31898]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-11T20:00:02.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:01 smithi097 ceph-mon[31898]: mgrmap e25: x(active, since 44s), standbys: y, z 2024-03-11T20:00:02.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:01 smithi114 ceph-mon[36376]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-11T20:00:02.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:01 smithi114 ceph-mon[36376]: mgrmap e25: x(active, since 44s), standbys: y, z 2024-03-11T20:00:02.282 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:01 smithi161 ceph-mon[36332]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-11T20:00:02.282 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:01 smithi161 ceph-mon[36332]: mgrmap e25: x(active, since 44s), standbys: y, z 2024-03-11T20:00:04.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:03 smithi097 ceph-mon[31898]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-03-11T20:00:04.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:03 smithi114 ceph-mon[36376]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-03-11T20:00:04.282 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:03 smithi161 ceph-mon[36332]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 13 KiB/s wr, 14 op/s 2024-03-11T20:00:06.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:05 smithi097 ceph-mon[31898]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-03-11T20:00:06.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:05 smithi114 ceph-mon[36376]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-03-11T20:00:06.282 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:05 smithi161 ceph-mon[36332]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-03-11T20:00:08.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:07 smithi097 ceph-mon[31898]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-11T20:00:08.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:07 smithi114 ceph-mon[36376]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-11T20:00:08.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:07 smithi161 ceph-mon[36332]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-11T20:00:10.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:09 smithi097 ceph-mon[31898]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-03-11T20:00:10.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:09 smithi114 ceph-mon[36376]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-03-11T20:00:10.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:09 smithi161 ceph-mon[36332]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-03-11T20:00:12.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:11 smithi097 ceph-mon[31898]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-11T20:00:12.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.181+0000 7f2dfd200640 1 -- 172.21.15.97:0/210611783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2df8100840 msgr2=0x7f2df8104d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:12.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.181+0000 7f2dfd200640 1 --2- 172.21.15.97:0/210611783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2df8100840 0x7f2df8104d00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2de4009900 tx=0x7f2de402bd10 comp rx=0 tx=0).stop 2024-03-11T20:00:12.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.181+0000 7f2dfd200640 1 -- 172.21.15.97:0/210611783 shutdown_connections 2024-03-11T20:00:12.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.181+0000 7f2dfd200640 1 --2- 172.21.15.97:0/210611783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2df8105320 0x7f2df81057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.177 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.181+0000 7f2dfd200640 1 --2- 172.21.15.97:0/210611783 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2df8100840 0x7f2df8104d00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.181+0000 7f2dfd200640 1 --2- 172.21.15.97:0/210611783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df80fff20 0x7f2df8100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.181+0000 7f2dfd200640 1 -- 172.21.15.97:0/210611783 >> 172.21.15.97:0/210611783 conn(0x7f2df80fb990 msgr2=0x7f2df80fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.181+0000 7f2dfd200640 1 -- 172.21.15.97:0/210611783 shutdown_connections 2024-03-11T20:00:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.181+0000 7f2dfd200640 1 -- 172.21.15.97:0/210611783 wait complete. 2024-03-11T20:00:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.182+0000 7f2dfd200640 1 Processor -- start 2024-03-11T20:00:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.183+0000 7f2dfd200640 1 -- start start 2024-03-11T20:00:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.183+0000 7f2dfd200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2df80fff20 0x7f2df81a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.183+0000 7f2dfd200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df8100840 0x7f2df81a3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.183+0000 7f2dfd200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2df8105320 0x7f2df819da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.183+0000 7f2dfd200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df819e080 con 0x7f2df80fff20 2024-03-11T20:00:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.183+0000 7f2dfd200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df819e1f0 con 0x7f2df8100840 2024-03-11T20:00:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.183+0000 7f2dfd200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df819e3b0 con 0x7f2df8105320 2024-03-11T20:00:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.183+0000 7f2df7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2df80fff20 0x7f2df81a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.184+0000 7f2df7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2df80fff20 0x7f2df81a3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37734/0 (socket says 172.21.15.97:37734) 2024-03-11T20:00:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.184+0000 7f2df7400640 1 -- 172.21.15.97:0/850790291 learned_addr learned my addr 172.21.15.97:0/850790291 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.184+0000 7f2df6a00640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df8100840 0x7f2df81a3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.184+0000 7f2df7e00640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2df8105320 0x7f2df819da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.184+0000 7f2df7400640 1 -- 172.21.15.97:0/850790291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2df8105320 msgr2=0x7f2df819da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.184+0000 7f2df7400640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2df8105320 0x7f2df819da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.184+0000 7f2df7400640 1 -- 172.21.15.97:0/850790291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df8100840 msgr2=0x7f2df81a3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.184+0000 7f2df7400640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df8100840 0x7f2df81a3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.184+0000 7f2df7400640 1 -- 172.21.15.97:0/850790291 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de8009490 con 0x7f2df80fff20 2024-03-11T20:00:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.184+0000 7f2df7400640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2df80fff20 0x7f2df81a3880 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2de800ea60 tx=0x7f2de800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.185+0000 7f2defe00640 1 -- 172.21.15.97:0/850790291 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de800cde0 con 0x7f2df80fff20 2024-03-11T20:00:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.185+0000 7f2defe00640 1 -- 172.21.15.97:0/850790291 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2de80035f0 con 0x7f2df80fff20 2024-03-11T20:00:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.185+0000 7f2defe00640 1 -- 172.21.15.97:0/850790291 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8010690 con 0x7f2df80fff20 2024-03-11T20:00:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.185+0000 7f2dfd200640 1 -- 172.21.15.97:0/850790291 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de4009560 con 0x7f2df80fff20 2024-03-11T20:00:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.185+0000 7f2dfd200640 1 -- 172.21.15.97:0/850790291 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df81aa740 con 0x7f2df80fff20 2024-03-11T20:00:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.188+0000 7f2defe00640 1 -- 172.21.15.97:0/850790291 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f2de8003af0 con 0x7f2df80fff20 2024-03-11T20:00:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.190+0000 7f2e013de740 1 Processor -- start 2024-03-11T20:00:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.190+0000 7f2e013de740 1 -- start start 2024-03-11T20:00:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.190+0000 7f2defe00640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f2ddc0a1d30 0x7f2ddc0a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.190+0000 7f2e013de740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59048d60 0x560a59047ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.191+0000 7f2e013de740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59047ff0 0x560a59048470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.191+0000 7f2defe00640 1 -- 172.21.15.97:0/850790291 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f2de8014070 con 0x7f2df80fff20 2024-03-11T20:00:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.191+0000 7f2e013de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59051570 0x560a59053960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.191+0000 7f2e013de740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a59003620 con 0x560a59047ff0 2024-03-11T20:00:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.191+0000 7f2e013de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a58fb58c0 con 0x560a59051570 2024-03-11T20:00:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.191+0000 7f2e013de740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a58fb2df0 con 0x560a59048d60 2024-03-11T20:00:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.191+0000 7f2df7e00640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f2ddc0a1d30 0x7f2ddc0a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.191+0000 7f2df6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59047ff0 0x560a59048470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.191+0000 7f2df6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59047ff0 0x560a59048470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37738/0 (socket says 172.21.15.97:37738) 2024-03-11T20:00:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.191+0000 7f2df6a00640 1 -- 172.21.15.97:0/396289079 learned_addr learned my addr 172.21.15.97:0/396289079 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.191+0000 7f2df7400640 1 --2- 172.21.15.97:0/396289079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59048d60 0x560a59047ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.192+0000 7f2df7e00640 1 --2- 172.21.15.97:0/396289079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59051570 0x560a59053960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.192+0000 7f2df6a00640 1 -- 172.21.15.97:0/396289079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59048d60 msgr2=0x560a59047ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.192+0000 7f2df6a00640 1 --2- 172.21.15.97:0/396289079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59048d60 0x560a59047ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.192+0000 7f2df6a00640 1 -- 172.21.15.97:0/396289079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59051570 msgr2=0x560a59053960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.192+0000 7f2df6a00640 1 --2- 172.21.15.97:0/396289079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59051570 0x560a59053960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.192+0000 7f2df6a00640 1 -- 172.21.15.97:0/396289079 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a58f2fb00 con 0x560a59047ff0 2024-03-11T20:00:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.192+0000 7f2df7e00640 1 --2- 172.21.15.97:0/396289079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59051570 0x560a59053960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T20:00:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.192+0000 7f2df6a00640 1 --2- 172.21.15.97:0/396289079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59047ff0 0x560a59048470 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2de4000c00 tx=0x7f2de4005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0a0649a81f1259 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.192+0000 7f2df7e00640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f2ddc0a1d30 0x7f2ddc0a41f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2df0005c40 tx=0x7f2df0014950 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.193+0000 7f2decc00640 1 -- 172.21.15.97:0/396289079 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4004900 con 0x560a59047ff0 2024-03-11T20:00:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.193+0000 7f2decc00640 1 -- 172.21.15.97:0/396289079 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2de4004a60 con 0x560a59047ff0 2024-03-11T20:00:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.193+0000 7f2decc00640 1 -- 172.21.15.97:0/396289079 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4005050 con 0x560a59047ff0 2024-03-11T20:00:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.194+0000 7f2e013de740 1 -- 172.21.15.97:0/396289079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59047ff0 msgr2=0x560a59048470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.194+0000 7f2e013de740 1 --2- 172.21.15.97:0/396289079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59047ff0 0x560a59048470 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2de4000c00 tx=0x7f2de4005bd0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.194+0000 7f2e013de740 1 -- 172.21.15.97:0/396289079 shutdown_connections 2024-03-11T20:00:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.194+0000 7f2e013de740 1 --2- 172.21.15.97:0/396289079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59051570 0x560a59053960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.194+0000 7f2e013de740 1 --2- 172.21.15.97:0/396289079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59047ff0 0x560a59048470 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.194+0000 7f2e013de740 1 --2- 172.21.15.97:0/396289079 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59048d60 0x560a59047ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.194+0000 7f2e013de740 1 -- 172.21.15.97:0/396289079 >> 172.21.15.97:0/396289079 conn(0x560a5900f4b0 msgr2=0x560a59042ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.194+0000 7f2e013de740 1 -- 172.21.15.97:0/396289079 shutdown_connections 2024-03-11T20:00:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.194+0000 7f2e013de740 1 -- 172.21.15.97:0/396289079 wait complete. 2024-03-11T20:00:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.195+0000 7f2e013de740 1 -- start start 2024-03-11T20:00:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.195+0000 7f2e013de740 1 Processor -- start 2024-03-11T20:00:12.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.196+0000 7f2e013de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59047ff0 0x560a5904e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.196+0000 7f2e013de740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59048d60 0x560a5904eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.196+0000 7f2e013de740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59051570 0x560a5904f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.196+0000 7f2e013de740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a58fae5d0 con 0x560a59048d60 2024-03-11T20:00:12.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.196+0000 7f2e013de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a58f53b50 con 0x560a59047ff0 2024-03-11T20:00:12.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.196+0000 7f2e013de740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a58f537f0 con 0x560a59051570 2024-03-11T20:00:12.193 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.196+0000 7f2e013de740 10 client.0 _ll_register_callbacks cb 0x560a58f67550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:00:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.196+0000 7f2df6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59048d60 0x560a5904eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.196+0000 7f2df7e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59051570 0x560a5904f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.196+0000 7f2df7e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59051570 0x560a5904f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:50340/0 (socket says 172.21.15.97:50340) 2024-03-11T20:00:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.196+0000 7f2df7e00640 1 -- 172.21.15.97:0/1378026229 learned_addr learned my addr 172.21.15.97:0/1378026229 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.196+0000 7f2df7400640 1 --2- 172.21.15.97:0/1378026229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59047ff0 0x560a5904e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.197+0000 7f2df7e00640 1 -- 172.21.15.97:0/1378026229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59047ff0 msgr2=0x560a5904e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.197+0000 7f2df7e00640 1 --2- 172.21.15.97:0/1378026229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59047ff0 0x560a5904e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.197+0000 7f2df7e00640 1 -- 172.21.15.97:0/1378026229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59048d60 msgr2=0x560a5904eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.197+0000 7f2df7e00640 1 --2- 172.21.15.97:0/1378026229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59048d60 0x560a5904eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.197+0000 7f2df7e00640 1 -- 172.21.15.97:0/1378026229 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df8114c80 con 0x560a59051570 2024-03-11T20:00:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.197+0000 7f2df7400640 1 --2- 172.21.15.97:0/1378026229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59047ff0 0x560a5904e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:00:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.197+0000 7f2df7e00640 1 --2- 172.21.15.97:0/1378026229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59051570 0x560a5904f040 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2df0014440 tx=0x7f2df0004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c793797227eb248 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.197+0000 7f2de1600640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-11T20:00:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.197+0000 7f2de1600640 1 -- 172.21.15.97:0/1378026229 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0010ee0 con 0x560a59051570 2024-03-11T20:00:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.198+0000 7f2e013de740 1 -- 172.21.15.97:0/1378026229 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560a58f80cd0 con 0x560a59051570 2024-03-11T20:00:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.198+0000 7f2de1600640 1 -- 172.21.15.97:0/1378026229 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f2df0011dd0 con 0x560a59051570 2024-03-11T20:00:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.198+0000 7f2de1600640 1 -- 172.21.15.97:0/1378026229 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2df0004b20 con 0x560a59051570 2024-03-11T20:00:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.198+0000 7f2e013de740 10 client.25648 fetch_fsmap learned FSMap version 29 2024-03-11T20:00:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.198+0000 7f2e013de740 1 -- 172.21.15.97:0/1378026229 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x560a58facc50 con 0x560a59051570 2024-03-11T20:00:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.199+0000 7f2de1600640 1 -- 172.21.15.97:0/1378026229 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560a58f80cd0 con 0x560a59051570 2024-03-11T20:00:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.199+0000 7f2de1600640 1 -- 172.21.15.97:0/1378026229 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2df0025030 con 0x560a59051570 2024-03-11T20:00:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.199+0000 7f2e013de740 10 client.25648 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:00:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.199+0000 7f2e013de740 10 client.25648 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:00:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.199+0000 7f2e013de740 20 client.25648 populate_metadata read hostname 'smithi097' 2024-03-11T20:00:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.199+0000 7f2e013de740 1 --2- 172.21.15.97:0/1378026229 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x560a590614f0 0x560a590638e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.199+0000 7f2e013de740 4 client.25648 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T20:00:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.199+0000 7f2e013de740 1 -- 172.21.15.97:0/1378026229 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a58fa4ce0 con 0x560a590614f0 2024-03-11T20:00:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.200+0000 7f2df7400640 1 --2- 172.21.15.97:0/1378026229 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x560a590614f0 0x560a590638e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.200+0000 7f2de1600640 10 client.25648 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.200+0000 7f2df7400640 1 --2- 172.21.15.97:0/1378026229 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x560a590614f0 0x560a590638e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eae85218c92f0e5 server_cookie=5f71f94369810083 in_seq=0 out_seq=0 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.201+0000 7f2de1600640 1 -- 172.21.15.97:0/1378026229 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560a58fa4ce0 con 0x560a590614f0 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.201+0000 7f2de1600640 10 client.25648 handle_command_reply: tid=0 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.201+0000 7f2e013de740 1 client.25648 shutdown 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.201+0000 7f2e013de740 1 -- 172.21.15.97:0/1378026229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59051570 msgr2=0x560a5904f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.201+0000 7f2e013de740 1 --2- 172.21.15.97:0/1378026229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59051570 0x560a5904f040 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2df0014440 tx=0x7f2df0004ab0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.201+0000 7f2e013de740 1 -- 172.21.15.97:0/1378026229 shutdown_connections 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.201+0000 7f2e013de740 1 --2- 172.21.15.97:0/1378026229 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59051570 0x560a5904f040 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.201+0000 7f2e013de740 1 --2- 172.21.15.97:0/1378026229 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x560a590614f0 0x560a590638e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.201+0000 7f2e013de740 1 --2- 172.21.15.97:0/1378026229 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59048d60 0x560a5904eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.201+0000 7f2e013de740 1 --2- 172.21.15.97:0/1378026229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59047ff0 0x560a5904e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.201+0000 7f2e013de740 1 -- 172.21.15.97:0/1378026229 >> 172.21.15.97:0/1378026229 conn(0x560a5900f4b0 msgr2=0x560a59044a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.201+0000 7f2de1600640 0 client.25648 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.201+0000 7f2e013de740 1 -- 172.21.15.97:0/1378026229 shutdown_connections 2024-03-11T20:00:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.202+0000 7f2e013de740 1 -- 172.21.15.97:0/1378026229 wait complete. 2024-03-11T20:00:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.202+0000 7f2e013de740 20 client.25648 trim_cache size 0 max 16384 2024-03-11T20:00:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.228+0000 7f2e013de740 1 Processor -- start 2024-03-11T20:00:12.225 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.228+0000 7f2e013de740 1 -- start start 2024-03-11T20:00:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.229+0000 7f2e013de740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59047ff0 0x560a59050cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.229+0000 7f2e013de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59048d60 0x560a59058330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.229+0000 7f2e013de740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59051570 0x560a59058870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.229+0000 7f2e013de740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a58cf6e60 con 0x560a59047ff0 2024-03-11T20:00:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.229+0000 7f2e013de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a58f80d00 con 0x560a59048d60 2024-03-11T20:00:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.229+0000 7f2e013de740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a59047510 con 0x560a59051570 2024-03-11T20:00:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.229+0000 7f2df6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59048d60 0x560a59058330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.229+0000 7f2df7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59047ff0 0x560a59050cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.229+0000 7f2df6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59048d60 0x560a59058330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:45250/0 (socket says 172.21.15.97:45250) 2024-03-11T20:00:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.229+0000 7f2df6a00640 1 -- 172.21.15.97:0/2626629578 learned_addr learned my addr 172.21.15.97:0/2626629578 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.229+0000 7f2df7400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59047ff0 0x560a59050cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:37752/0 (socket says 172.21.15.97:37752) 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.229+0000 7f2df7e00640 1 --2- 172.21.15.97:0/2626629578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59051570 0x560a59058870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.230+0000 7f2df7e00640 1 -- 172.21.15.97:0/2626629578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59048d60 msgr2=0x560a59058330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.230+0000 7f2df7e00640 1 --2- 172.21.15.97:0/2626629578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59048d60 0x560a59058330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.230+0000 7f2df7e00640 1 -- 172.21.15.97:0/2626629578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59047ff0 msgr2=0x560a59050cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.230+0000 7f2df7e00640 1 --2- 172.21.15.97:0/2626629578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59047ff0 0x560a59050cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.230+0000 7f2df7e00640 1 -- 172.21.15.97:0/2626629578 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a58f49530 con 0x560a59051570 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.230+0000 7f2df6a00640 1 --2- 172.21.15.97:0/2626629578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59048d60 0x560a59058330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.230+0000 7f2df7e00640 1 --2- 172.21.15.97:0/2626629578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59051570 0x560a59058870 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2df0006010 tx=0x7f2df0000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=591ad52ffcda91df server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.230+0000 7f2de3e00640 1 -- 172.21.15.97:0/2626629578 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df002b070 con 0x560a59051570 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.230+0000 7f2de3e00640 1 -- 172.21.15.97:0/2626629578 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2df0011810 con 0x560a59051570 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.230+0000 7f2de3e00640 1 -- 172.21.15.97:0/2626629578 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0011d30 con 0x560a59051570 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.231+0000 7f2e013de740 1 -- 172.21.15.97:0/2626629578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59051570 msgr2=0x560a59058870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.231+0000 7f2e013de740 1 --2- 172.21.15.97:0/2626629578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59051570 0x560a59058870 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2df0006010 tx=0x7f2df0000f40 comp rx=0 tx=0).stop 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.231+0000 7f2e013de740 1 -- 172.21.15.97:0/2626629578 shutdown_connections 2024-03-11T20:00:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.231+0000 7f2e013de740 1 --2- 172.21.15.97:0/2626629578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59051570 0x560a59058870 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.231+0000 7f2e013de740 1 --2- 172.21.15.97:0/2626629578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59048d60 0x560a59058330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.231+0000 7f2e013de740 1 --2- 172.21.15.97:0/2626629578 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59047ff0 0x560a59050cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.231+0000 7f2e013de740 1 -- 172.21.15.97:0/2626629578 >> 172.21.15.97:0/2626629578 conn(0x560a5900f4b0 msgr2=0x560a5904ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.231+0000 7f2e013de740 1 -- 172.21.15.97:0/2626629578 shutdown_connections 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.231+0000 7f2e013de740 1 -- 172.21.15.97:0/2626629578 wait complete. 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.231+0000 7f2e013de740 1 -- start start 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.232+0000 7f2e013de740 1 Processor -- start 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.232+0000 7f2e013de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59047ff0 0x560a59058610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.232+0000 7f2e013de740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59048d60 0x560a5904ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.232+0000 7f2e013de740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59051570 0x560a5904b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.232+0000 7f2e013de740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5904ba90 con 0x560a59051570 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.232+0000 7f2e013de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5904bc00 con 0x560a59047ff0 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.232+0000 7f2e013de740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a590b79e0 con 0x560a59048d60 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.232+0000 7f2e013de740 10 client.0 _ll_register_callbacks cb 0x7f2df0009980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:00:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2df7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59047ff0 0x560a59058610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2df7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59047ff0 0x560a59058610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:45262/0 (socket says 172.21.15.97:45262) 2024-03-11T20:00:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2df7400640 1 -- 172.21.15.97:0/1131405162 learned_addr learned my addr 172.21.15.97:0/1131405162 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2df6a00640 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59048d60 0x560a5904ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2df7400640 1 -- 172.21.15.97:0/1131405162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59048d60 msgr2=0x560a5904ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2df7400640 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59048d60 0x560a5904ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2df7400640 1 -- 172.21.15.97:0/1131405162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59051570 msgr2=0x560a5904b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2df7e00640 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59051570 0x560a5904b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2df7400640 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59051570 0x560a5904b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2df7400640 1 -- 172.21.15.97:0/1131405162 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a58f2fb00 con 0x560a59047ff0 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2df7e00640 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59051570 0x560a5904b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2df7400640 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59047ff0 0x560a59058610 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2de800c760 tx=0x7f2de8096bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=375d3c13b6b255df server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2de1600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.233+0000 7f2de1600640 1 -- 172.21.15.97:0/1131405162 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8098c30 con 0x560a59047ff0 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.234+0000 7f2de1600640 1 -- 172.21.15.97:0/1131405162 <== mon.2 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f2de8096fa0 con 0x560a59047ff0 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.234+0000 7f2e013de740 1 -- 172.21.15.97:0/1131405162 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560a58fd3d30 con 0x560a59047ff0 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.234+0000 7f2de1600640 1 -- 172.21.15.97:0/1131405162 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2de8097160 con 0x560a59047ff0 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.234+0000 7f2e013de740 10 client.25661 fetch_fsmap learned FSMap version 29 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.234+0000 7f2e013de740 1 -- 172.21.15.97:0/1131405162 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x560a58facc50 con 0x560a59047ff0 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.234+0000 7f2de1600640 1 -- 172.21.15.97:0/1131405162 <== mon.2 v2:172.21.15.114:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560a58fd3d30 con 0x560a59047ff0 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.234+0000 7f2de1600640 1 -- 172.21.15.97:0/1131405162 <== mon.2 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2de80cb890 con 0x560a59047ff0 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.234+0000 7f2e013de740 10 client.25661 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.234+0000 7f2e013de740 10 client.25661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.234+0000 7f2e013de740 20 client.25661 populate_metadata read hostname 'smithi097' 2024-03-11T20:00:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.234+0000 7f2e013de740 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x560a590b9070 0x560a590bb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.234+0000 7f2e013de740 4 client.25661 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T20:00:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.234+0000 7f2e013de740 1 -- 172.21.15.97:0/1131405162 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560a58cae600 con 0x560a590b9070 2024-03-11T20:00:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.235+0000 7f2df6a00640 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x560a590b9070 0x560a590bb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.236+0000 7f2de1600640 10 client.25661 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.236+0000 7f2df6a00640 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x560a590b9070 0x560a590bb460 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e5a8f185d68b06f server_cookie=c3e3eead4308de6e in_seq=0 out_seq=0 2024-03-11T20:00:12.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:11 smithi114 ceph-mon[36376]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-11T20:00:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.242+0000 7f2de1600640 1 -- 172.21.15.97:0/1131405162 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560a58cae600 con 0x560a590b9070 2024-03-11T20:00:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.242+0000 7f2de1600640 10 client.25661 handle_command_reply: tid=0 2024-03-11T20:00:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.242+0000 7f2e013de740 1 client.25661 shutdown 2024-03-11T20:00:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.242+0000 7f2e013de740 1 -- 172.21.15.97:0/1131405162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59047ff0 msgr2=0x560a59058610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.242+0000 7f2e013de740 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59047ff0 0x560a59058610 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2de800c760 tx=0x7f2de8096bd0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.242+0000 7f2e013de740 1 -- 172.21.15.97:0/1131405162 shutdown_connections 2024-03-11T20:00:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.242+0000 7f2e013de740 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x560a590b9070 0x560a590bb460 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.242+0000 7f2e013de740 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x560a59051570 0x560a5904b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.242+0000 7f2e013de740 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x560a59048d60 0x560a5904ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.242+0000 7f2e013de740 1 --2- 172.21.15.97:0/1131405162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560a59047ff0 0x560a59058610 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.242+0000 7f2e013de740 1 -- 172.21.15.97:0/1131405162 >> 172.21.15.97:0/1131405162 conn(0x560a5900f4b0 msgr2=0x560a59041d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:12.239 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.242+0000 7f2de1600640 0 client.25661 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.243+0000 7f2e013de740 1 -- 172.21.15.97:0/1131405162 shutdown_connections 2024-03-11T20:00:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.243+0000 7f2e013de740 1 -- 172.21.15.97:0/1131405162 wait complete. 2024-03-11T20:00:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.243+0000 7f2e013de740 20 client.25661 trim_cache size 0 max 16384 2024-03-11T20:00:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-03-11T20:00:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-03-11T20:00:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-03-11T20:00:12.240 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-03-11T20:00:12.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.243+0000 7f2dfd200640 1 -- 172.21.15.97:0/850790291 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f2ddc0a1d30 msgr2=0x7f2ddc0a41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:12.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.243+0000 7f2dfd200640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f2ddc0a1d30 0x7f2ddc0a41f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2df0005c40 tx=0x7f2df0014950 comp rx=0 tx=0).stop 2024-03-11T20:00:12.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.243+0000 7f2dfd200640 1 -- 172.21.15.97:0/850790291 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2df80fff20 msgr2=0x7f2df81a3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:12.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.243+0000 7f2dfd200640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2df80fff20 0x7f2df81a3880 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2de800ea60 tx=0x7f2de800ef20 comp rx=0 tx=0).stop 2024-03-11T20:00:12.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.244+0000 7f2dfd200640 1 -- 172.21.15.97:0/850790291 shutdown_connections 2024-03-11T20:00:12.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.244+0000 7f2dfd200640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f2ddc0a1d30 0x7f2ddc0a41f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.244+0000 7f2dfd200640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2df8105320 0x7f2df819da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.244+0000 7f2dfd200640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2df8100840 0x7f2df81a3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.244+0000 7f2dfd200640 1 --2- 172.21.15.97:0/850790291 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2df80fff20 0x7f2df81a3880 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:12.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.244+0000 7f2dfd200640 1 -- 172.21.15.97:0/850790291 >> 172.21.15.97:0/850790291 conn(0x7f2df80fb990 msgr2=0x7f2df80fca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:12.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.244+0000 7f2dfd200640 1 -- 172.21.15.97:0/850790291 shutdown_connections 2024-03-11T20:00:12.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:12.244+0000 7f2dfd200640 1 -- 172.21.15.97:0/850790291 wait complete. 2024-03-11T20:00:12.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:11 smithi161 ceph-mon[36332]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-11T20:00:14.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:13 smithi097 ceph-mon[31898]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-03-11T20:00:14.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:13 smithi114 ceph-mon[36376]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-03-11T20:00:14.282 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:13 smithi161 ceph-mon[36332]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-03-11T20:00:16.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:15 smithi097 ceph-mon[31898]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 24 op/s 2024-03-11T20:00:16.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:15 smithi114 ceph-mon[36376]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 24 op/s 2024-03-11T20:00:16.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:15 smithi161 ceph-mon[36332]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 24 op/s 2024-03-11T20:00:18.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:17 smithi097 ceph-mon[31898]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-11T20:00:18.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:17 smithi114 ceph-mon[36376]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-11T20:00:18.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:17 smithi161 ceph-mon[36332]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-11T20:00:20.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:19 smithi097 ceph-mon[31898]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-11T20:00:20.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:19 smithi114 ceph-mon[36376]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-11T20:00:20.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:19 smithi161 ceph-mon[36332]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-11T20:00:22.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:21 smithi097 ceph-mon[31898]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-11T20:00:22.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:21 smithi114 ceph-mon[36376]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-11T20:00:22.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:21 smithi161 ceph-mon[36332]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-11T20:00:24.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:23 smithi097 ceph-mon[31898]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-11T20:00:24.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:23 smithi114 ceph-mon[36376]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-11T20:00:24.282 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:23 smithi161 ceph-mon[36332]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-11T20:00:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.422+0000 7fe45a600640 1 -- 172.21.15.97:0/4155168784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4541035c0 msgr2=0x7fe4541039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.422+0000 7fe45a600640 1 --2- 172.21.15.97:0/4155168784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4541035c0 0x7fe4541039a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe444009930 tx=0x7fe44402bc50 comp rx=0 tx=0).stop 2024-03-11T20:00:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.422+0000 7fe45a600640 1 -- 172.21.15.97:0/4155168784 shutdown_connections 2024-03-11T20:00:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.422+0000 7fe45a600640 1 --2- 172.21.15.97:0/4155168784 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe454108690 0x7fe454110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.422+0000 7fe45a600640 1 --2- 172.21.15.97:0/4155168784 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe454103f70 0x7fe454107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.422+0000 7fe45a600640 1 --2- 172.21.15.97:0/4155168784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4541035c0 0x7fe4541039a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.422+0000 7fe45a600640 1 -- 172.21.15.97:0/4155168784 >> 172.21.15.97:0/4155168784 conn(0x7fe4540fee40 msgr2=0x7fe454101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.422+0000 7fe45a600640 1 -- 172.21.15.97:0/4155168784 shutdown_connections 2024-03-11T20:00:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.422+0000 7fe45a600640 1 -- 172.21.15.97:0/4155168784 wait complete. 2024-03-11T20:00:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.424+0000 7fe45a600640 1 Processor -- start 2024-03-11T20:00:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.424+0000 7fe45a600640 1 -- start start 2024-03-11T20:00:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.424+0000 7fe45a600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4541035c0 0x7fe45419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.424+0000 7fe45a600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe454108690 0x7fe45419fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.424+0000 7fe45a600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4541998e0 0x7fe454199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.424+0000 7fe45a600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45419a2d0 con 0x7fe454108690 2024-03-11T20:00:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.424+0000 7fe45a600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45419a440 con 0x7fe4541035c0 2024-03-11T20:00:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.424+0000 7fe45a600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45419a600 con 0x7fe4541998e0 2024-03-11T20:00:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.425+0000 7fe452a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe454108690 0x7fe45419fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.425+0000 7fe453e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4541998e0 0x7fe454199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.425+0000 7fe453e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4541998e0 0x7fe454199d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:36286/0 (socket says 172.21.15.97:36286) 2024-03-11T20:00:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.425+0000 7fe453e00640 1 -- 172.21.15.97:0/1164469704 learned_addr learned my addr 172.21.15.97:0/1164469704 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.426+0000 7fe453400640 1 --2- 172.21.15.97:0/1164469704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4541035c0 0x7fe45419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.426+0000 7fe452a00640 1 -- 172.21.15.97:0/1164469704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4541998e0 msgr2=0x7fe454199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:25.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.426+0000 7fe452a00640 1 --2- 172.21.15.97:0/1164469704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4541998e0 0x7fe454199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.426+0000 7fe452a00640 1 -- 172.21.15.97:0/1164469704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4541035c0 msgr2=0x7fe45419f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:25.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.426+0000 7fe452a00640 1 --2- 172.21.15.97:0/1164469704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4541035c0 0x7fe45419f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.423 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.426+0000 7fe452a00640 1 -- 172.21.15.97:0/1164469704 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe444009560 con 0x7fe454108690 2024-03-11T20:00:25.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.426+0000 7fe452a00640 1 --2- 172.21.15.97:0/1164469704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe454108690 0x7fe45419fc20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe44000e9c0 tx=0x7fe44000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:25.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.427+0000 7fe44be00640 1 -- 172.21.15.97:0/1164469704 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44000ccc0 con 0x7fe454108690 2024-03-11T20:00:25.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.427+0000 7fe45a600640 1 -- 172.21.15.97:0/1164469704 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4541a65a0 con 0x7fe454108690 2024-03-11T20:00:25.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.427+0000 7fe44be00640 1 -- 172.21.15.97:0/1164469704 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe44000ce20 con 0x7fe454108690 2024-03-11T20:00:25.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.427+0000 7fe44be00640 1 -- 172.21.15.97:0/1164469704 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe440018880 con 0x7fe454108690 2024-03-11T20:00:25.424 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.427+0000 7fe45a600640 1 -- 172.21.15.97:0/1164469704 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4541a6b00 con 0x7fe454108690 2024-03-11T20:00:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.431+0000 7fe45d384740 1 Processor -- start 2024-03-11T20:00:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.432+0000 7fe45d384740 1 -- start start 2024-03-11T20:00:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.432+0000 7fe45d384740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456d8d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.432+0000 7fe45d384740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d8ddb0 0x563456d8e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.432+0000 7fe45d384740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d97330 0x563456d99720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.432+0000 7fe45d384740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x563456d49620 con 0x563456d8ddb0 2024-03-11T20:00:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.432+0000 7fe45d384740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x563456cfb8c0 con 0x563456d8eb20 2024-03-11T20:00:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.432+0000 7fe45d384740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x563456cf8df0 con 0x563456d97330 2024-03-11T20:00:25.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.433+0000 7fe453e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456d8d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.433+0000 7fe453e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456d8d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:59898/0 (socket says 172.21.15.97:59898) 2024-03-11T20:00:25.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.433+0000 7fe453e00640 1 -- 172.21.15.97:0/2484446172 learned_addr learned my addr 172.21.15.97:0/2484446172 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:25.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.433+0000 7fe453400640 1 --2- 172.21.15.97:0/2484446172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d8ddb0 0x563456d8e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.433+0000 7fe453e00640 1 -- 172.21.15.97:0/2484446172 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d97330 msgr2=0x563456d99720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T20:00:25.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.433+0000 7fe453e00640 1 --2- 172.21.15.97:0/2484446172 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d97330 0x563456d99720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.433+0000 7fe453e00640 1 -- 172.21.15.97:0/2484446172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d8ddb0 msgr2=0x563456d8e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:25.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.433+0000 7fe453e00640 1 --2- 172.21.15.97:0/2484446172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d8ddb0 0x563456d8e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.430 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.433+0000 7fe453e00640 1 -- 172.21.15.97:0/2484446172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563456c75b00 con 0x563456d8eb20 2024-03-11T20:00:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.434+0000 7fe44be00640 1 -- 172.21.15.97:0/1164469704 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fe4400189e0 con 0x7fe454108690 2024-03-11T20:00:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.434+0000 7fe453e00640 1 --2- 172.21.15.97:0/2484446172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456d8d870 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00e270 tx=0x7fe44c009710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc0f311de289fee3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.434+0000 7fe448c00640 1 -- 172.21.15.97:0/2484446172 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c014890 con 0x563456d8eb20 2024-03-11T20:00:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.434+0000 7fe448c00640 1 -- 172.21.15.97:0/2484446172 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe44c00f040 con 0x563456d8eb20 2024-03-11T20:00:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.434+0000 7fe448c00640 1 -- 172.21.15.97:0/2484446172 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c013db0 con 0x563456d8eb20 2024-03-11T20:00:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.435+0000 7fe45d384740 1 -- 172.21.15.97:0/2484446172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 msgr2=0x563456d8d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.435+0000 7fe45d384740 1 --2- 172.21.15.97:0/2484446172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456d8d870 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00e270 tx=0x7fe44c009710 comp rx=0 tx=0).stop 2024-03-11T20:00:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.435+0000 7fe45d384740 1 -- 172.21.15.97:0/2484446172 shutdown_connections 2024-03-11T20:00:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.435+0000 7fe45d384740 1 --2- 172.21.15.97:0/2484446172 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d97330 0x563456d99720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.435+0000 7fe45d384740 1 --2- 172.21.15.97:0/2484446172 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d8ddb0 0x563456d8e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.435+0000 7fe45d384740 1 --2- 172.21.15.97:0/2484446172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456d8d870 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.435+0000 7fe45d384740 1 -- 172.21.15.97:0/2484446172 >> 172.21.15.97:0/2484446172 conn(0x563456d554b0 msgr2=0x563456d88aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.436+0000 7fe45d384740 1 -- 172.21.15.97:0/2484446172 shutdown_connections 2024-03-11T20:00:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.436+0000 7fe45d384740 1 -- 172.21.15.97:0/2484446172 wait complete. 2024-03-11T20:00:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.436+0000 7fe44be00640 1 --2- 172.21.15.97:0/1164469704 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe4380a1cf0 0x7fe4380a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.436+0000 7fe44be00640 1 -- 172.21.15.97:0/1164469704 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7fe440014070 con 0x7fe454108690 2024-03-11T20:00:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.436+0000 7fe45d384740 1 -- start start 2024-03-11T20:00:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.437+0000 7fe453400640 1 --2- 172.21.15.97:0/1164469704 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe4380a1cf0 0x7fe4380a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.437+0000 7fe45d384740 1 Processor -- start 2024-03-11T20:00:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.438+0000 7fe45d384740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8ddb0 0x563456d94350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.438+0000 7fe45d384740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456d94890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.438+0000 7fe45d384740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d97330 0x563456d94dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.438+0000 7fe45d384740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x563456cf45d0 con 0x563456d97330 2024-03-11T20:00:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.438+0000 7fe45d384740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x563456c99b50 con 0x563456d8eb20 2024-03-11T20:00:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.438+0000 7fe45d384740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x563456c997f0 con 0x563456d8ddb0 2024-03-11T20:00:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.438+0000 7fe45d384740 10 client.0 _ll_register_callbacks cb 0x563456cad550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:00:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.438+0000 7fe453400640 1 --2- 172.21.15.97:0/1164469704 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe4380a1cf0 0x7fe4380a41b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x563456d8f280 tx=0x7fe444005a00 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:25.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.438+0000 7fe452a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456d94890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.438+0000 7fe453400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8ddb0 0x563456d94350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.438+0000 7fe452a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456d94890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:59908/0 (socket says 172.21.15.97:59908) 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.438+0000 7fe452a00640 1 -- 172.21.15.97:0/2631600261 learned_addr learned my addr 172.21.15.97:0/2631600261 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.438+0000 7fe453e00640 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d97330 0x563456d94dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.439+0000 7fe452a00640 1 -- 172.21.15.97:0/2631600261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8ddb0 msgr2=0x563456d94350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.439+0000 7fe452a00640 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8ddb0 0x563456d94350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.439+0000 7fe452a00640 1 -- 172.21.15.97:0/2631600261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d97330 msgr2=0x563456d94dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.439+0000 7fe452a00640 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d97330 0x563456d94dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.439+0000 7fe452a00640 1 -- 172.21.15.97:0/2631600261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44c009dc0 con 0x563456d8eb20 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.439+0000 7fe453400640 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8ddb0 0x563456d94350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.439+0000 7fe453e00640 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d97330 0x563456d94dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.439+0000 7fe452a00640 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456d94890 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe44000c6a0 tx=0x7fe44009a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccaad22d92f16c14 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.439+0000 7fe43d600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.439+0000 7fe43d600640 1 -- 172.21.15.97:0/2631600261 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe440085300 con 0x563456d8eb20 2024-03-11T20:00:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.439+0000 7fe43d600640 1 -- 172.21.15.97:0/2631600261 <== mon.2 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7fe4400a2e30 con 0x563456d8eb20 2024-03-11T20:00:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.439+0000 7fe45d384740 1 -- 172.21.15.97:0/2631600261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563456cc6cd0 con 0x563456d8eb20 2024-03-11T20:00:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.440+0000 7fe43d600640 1 -- 172.21.15.97:0/2631600261 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4400b30d0 con 0x563456d8eb20 2024-03-11T20:00:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.440+0000 7fe45d384740 10 client.25670 fetch_fsmap learned FSMap version 29 2024-03-11T20:00:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.440+0000 7fe45d384740 1 -- 172.21.15.97:0/2631600261 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x563456cf2c50 con 0x563456d8eb20 2024-03-11T20:00:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.440+0000 7fe43d600640 1 -- 172.21.15.97:0/2631600261 <== mon.2 v2:172.21.15.114:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563456cc6cd0 con 0x563456d8eb20 2024-03-11T20:00:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.441+0000 7fe43d600640 1 -- 172.21.15.97:0/2631600261 <== mon.2 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe440099ae0 con 0x563456d8eb20 2024-03-11T20:00:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.441+0000 7fe45d384740 10 client.25670 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:00:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.441+0000 7fe45d384740 10 client.25670 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:00:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.441+0000 7fe45d384740 20 client.25670 populate_metadata read hostname 'smithi097' 2024-03-11T20:00:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.441+0000 7fe45d384740 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563456dafbe0 0x563456db2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.441+0000 7fe45d384740 4 client.25670 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T20:00:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.441+0000 7fe45d384740 1 -- 172.21.15.97:0/2631600261 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563456ceace0 con 0x563456dafbe0 2024-03-11T20:00:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.442+0000 7fe453400640 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563456dafbe0 0x563456db2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.442+0000 7fe43d600640 10 client.25670 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.442+0000 7fe453400640 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563456dafbe0 0x563456db2000 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd7bf8afb61bc906 server_cookie=65d6f84af133d763 in_seq=0 out_seq=0 2024-03-11T20:00:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.443+0000 7fe43d600640 1 -- 172.21.15.97:0/2631600261 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563456ceace0 con 0x563456dafbe0 2024-03-11T20:00:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.443+0000 7fe43d600640 10 client.25670 handle_command_reply: tid=0 2024-03-11T20:00:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.443+0000 7fe45d384740 1 client.25670 shutdown 2024-03-11T20:00:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.443+0000 7fe45d384740 1 -- 172.21.15.97:0/2631600261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 msgr2=0x563456d94890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.443+0000 7fe45d384740 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456d94890 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe44000c6a0 tx=0x7fe44009a0f0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.444+0000 7fe45d384740 1 -- 172.21.15.97:0/2631600261 shutdown_connections 2024-03-11T20:00:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.444+0000 7fe45d384740 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563456dafbe0 0x563456db2000 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.444+0000 7fe45d384740 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d97330 0x563456d94dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.444+0000 7fe43d600640 0 client.25670 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:25.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.444+0000 7fe45d384740 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456d94890 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.444+0000 7fe45d384740 1 --2- 172.21.15.97:0/2631600261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8ddb0 0x563456d94350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.444+0000 7fe45d384740 1 -- 172.21.15.97:0/2631600261 >> 172.21.15.97:0/2631600261 conn(0x563456d554b0 msgr2=0x563456d8a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:25.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.444+0000 7fe45d384740 1 -- 172.21.15.97:0/2631600261 shutdown_connections 2024-03-11T20:00:25.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.444+0000 7fe45d384740 1 -- 172.21.15.97:0/2631600261 wait complete. 2024-03-11T20:00:25.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.444+0000 7fe45d384740 20 client.25670 trim_cache size 0 max 16384 2024-03-11T20:00:25.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.471+0000 7fe45d384740 1 Processor -- start 2024-03-11T20:00:25.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.471+0000 7fe45d384740 1 -- start start 2024-03-11T20:00:25.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.471+0000 7fe45d384740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d8ddb0 0x563456d96a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.471+0000 7fe45d384740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8eb20 0x563456d915d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.471+0000 7fe45d384740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d97330 0x563456d91b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.472+0000 7fe45d384740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x563456a3ce60 con 0x563456d8ddb0 2024-03-11T20:00:25.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.472+0000 7fe45d384740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x563456d84370 con 0x563456d97330 2024-03-11T20:00:25.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.472+0000 7fe45d384740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x563456d8ff20 con 0x563456d8eb20 2024-03-11T20:00:25.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.472+0000 7fe453400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d8ddb0 0x563456d96a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.472+0000 7fe452a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8eb20 0x563456d915d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.472+0000 7fe453400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d8ddb0 0x563456d96a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:57970/0 (socket says 172.21.15.97:57970) 2024-03-11T20:00:25.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.472+0000 7fe453400640 1 -- 172.21.15.97:0/1171206052 learned_addr learned my addr 172.21.15.97:0/1171206052 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:25.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.472+0000 7fe453e00640 1 --2- 172.21.15.97:0/1171206052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d97330 0x563456d91b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.472+0000 7fe453e00640 1 -- 172.21.15.97:0/1171206052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8eb20 msgr2=0x563456d915d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:25.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.472+0000 7fe453e00640 1 --2- 172.21.15.97:0/1171206052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8eb20 0x563456d915d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.472+0000 7fe453e00640 1 -- 172.21.15.97:0/1171206052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d8ddb0 msgr2=0x563456d96a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:25.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.473+0000 7fe453e00640 1 --2- 172.21.15.97:0/1171206052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d8ddb0 0x563456d96a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.473+0000 7fe453e00640 1 -- 172.21.15.97:0/1171206052 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563456c8f530 con 0x563456d97330 2024-03-11T20:00:25.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.473+0000 7fe453400640 1 --2- 172.21.15.97:0/1171206052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d8ddb0 0x563456d96a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:00:25.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.473+0000 7fe453e00640 1 --2- 172.21.15.97:0/1171206052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d97330 0x563456d91b10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00e240 tx=0x7fe44c003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb7ff5130d695431 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:25.471 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.473+0000 7fe43fe00640 1 -- 172.21.15.97:0/1171206052 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c003e80 con 0x563456d97330 2024-03-11T20:00:25.471 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.473+0000 7fe43fe00640 1 -- 172.21.15.97:0/1171206052 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe44c014b70 con 0x563456d97330 2024-03-11T20:00:25.471 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.473+0000 7fe43fe00640 1 -- 172.21.15.97:0/1171206052 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c004990 con 0x563456d97330 2024-03-11T20:00:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.474+0000 7fe45d384740 1 -- 172.21.15.97:0/1171206052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d97330 msgr2=0x563456d91b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.474+0000 7fe45d384740 1 --2- 172.21.15.97:0/1171206052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d97330 0x563456d91b10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00e240 tx=0x7fe44c003a90 comp rx=0 tx=0).stop 2024-03-11T20:00:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.475+0000 7fe45d384740 1 -- 172.21.15.97:0/1171206052 shutdown_connections 2024-03-11T20:00:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.475+0000 7fe45d384740 1 --2- 172.21.15.97:0/1171206052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d97330 0x563456d91b10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.475+0000 7fe45d384740 1 --2- 172.21.15.97:0/1171206052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8eb20 0x563456d915d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.475+0000 7fe45d384740 1 --2- 172.21.15.97:0/1171206052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d8ddb0 0x563456d96a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.475+0000 7fe45d384740 1 -- 172.21.15.97:0/1171206052 >> 172.21.15.97:0/1171206052 conn(0x563456d554b0 msgr2=0x563456d92c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.475+0000 7fe45d384740 1 -- 172.21.15.97:0/1171206052 shutdown_connections 2024-03-11T20:00:25.472 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.475+0000 7fe45d384740 1 -- 172.21.15.97:0/1171206052 wait complete. 2024-03-11T20:00:25.473 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.476+0000 7fe45d384740 1 -- start start 2024-03-11T20:00:25.473 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.476+0000 7fe45d384740 1 Processor -- start 2024-03-11T20:00:25.473 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.477+0000 7fe45d384740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8ddb0 0x563456da5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.477+0000 7fe45d384740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456da6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.477+0000 7fe45d384740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d97330 0x563456e000f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.477+0000 7fe45d384740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x563456e00630 con 0x563456d97330 2024-03-11T20:00:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.477+0000 7fe45d384740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x563456e007a0 con 0x563456d8eb20 2024-03-11T20:00:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.477+0000 7fe45d384740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x563456e00960 con 0x563456d8ddb0 2024-03-11T20:00:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.477+0000 7fe45d384740 10 client.0 _ll_register_callbacks cb 0x7fe440096930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:00:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.477+0000 7fe453400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8ddb0 0x563456da5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.477+0000 7fe453400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8ddb0 0x563456da5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:36316/0 (socket says 172.21.15.97:36316) 2024-03-11T20:00:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.477+0000 7fe452a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456da6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.477+0000 7fe453400640 1 -- 172.21.15.97:0/1502719579 learned_addr learned my addr 172.21.15.97:0/1502719579 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.478+0000 7fe453400640 1 -- 172.21.15.97:0/1502719579 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 msgr2=0x563456da6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.478+0000 7fe453400640 1 --2- 172.21.15.97:0/1502719579 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456da6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.478+0000 7fe453400640 1 -- 172.21.15.97:0/1502719579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d97330 msgr2=0x563456e000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.478+0000 7fe453400640 1 --2- 172.21.15.97:0/1502719579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d97330 0x563456e000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.478+0000 7fe453400640 1 -- 172.21.15.97:0/1502719579 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe454110a70 con 0x563456d8ddb0 2024-03-11T20:00:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.478+0000 7fe452a00640 1 --2- 172.21.15.97:0/1502719579 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456da6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:00:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.478+0000 7fe453400640 1 --2- 172.21.15.97:0/1502719579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8ddb0 0x563456da5cd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe444009900 tx=0x7fe44402e050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b96a4768fb5924c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.478+0000 7fe43d600640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-11T20:00:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.478+0000 7fe43d600640 1 -- 172.21.15.97:0/1502719579 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444004450 con 0x563456d8ddb0 2024-03-11T20:00:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.478+0000 7fe43d600640 1 -- 172.21.15.97:0/1502719579 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7fe444004df0 con 0x563456d8ddb0 2024-03-11T20:00:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.479+0000 7fe45d384740 1 -- 172.21.15.97:0/1502719579 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563456d19d30 con 0x563456d8ddb0 2024-03-11T20:00:25.476 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.479+0000 7fe43d600640 1 -- 172.21.15.97:0/1502719579 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe444004fb0 con 0x563456d8ddb0 2024-03-11T20:00:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.479+0000 7fe45d384740 10 client.25666 fetch_fsmap learned FSMap version 29 2024-03-11T20:00:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.479+0000 7fe45d384740 1 -- 172.21.15.97:0/1502719579 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x563456c75b00 con 0x563456d8ddb0 2024-03-11T20:00:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.479+0000 7fe43d600640 1 -- 172.21.15.97:0/1502719579 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563456d19d30 con 0x563456d8ddb0 2024-03-11T20:00:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.479+0000 7fe43d600640 1 -- 172.21.15.97:0/1502719579 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe44403c040 con 0x563456d8ddb0 2024-03-11T20:00:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.480+0000 7fe45d384740 10 client.25666 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:00:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.480+0000 7fe45d384740 10 client.25666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:00:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.480+0000 7fe45d384740 20 client.25666 populate_metadata read hostname 'smithi097' 2024-03-11T20:00:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.480+0000 7fe45d384740 1 --2- 172.21.15.97:0/1502719579 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563456e08200 0x563456e0a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:25.478 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.480+0000 7fe45d384740 4 client.25666 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T20:00:25.478 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.480+0000 7fe45d384740 1 -- 172.21.15.97:0/1502719579 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5634569f4600 con 0x563456e08200 2024-03-11T20:00:25.478 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.480+0000 7fe452a00640 1 --2- 172.21.15.97:0/1502719579 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563456e08200 0x563456e0a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:25.478 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.480+0000 7fe43d600640 10 client.25666 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:25.478 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.481+0000 7fe452a00640 1 --2- 172.21.15.97:0/1502719579 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563456e08200 0x563456e0a640 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b54365c9e2949183 server_cookie=1e6668c7b13902a6 in_seq=0 out_seq=0 2024-03-11T20:00:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.486+0000 7fe43d600640 1 -- 172.21.15.97:0/1502719579 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5634569f4600 con 0x563456e08200 2024-03-11T20:00:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.486+0000 7fe43d600640 10 client.25666 handle_command_reply: tid=0 2024-03-11T20:00:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.486+0000 7fe45d384740 1 client.25666 shutdown 2024-03-11T20:00:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.486+0000 7fe45d384740 1 -- 172.21.15.97:0/1502719579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8ddb0 msgr2=0x563456da5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.486+0000 7fe45d384740 1 --2- 172.21.15.97:0/1502719579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8ddb0 0x563456da5cd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe444009900 tx=0x7fe44402e050 comp rx=0 tx=0).stop 2024-03-11T20:00:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.487+0000 7fe45d384740 1 -- 172.21.15.97:0/1502719579 shutdown_connections 2024-03-11T20:00:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.487+0000 7fe45d384740 1 --2- 172.21.15.97:0/1502719579 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x563456e08200 0x563456e0a640 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.487+0000 7fe45d384740 1 --2- 172.21.15.97:0/1502719579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x563456d97330 0x563456e000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.487+0000 7fe45d384740 1 --2- 172.21.15.97:0/1502719579 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563456d8eb20 0x563456da6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.487+0000 7fe45d384740 1 --2- 172.21.15.97:0/1502719579 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563456d8ddb0 0x563456da5cd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.487+0000 7fe45d384740 1 -- 172.21.15.97:0/1502719579 >> 172.21.15.97:0/1502719579 conn(0x563456d554b0 msgr2=0x563456d89ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.487+0000 7fe43d600640 0 client.25666 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:25.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.487+0000 7fe45d384740 1 -- 172.21.15.97:0/1502719579 shutdown_connections 2024-03-11T20:00:25.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.487+0000 7fe45d384740 1 -- 172.21.15.97:0/1502719579 wait complete. 2024-03-11T20:00:25.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.488+0000 7fe45d384740 20 client.25666 trim_cache size 0 max 16384 2024-03-11T20:00:25.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-03-11T20:00:25.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-03-11T20:00:25.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-03-11T20:00:25.485 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-03-11T20:00:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.488+0000 7fe45a600640 1 -- 172.21.15.97:0/1164469704 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe4380a1cf0 msgr2=0x7fe4380a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.488+0000 7fe45a600640 1 --2- 172.21.15.97:0/1164469704 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe4380a1cf0 0x7fe4380a41b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x563456d8f280 tx=0x7fe444005a00 comp rx=0 tx=0).stop 2024-03-11T20:00:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.488+0000 7fe45a600640 1 -- 172.21.15.97:0/1164469704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe454108690 msgr2=0x7fe45419fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.488+0000 7fe45a600640 1 --2- 172.21.15.97:0/1164469704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe454108690 0x7fe45419fc20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe44000e9c0 tx=0x7fe44000ee80 comp rx=0 tx=0).stop 2024-03-11T20:00:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.489+0000 7fe45a600640 1 -- 172.21.15.97:0/1164469704 shutdown_connections 2024-03-11T20:00:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.489+0000 7fe45a600640 1 --2- 172.21.15.97:0/1164469704 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe4380a1cf0 0x7fe4380a41b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.489+0000 7fe45a600640 1 --2- 172.21.15.97:0/1164469704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4541998e0 0x7fe454199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.489+0000 7fe45a600640 1 --2- 172.21.15.97:0/1164469704 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe454108690 0x7fe45419fc20 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.489+0000 7fe45a600640 1 --2- 172.21.15.97:0/1164469704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4541035c0 0x7fe45419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.489+0000 7fe45a600640 1 -- 172.21.15.97:0/1164469704 >> 172.21.15.97:0/1164469704 conn(0x7fe4540fee40 msgr2=0x7fe454101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.489+0000 7fe45a600640 1 -- 172.21.15.97:0/1164469704 shutdown_connections 2024-03-11T20:00:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:25.489+0000 7fe45a600640 1 -- 172.21.15.97:0/1164469704 wait complete. 2024-03-11T20:00:26.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:25 smithi097 ceph-mon[31898]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T20:00:26.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:25 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:00:26.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:25 smithi114 ceph-mon[36376]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T20:00:26.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:25 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:00:26.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:25 smithi161 ceph-mon[36332]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-11T20:00:26.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:25 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:00:27.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:26 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:00:27.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:26 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:00:27.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:26 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:00:27.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:26 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:00:27.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:26 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:00:27.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:26 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:00:27.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:26 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:00:27.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:26 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:00:27.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:26 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:00:28.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:27 smithi097 ceph-mon[31898]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:00:28.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:27 smithi114 ceph-mon[36376]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:00:28.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:27 smithi161 ceph-mon[36332]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:00:30.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:29 smithi097 ceph-mon[31898]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 341 B/s wr, 13 op/s 2024-03-11T20:00:30.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:29 smithi114 ceph-mon[36376]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 341 B/s wr, 13 op/s 2024-03-11T20:00:30.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:29 smithi161 ceph-mon[36332]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 341 B/s wr, 13 op/s 2024-03-11T20:00:31.671 INFO:tasks.workunit.client.0.smithi097.stderr:Updating files: 64% (7947/12344) Updating files: 65% (8024/12344) Updating files: 66% (8148/12344) Updating files: 67% (8271/12344) Updating files: 68% (8394/12344) Updating files: 69% (8518/12344) Updating files: 70% (8641/12344) Updating files: 71% (8765/12344) Updating files: 72% (8888/12344) Updating files: 73% (9012/12344) Updating files: 74% (9135/12344) Updating files: 75% (9258/12344) Updating files: 76% (9382/12344) Updating files: 77% (9505/12344) Updating files: 78% (9629/12344) Updating files: 79% (9752/12344) Updating files: 80% (9876/12344) Updating files: 81% (9999/12344) Updating files: 82% (10123/12344) Updating files: 83% (10246/12344) Updating files: 84% (10369/12344) Updating files: 85% (10493/12344) Updating files: 86% (10616/12344) Updating files: 87% (10740/12344) Updating files: 88% (10863/12344) Updating files: 89% (10987/12344) Updating files: 90% (11110/12344) Updating files: 91% (11234/12344) Updating files: 92% (11357/12344) Updating files: 93% (11480/12344) Updating files: 94% (11604/12344) Updating files: 95% (11727/12344) Updating files: 96% (11851/12344) Updating files: 97% (11974/12344) Updating files: 98% (12098/12344) Updating files: 99% (12221/12344) Updating files: 100% (12344/12344) Updating files: 100% (12344/12344), done. 2024-03-11T20:00:32.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:31 smithi097 ceph-mon[31898]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-11T20:00:32.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:31 smithi114 ceph-mon[36376]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-11T20:00:32.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:31 smithi161 ceph-mon[36332]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-11T20:00:32.324 INFO:tasks.workunit.client.0.smithi097.stderr:Note: switching to 'b6a4997be2b8bb74411268cc42753f29ae9a78ad'. 2024-03-11T20:00:32.324 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-03-11T20:00:32.324 INFO:tasks.workunit.client.0.smithi097.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-11T20:00:32.324 INFO:tasks.workunit.client.0.smithi097.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-11T20:00:32.324 INFO:tasks.workunit.client.0.smithi097.stderr:state without impacting any branches by switching back to a branch. 2024-03-11T20:00:32.324 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-03-11T20:00:32.324 INFO:tasks.workunit.client.0.smithi097.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-11T20:00:32.324 INFO:tasks.workunit.client.0.smithi097.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-11T20:00:32.325 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-03-11T20:00:32.325 INFO:tasks.workunit.client.0.smithi097.stderr: git switch -c 2024-03-11T20:00:32.325 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-03-11T20:00:32.325 INFO:tasks.workunit.client.0.smithi097.stderr:Or undo this operation with: 2024-03-11T20:00:32.325 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-03-11T20:00:32.325 INFO:tasks.workunit.client.0.smithi097.stderr: git switch - 2024-03-11T20:00:32.325 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-03-11T20:00:32.325 INFO:tasks.workunit.client.0.smithi097.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-11T20:00:32.325 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-03-11T20:00:32.325 INFO:tasks.workunit.client.0.smithi097.stderr:HEAD is now at b6a4997be2b Merge pull request #56060 from rhcs-dashboard/wip-64808-squid 2024-03-11T20:00:32.331 DEBUG:teuthology.orchestra.run.smithi097:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-11T20:00:32.428 INFO:tasks.workunit.client.0.smithi097.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-11T20:00:32.434 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-11T20:00:32.434 INFO:tasks.workunit.client.0.smithi097.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-11T20:00:33.443 INFO:tasks.workunit.client.0.smithi097.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-11T20:00:33.585 INFO:tasks.workunit.client.0.smithi097.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-11T20:00:33.666 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-11T20:00:33.672 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-11T20:00:33.672 INFO:tasks.workunit.client.0.smithi097.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-11T20:00:33.757 INFO:tasks.workunit.client.0.smithi097.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-11T20:00:33.766 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T20:00:33.766 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-11T20:00:33.816 INFO:tasks.workunit:Running workunits matching suites/fsync-tester.sh on client.0... 2024-03-11T20:00:33.817 INFO:tasks.workunit:Running workunit suites/fsync-tester.sh... 2024-03-11T20:00:33.818 DEBUG:teuthology.orchestra.run.smithi097:workunit test suites/fsync-tester.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6a4997be2b8bb74411268cc42753f29ae9a78ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsync-tester.sh 2024-03-11T20:00:33.930 INFO:tasks.workunit.client.0.smithi097.stderr:+ wget http://download.ceph.com/qa/fsync-tester.c 2024-03-11T20:00:33.970 INFO:tasks.workunit.client.0.smithi097.stderr:--2024-03-11 20:00:33-- http://download.ceph.com/qa/fsync-tester.c 2024-03-11T20:00:33.971 INFO:tasks.workunit.client.0.smithi097.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-11T20:00:34.094 INFO:tasks.workunit.client.0.smithi097.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-11T20:00:34.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:33 smithi097 ceph-mon[31898]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 12 op/s 2024-03-11T20:00:34.210 INFO:tasks.workunit.client.0.smithi097.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-11T20:00:34.210 INFO:tasks.workunit.client.0.smithi097.stderr:Length: 3024 (3.0K) [application/octet-stream] 2024-03-11T20:00:34.213 INFO:tasks.workunit.client.0.smithi097.stderr:Saving to: ‘fsync-tester.c’ 2024-03-11T20:00:34.220 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-03-11T20:00:34.220 INFO:tasks.workunit.client.0.smithi097.stderr: 0K .. 100% 104M=0s 2024-03-11T20:00:34.220 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-03-11T20:00:34.220 INFO:tasks.workunit.client.0.smithi097.stderr:2024-03-11 20:00:34 (104 MB/s) - ‘fsync-tester.c’ saved [3024/3024] 2024-03-11T20:00:34.220 INFO:tasks.workunit.client.0.smithi097.stderr: 2024-03-11T20:00:34.222 INFO:tasks.workunit.client.0.smithi097.stderr:+ gcc -D_GNU_SOURCE fsync-tester.c -o fsync-tester 2024-03-11T20:00:34.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:33 smithi114 ceph-mon[36376]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 12 op/s 2024-03-11T20:00:34.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:33 smithi161 ceph-mon[36332]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 12 op/s 2024-03-11T20:00:34.345 INFO:tasks.workunit.client.0.smithi097.stderr:+ ./fsync-tester 2024-03-11T20:00:36.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:35 smithi097 ceph-mon[31898]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-11T20:00:36.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:35 smithi114 ceph-mon[36376]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-11T20:00:36.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:35 smithi161 ceph-mon[36332]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-11T20:00:38.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:37 smithi097 ceph-mon[31898]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-11T20:00:38.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:37 smithi114 ceph-mon[36376]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-11T20:00:38.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:37 smithi161 ceph-mon[36332]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-11T20:00:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.630+0000 7fe005c00640 1 -- 172.21.15.97:0/1452880808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe000101d50 msgr2=0x7fe000105d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.630+0000 7fe005c00640 1 --2- 172.21.15.97:0/1452880808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe000101d50 0x7fe000105d90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009860 tx=0x7fdfec02bb70 comp rx=0 tx=0).stop 2024-03-11T20:00:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.630+0000 7fe005c00640 1 -- 172.21.15.97:0/1452880808 shutdown_connections 2024-03-11T20:00:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.630+0000 7fe005c00640 1 --2- 172.21.15.97:0/1452880808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe000106470 0x7fe00010e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.630+0000 7fe005c00640 1 --2- 172.21.15.97:0/1452880808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe000101d50 0x7fe000105d90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.630+0000 7fe005c00640 1 --2- 172.21.15.97:0/1452880808 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0001013a0 0x7fe000101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.630+0000 7fe005c00640 1 -- 172.21.15.97:0/1452880808 >> 172.21.15.97:0/1452880808 conn(0x7fe0000fcc40 msgr2=0x7fe0000ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.630+0000 7fe005c00640 1 -- 172.21.15.97:0/1452880808 shutdown_connections 2024-03-11T20:00:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.630+0000 7fe005c00640 1 -- 172.21.15.97:0/1452880808 wait complete. 2024-03-11T20:00:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.631+0000 7fe005c00640 1 Processor -- start 2024-03-11T20:00:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.631+0000 7fe005c00640 1 -- start start 2024-03-11T20:00:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.631+0000 7fe005c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0001013a0 0x7fe00019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.631+0000 7fe005c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe000106470 0x7fe00019fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.631+0000 7fe005c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0001997f0 0x7fe000199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.631+0000 7fe005c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00019a2a0 con 0x7fe0001997f0 2024-03-11T20:00:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.631+0000 7fe005c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00019a410 con 0x7fe0001013a0 2024-03-11T20:00:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.631+0000 7fe005c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00019a580 con 0x7fe000106470 2024-03-11T20:00:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.632+0000 7fdffea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0001013a0 0x7fe00019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.632+0000 7fdffea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0001013a0 0x7fe00019f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:46878/0 (socket says 172.21.15.97:46878) 2024-03-11T20:00:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.632+0000 7fdffea00640 1 -- 172.21.15.97:0/3234514694 learned_addr learned my addr 172.21.15.97:0/3234514694 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:38.630 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.632+0000 7fdfff400640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0001997f0 0x7fe000199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.632+0000 7fdffe000640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe000106470 0x7fe00019fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.632+0000 7fdffea00640 1 -- 172.21.15.97:0/3234514694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe000106470 msgr2=0x7fe00019fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.632+0000 7fdffea00640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe000106470 0x7fe00019fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.632+0000 7fdffea00640 1 -- 172.21.15.97:0/3234514694 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0001997f0 msgr2=0x7fe000199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.632+0000 7fdffea00640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0001997f0 0x7fe000199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.632+0000 7fdffea00640 1 -- 172.21.15.97:0/3234514694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff0009560 con 0x7fe0001013a0 2024-03-11T20:00:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.632+0000 7fdffea00640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0001013a0 0x7fe00019f680 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdff000e9c0 tx=0x7fdff000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.632+0000 7fdff7400640 1 -- 172.21.15.97:0/3234514694 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff000cde0 con 0x7fe0001013a0 2024-03-11T20:00:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.633+0000 7fdff7400640 1 -- 172.21.15.97:0/3234514694 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdff00035f0 con 0x7fe0001013a0 2024-03-11T20:00:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.633+0000 7fe005c00640 1 -- 172.21.15.97:0/3234514694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfec009490 con 0x7fe0001013a0 2024-03-11T20:00:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.633+0000 7fdff7400640 1 -- 172.21.15.97:0/3234514694 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff0010670 con 0x7fe0001013a0 2024-03-11T20:00:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.633+0000 7fe005c00640 1 -- 172.21.15.97:0/3234514694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0001a6830 con 0x7fe0001013a0 2024-03-11T20:00:38.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.636+0000 7fe00883e740 1 Processor -- start 2024-03-11T20:00:38.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.636+0000 7fdff7400640 1 -- 172.21.15.97:0/3234514694 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fdff0003ad0 con 0x7fe0001013a0 2024-03-11T20:00:38.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.636+0000 7fe00883e740 1 -- start start 2024-03-11T20:00:38.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.636+0000 7fe00883e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 0x56018a9fc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.636+0000 7fe00883e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fcdb0 0x56018a9fd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.636+0000 7fe00883e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 0x56018aa08720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.636+0000 7fe00883e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018a9b8620 con 0x56018a9fcdb0 2024-03-11T20:00:38.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.636+0000 7fe00883e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018a96a8c0 con 0x56018a9fdb20 2024-03-11T20:00:38.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.636+0000 7fe00883e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018a967df0 con 0x56018aa06330 2024-03-11T20:00:38.633 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.636+0000 7fdfff400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 0x56018a9fc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.636+0000 7fdffea00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fcdb0 0x56018a9fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.637+0000 7fdfff400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 0x56018a9fc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:46890/0 (socket says 172.21.15.97:46890) 2024-03-11T20:00:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.637+0000 7fdfff400640 1 -- 172.21.15.97:0/2730879009 learned_addr learned my addr 172.21.15.97:0/2730879009 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:38.634 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.637+0000 7fdffe000640 1 --2- 172.21.15.97:0/2730879009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 0x56018aa08720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.637+0000 7fdfff400640 1 -- 172.21.15.97:0/2730879009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 msgr2=0x56018aa08720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.637+0000 7fdfff400640 1 --2- 172.21.15.97:0/2730879009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 0x56018aa08720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.637+0000 7fdfff400640 1 -- 172.21.15.97:0/2730879009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fcdb0 msgr2=0x56018a9fd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.637+0000 7fdfff400640 1 --2- 172.21.15.97:0/2730879009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fcdb0 0x56018a9fd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.637+0000 7fdfff400640 1 -- 172.21.15.97:0/2730879009 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56018a8e4b00 con 0x56018a9fdb20 2024-03-11T20:00:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.637+0000 7fdfff400640 1 --2- 172.21.15.97:0/2730879009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 0x56018a9fc870 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009f50 tx=0x7fdff80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8ce43f298681a10 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.637+0000 7fdff7400640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fdfe40a1d60 0x7fdfe40a4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.637+0000 7fdfebe00640 1 -- 172.21.15.97:0/2730879009 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8004d20 con 0x56018a9fdb20 2024-03-11T20:00:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.638+0000 7fdfebe00640 1 -- 172.21.15.97:0/2730879009 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdff8004e80 con 0x56018a9fdb20 2024-03-11T20:00:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.638+0000 7fdfebe00640 1 -- 172.21.15.97:0/2730879009 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8005220 con 0x56018a9fdb20 2024-03-11T20:00:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.638+0000 7fdff7400640 1 -- 172.21.15.97:0/3234514694 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7fdff0014070 con 0x7fe0001013a0 2024-03-11T20:00:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.638+0000 7fdffea00640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fdfe40a1d60 0x7fdfe40a4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.638+0000 7fe00883e740 1 -- 172.21.15.97:0/2730879009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 msgr2=0x56018a9fc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.638+0000 7fdffea00640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fdfe40a1d60 0x7fdfe40a4220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdff00cb4c0 tx=0x7fdff00d1b20 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.638+0000 7fe00883e740 1 --2- 172.21.15.97:0/2730879009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 0x56018a9fc870 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009f50 tx=0x7fdff80118d0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.638+0000 7fe00883e740 1 -- 172.21.15.97:0/2730879009 shutdown_connections 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.638+0000 7fe00883e740 1 --2- 172.21.15.97:0/2730879009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 0x56018aa08720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.638+0000 7fe00883e740 1 --2- 172.21.15.97:0/2730879009 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fcdb0 0x56018a9fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.638+0000 7fe00883e740 1 --2- 172.21.15.97:0/2730879009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 0x56018a9fc870 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.638+0000 7fe00883e740 1 -- 172.21.15.97:0/2730879009 >> 172.21.15.97:0/2730879009 conn(0x56018a9c44b0 msgr2=0x56018a9f7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.639+0000 7fe00883e740 1 -- 172.21.15.97:0/2730879009 shutdown_connections 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.639+0000 7fe00883e740 1 -- 172.21.15.97:0/2730879009 wait complete. 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.639+0000 7fe00883e740 1 -- start start 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.639+0000 7fe00883e740 1 Processor -- start 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fe00883e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 0x56018aa03300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fe00883e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fdb20 0x56018aa03840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fe00883e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 0x56018aa03d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fe00883e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018a9635d0 con 0x56018a9fdb20 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fe00883e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018a908b50 con 0x56018a9fcdb0 2024-03-11T20:00:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fe00883e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018a9087f0 con 0x56018aa06330 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fe00883e740 10 client.0 _ll_register_callbacks cb 0x56018a91c550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fdfff400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 0x56018aa03300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fdffe000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fdb20 0x56018aa03840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fdfff400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 0x56018aa03300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:46894/0 (socket says 172.21.15.97:46894) 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fdffe000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fdb20 0x56018aa03840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:54840/0 (socket says 172.21.15.97:54840) 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fdfff400640 1 -- 172.21.15.97:0/2061342504 learned_addr learned my addr 172.21.15.97:0/2061342504 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fdfff400640 1 --2- 172.21.15.97:0/2061342504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 0x56018aa03d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fdffe000640 1 -- 172.21.15.97:0/2061342504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 msgr2=0x56018aa03d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fdffe000640 1 --2- 172.21.15.97:0/2061342504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 0x56018aa03d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fdffe000640 1 -- 172.21.15.97:0/2061342504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 msgr2=0x56018aa03300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fdffe000640 1 --2- 172.21.15.97:0/2061342504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 0x56018aa03300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fdffe000640 1 -- 172.21.15.97:0/2061342504 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff8009c00 con 0x56018a9fdb20 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.640+0000 7fdfff400640 1 --2- 172.21.15.97:0/2061342504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 0x56018aa03300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:00:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.641+0000 7fdffe000640 1 --2- 172.21.15.97:0/2061342504 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fdb20 0x56018aa03840 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdfec037040 tx=0x7fdfec034440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c4d0c5adbc2f2d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.641+0000 7fdfe8c00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.641+0000 7fdfe8c00640 1 -- 172.21.15.97:0/2061342504 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec005050 con 0x56018a9fdb20 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.641+0000 7fdfe8c00640 1 -- 172.21.15.97:0/2061342504 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7fdfec0051b0 con 0x56018a9fdb20 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.641+0000 7fe00883e740 1 -- 172.21.15.97:0/2061342504 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56018a935cd0 con 0x56018a9fdb20 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.641+0000 7fdfe8c00640 1 -- 172.21.15.97:0/2061342504 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfec005370 con 0x56018a9fdb20 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.641+0000 7fe00883e740 10 client.15570 fetch_fsmap learned FSMap version 29 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.641+0000 7fe00883e740 1 -- 172.21.15.97:0/2061342504 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56018a961c50 con 0x56018a9fdb20 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.642+0000 7fdfe8c00640 1 -- 172.21.15.97:0/2061342504 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56018a935cd0 con 0x56018a9fdb20 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.642+0000 7fdfe8c00640 1 -- 172.21.15.97:0/2061342504 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fdfec0048c0 con 0x56018a9fdb20 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.642+0000 7fe00883e740 10 client.15570 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.642+0000 7fe00883e740 10 client.15570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.642+0000 7fe00883e740 20 client.15570 populate_metadata read hostname 'smithi097' 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.642+0000 7fe00883e740 1 --2- 172.21.15.97:0/2061342504 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x56018aa16170 0x56018aa18590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.642+0000 7fe00883e740 4 client.15570 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.642+0000 7fe00883e740 1 -- 172.21.15.97:0/2061342504 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56018a959ce0 con 0x56018aa16170 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.642+0000 7fdfff400640 1 --2- 172.21.15.97:0/2061342504 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x56018aa16170 0x56018aa18590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.643+0000 7fdfff400640 1 --2- 172.21.15.97:0/2061342504 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x56018aa16170 0x56018aa18590 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f69ab7c8f4c8ca31 server_cookie=857b3fc0a742439b in_seq=0 out_seq=0 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.643+0000 7fdfe8c00640 10 client.15570 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fdfe8c00640 1 -- 172.21.15.97:0/2061342504 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56018a959ce0 con 0x56018aa16170 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fdfe8c00640 10 client.15570 handle_command_reply: tid=0 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fe00883e740 1 client.15570 shutdown 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fe00883e740 1 -- 172.21.15.97:0/2061342504 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fdb20 msgr2=0x56018aa03840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fe00883e740 1 --2- 172.21.15.97:0/2061342504 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fdb20 0x56018aa03840 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdfec037040 tx=0x7fdfec034440 comp rx=0 tx=0).stop 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fe00883e740 1 -- 172.21.15.97:0/2061342504 shutdown_connections 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fe00883e740 1 --2- 172.21.15.97:0/2061342504 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 0x56018aa03d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fe00883e740 1 --2- 172.21.15.97:0/2061342504 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x56018aa16170 0x56018aa18590 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fe00883e740 1 --2- 172.21.15.97:0/2061342504 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fdb20 0x56018aa03840 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fe00883e740 1 --2- 172.21.15.97:0/2061342504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 0x56018aa03300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fdfe8c00640 0 client.15570 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fe00883e740 1 -- 172.21.15.97:0/2061342504 >> 172.21.15.97:0/2061342504 conn(0x56018a9c44b0 msgr2=0x56018a9f97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fe00883e740 1 -- 172.21.15.97:0/2061342504 shutdown_connections 2024-03-11T20:00:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fe00883e740 1 -- 172.21.15.97:0/2061342504 wait complete. 2024-03-11T20:00:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.644+0000 7fe00883e740 20 client.15570 trim_cache size 0 max 16384 2024-03-11T20:00:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.658+0000 7fe00883e740 1 Processor -- start 2024-03-11T20:00:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.658+0000 7fe00883e740 1 -- start start 2024-03-11T20:00:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fe00883e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fcdb0 0x56018aa05a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fe00883e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 0x56018aa03ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fe00883e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 0x56018aa0d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fe00883e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018a6abe60 con 0x56018a9fcdb0 2024-03-11T20:00:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fe00883e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018a9f37a0 con 0x56018a9fdb20 2024-03-11T20:00:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fe00883e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018aa12dd0 con 0x56018aa06330 2024-03-11T20:00:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fdffe000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 0x56018aa03ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fdffe000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 0x56018aa03ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:46896/0 (socket says 172.21.15.97:46896) 2024-03-11T20:00:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fdffe000640 1 -- 172.21.15.97:0/715505791 learned_addr learned my addr 172.21.15.97:0/715505791 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fdfff400640 1 --2- 172.21.15.97:0/715505791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fcdb0 0x56018aa05a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fdffe000640 1 -- 172.21.15.97:0/715505791 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 msgr2=0x56018aa0d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fdfff400640 1 --2- 172.21.15.97:0/715505791 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 0x56018aa0d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fdffe000640 1 --2- 172.21.15.97:0/715505791 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 0x56018aa0d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fdffe000640 1 -- 172.21.15.97:0/715505791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fcdb0 msgr2=0x56018aa05a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fdffe000640 1 --2- 172.21.15.97:0/715505791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fcdb0 0x56018aa05a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fdffe000640 1 -- 172.21.15.97:0/715505791 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56018a8fe530 con 0x56018a9fdb20 2024-03-11T20:00:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.659+0000 7fdfff400640 1 --2- 172.21.15.97:0/715505791 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 0x56018aa0d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T20:00:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.660+0000 7fdffe000640 1 --2- 172.21.15.97:0/715505791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 0x56018aa03ba0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdfec004c50 tx=0x7fdfec02bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61a4629357f763e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.660+0000 7fdfeb400640 1 -- 172.21.15.97:0/715505791 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec02c270 con 0x56018a9fdb20 2024-03-11T20:00:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.660+0000 7fdfeb400640 1 -- 172.21.15.97:0/715505791 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfec0344b0 con 0x56018a9fdb20 2024-03-11T20:00:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.660+0000 7fdfeb400640 1 -- 172.21.15.97:0/715505791 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec02cb50 con 0x56018a9fdb20 2024-03-11T20:00:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.661+0000 7fe00883e740 1 -- 172.21.15.97:0/715505791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 msgr2=0x56018aa03ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.661+0000 7fe00883e740 1 --2- 172.21.15.97:0/715505791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 0x56018aa03ba0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdfec004c50 tx=0x7fdfec02bb70 comp rx=0 tx=0).stop 2024-03-11T20:00:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.661+0000 7fe00883e740 1 -- 172.21.15.97:0/715505791 shutdown_connections 2024-03-11T20:00:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.661+0000 7fe00883e740 1 --2- 172.21.15.97:0/715505791 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018aa06330 0x56018aa0d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.661+0000 7fe00883e740 1 --2- 172.21.15.97:0/715505791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fdb20 0x56018aa03ba0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.661+0000 7fe00883e740 1 --2- 172.21.15.97:0/715505791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018a9fcdb0 0x56018aa05a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.661+0000 7fe00883e740 1 -- 172.21.15.97:0/715505791 >> 172.21.15.97:0/715505791 conn(0x56018a9c44b0 msgr2=0x56018aa01bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.661+0000 7fe00883e740 1 -- 172.21.15.97:0/715505791 shutdown_connections 2024-03-11T20:00:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.661+0000 7fe00883e740 1 -- 172.21.15.97:0/715505791 wait complete. 2024-03-11T20:00:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.661+0000 7fe00883e740 1 -- start start 2024-03-11T20:00:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.662+0000 7fe00883e740 1 Processor -- start 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.662+0000 7fe00883e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 0x56018aa0d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.662+0000 7fe00883e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018a9fdb20 0x56018a9ffb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.662+0000 7fe00883e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018aa06330 0x56018aa00090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.662+0000 7fe00883e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018aa005d0 con 0x56018aa06330 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.662+0000 7fe00883e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018aa00740 con 0x56018a9fcdb0 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.662+0000 7fe00883e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018aa00900 con 0x56018a9fdb20 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.662+0000 7fe00883e740 10 client.0 _ll_register_callbacks cb 0x7fdff80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.662+0000 7fdfff400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018aa06330 0x56018aa00090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.662+0000 7fdfff400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 0x56018aa0d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.662+0000 7fdffe000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018a9fdb20 0x56018a9ffb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.662+0000 7fdfff400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 0x56018aa0d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:46906/0 (socket says 172.21.15.97:46906) 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.662+0000 7fdfff400640 1 -- 172.21.15.97:0/2640881905 learned_addr learned my addr 172.21.15.97:0/2640881905 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.663+0000 7fdfff400640 1 -- 172.21.15.97:0/2640881905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018a9fdb20 msgr2=0x56018a9ffb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.663+0000 7fdfff400640 1 --2- 172.21.15.97:0/2640881905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018a9fdb20 0x56018a9ffb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.663+0000 7fdfff400640 1 -- 172.21.15.97:0/2640881905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018aa06330 msgr2=0x56018aa00090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.663+0000 7fdfff400640 1 --2- 172.21.15.97:0/2640881905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018aa06330 0x56018aa00090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.663+0000 7fdfff400640 1 -- 172.21.15.97:0/2640881905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56018a961c50 con 0x56018a9fcdb0 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.663+0000 7fdfff400640 1 --2- 172.21.15.97:0/2640881905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 0x56018aa0d350 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009ef0 tx=0x7fdff80046f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d787bb3ac224482 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.663+0000 7fdfe8c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.663+0000 7fdfe8c00640 1 -- 172.21.15.97:0/2640881905 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8003c80 con 0x56018a9fcdb0 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.663+0000 7fe00883e740 1 -- 172.21.15.97:0/2640881905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56018a663600 con 0x56018a9fcdb0 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.663+0000 7fdfe8c00640 1 -- 172.21.15.97:0/2640881905 <== mon.2 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7fdff802d0b0 con 0x56018a9fcdb0 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.664+0000 7fdfe8c00640 1 -- 172.21.15.97:0/2640881905 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdff8003de0 con 0x56018a9fcdb0 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.664+0000 7fe00883e740 10 client.25694 fetch_fsmap learned FSMap version 29 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.664+0000 7fe00883e740 1 -- 172.21.15.97:0/2640881905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56018a9242c0 con 0x56018a9fcdb0 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.664+0000 7fdfe8c00640 1 -- 172.21.15.97:0/2640881905 <== mon.2 v2:172.21.15.114:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56018a663600 con 0x56018a9fcdb0 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.664+0000 7fdfe8c00640 1 -- 172.21.15.97:0/2640881905 <== mon.2 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fdff8037070 con 0x56018a9fcdb0 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.664+0000 7fe00883e740 10 client.25694 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.664+0000 7fe00883e740 10 client.25694 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.664+0000 7fe00883e740 20 client.25694 populate_metadata read hostname 'smithi097' 2024-03-11T20:00:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.664+0000 7fe00883e740 1 --2- 172.21.15.97:0/2640881905 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x56018aa6da10 0x56018aa6fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.664+0000 7fe00883e740 4 client.25694 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T20:00:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.664+0000 7fe00883e740 1 -- 172.21.15.97:0/2640881905 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56018a6b41b0 con 0x56018aa6da10 2024-03-11T20:00:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.664+0000 7fdffe000640 1 --2- 172.21.15.97:0/2640881905 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x56018aa6da10 0x56018aa6fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.665+0000 7fdffe000640 1 --2- 172.21.15.97:0/2640881905 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x56018aa6da10 0x56018aa6fe30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3185ee83d9c53441 server_cookie=eecccfed559e6a3d in_seq=0 out_seq=0 2024-03-11T20:00:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.665+0000 7fdfe8c00640 10 client.25694 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fdfe8c00640 1 -- 172.21.15.97:0/2640881905 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56018a6b41b0 con 0x56018aa6da10 2024-03-11T20:00:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fdfe8c00640 10 client.25694 handle_command_reply: tid=0 2024-03-11T20:00:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fe00883e740 1 client.25694 shutdown 2024-03-11T20:00:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fe00883e740 1 -- 172.21.15.97:0/2640881905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 msgr2=0x56018aa0d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fe00883e740 1 --2- 172.21.15.97:0/2640881905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 0x56018aa0d350 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009ef0 tx=0x7fdff80046f0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fe00883e740 1 -- 172.21.15.97:0/2640881905 shutdown_connections 2024-03-11T20:00:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fe00883e740 1 --2- 172.21.15.97:0/2640881905 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x56018aa6da10 0x56018aa6fe30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fe00883e740 1 --2- 172.21.15.97:0/2640881905 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x56018aa06330 0x56018aa00090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fe00883e740 1 --2- 172.21.15.97:0/2640881905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56018a9fdb20 0x56018a9ffb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fdfe8c00640 0 client.25694 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fe00883e740 1 --2- 172.21.15.97:0/2640881905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56018a9fcdb0 0x56018aa0d350 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fe00883e740 1 -- 172.21.15.97:0/2640881905 >> 172.21.15.97:0/2640881905 conn(0x56018a9c44b0 msgr2=0x56018a9f64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fe00883e740 1 -- 172.21.15.97:0/2640881905 shutdown_connections 2024-03-11T20:00:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fe00883e740 1 -- 172.21.15.97:0/2640881905 wait complete. 2024-03-11T20:00:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.677+0000 7fe00883e740 20 client.25694 trim_cache size 0 max 16384 2024-03-11T20:00:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-03-11T20:00:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-03-11T20:00:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-03-11T20:00:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-03-11T20:00:38.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.678+0000 7fe005c00640 1 -- 172.21.15.97:0/3234514694 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fdfe40a1d60 msgr2=0x7fdfe40a4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:38.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.678+0000 7fe005c00640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fdfe40a1d60 0x7fdfe40a4220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdff00cb4c0 tx=0x7fdff00d1b20 comp rx=0 tx=0).stop 2024-03-11T20:00:38.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.678+0000 7fe005c00640 1 -- 172.21.15.97:0/3234514694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0001013a0 msgr2=0x7fe00019f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:38.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.678+0000 7fe005c00640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0001013a0 0x7fe00019f680 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdff000e9c0 tx=0x7fdff000ee80 comp rx=0 tx=0).stop 2024-03-11T20:00:38.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.678+0000 7fe005c00640 1 -- 172.21.15.97:0/3234514694 shutdown_connections 2024-03-11T20:00:38.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.678+0000 7fe005c00640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fdfe40a1d60 0x7fdfe40a4220 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.676 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.678+0000 7fe005c00640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe0001997f0 0x7fe000199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.678+0000 7fe005c00640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe000106470 0x7fe00019fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.678+0000 7fe005c00640 1 --2- 172.21.15.97:0/3234514694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0001013a0 0x7fe00019f680 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.678+0000 7fe005c00640 1 -- 172.21.15.97:0/3234514694 >> 172.21.15.97:0/3234514694 conn(0x7fe0000fcc40 msgr2=0x7fe0000fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.678+0000 7fe005c00640 1 -- 172.21.15.97:0/3234514694 shutdown_connections 2024-03-11T20:00:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:38.678+0000 7fe005c00640 1 -- 172.21.15.97:0/3234514694 wait complete. 2024-03-11T20:00:39.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:38 smithi097 ceph-mon[31898]: pgmap v44: 129 pgs: 129 active+clean; 2.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 263 KiB/s wr, 14 op/s 2024-03-11T20:00:39.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:38 smithi114 ceph-mon[36376]: pgmap v44: 129 pgs: 129 active+clean; 2.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 263 KiB/s wr, 14 op/s 2024-03-11T20:00:39.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:38 smithi161 ceph-mon[36332]: pgmap v44: 129 pgs: 129 active+clean; 2.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 263 KiB/s wr, 14 op/s 2024-03-11T20:00:42.031 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:41 smithi161 ceph-mon[36332]: pgmap v45: 129 pgs: 129 active+clean; 2.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 262 KiB/s wr, 10 op/s 2024-03-11T20:00:42.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:41 smithi097 ceph-mon[31898]: pgmap v45: 129 pgs: 129 active+clean; 2.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 262 KiB/s wr, 10 op/s 2024-03-11T20:00:42.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:41 smithi114 ceph-mon[36376]: pgmap v45: 129 pgs: 129 active+clean; 2.4 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 262 KiB/s wr, 10 op/s 2024-03-11T20:00:44.031 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:43 smithi161 ceph-mon[36332]: pgmap v46: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 695 KiB/s wr, 19 op/s 2024-03-11T20:00:44.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:43 smithi097 ceph-mon[31898]: pgmap v46: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 695 KiB/s wr, 19 op/s 2024-03-11T20:00:44.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:43 smithi114 ceph-mon[36376]: pgmap v46: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 695 KiB/s wr, 19 op/s 2024-03-11T20:00:46.031 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:45 smithi161 ceph-mon[36332]: pgmap v47: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 695 KiB/s wr, 15 op/s 2024-03-11T20:00:46.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:45 smithi097 ceph-mon[31898]: pgmap v47: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 695 KiB/s wr, 15 op/s 2024-03-11T20:00:46.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:45 smithi114 ceph-mon[36376]: pgmap v47: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 695 KiB/s wr, 15 op/s 2024-03-11T20:00:48.031 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:47 smithi161 ceph-mon[36332]: pgmap v48: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 695 KiB/s wr, 15 op/s 2024-03-11T20:00:48.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:47 smithi097 ceph-mon[31898]: pgmap v48: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 695 KiB/s wr, 15 op/s 2024-03-11T20:00:48.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:47 smithi114 ceph-mon[36376]: pgmap v48: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 695 KiB/s wr, 15 op/s 2024-03-11T20:00:50.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:49 smithi097 ceph-mon[31898]: pgmap v49: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.1 MiB/s wr, 21 op/s 2024-03-11T20:00:50.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:49 smithi097 ceph-mon[31898]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-03-11T20:00:50.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:49 smithi114 ceph-mon[36376]: pgmap v49: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.1 MiB/s wr, 21 op/s 2024-03-11T20:00:50.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:49 smithi114 ceph-mon[36376]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-03-11T20:00:50.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:49 smithi161 ceph-mon[36332]: pgmap v49: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.1 MiB/s wr, 21 op/s 2024-03-11T20:00:50.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:49 smithi161 ceph-mon[36332]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-03-11T20:00:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.814+0000 7f7ab5600640 1 -- 172.21.15.97:0/3709492084 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ab00fffa0 msgr2=0x7f7ab010c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.814+0000 7f7ab5600640 1 --2- 172.21.15.97:0/3709492084 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ab00fffa0 0x7f7ab010c4b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7aa800ade0 tx=0x7f7aa802cfd0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.814+0000 7f7ab5600640 1 -- 172.21.15.97:0/3709492084 shutdown_connections 2024-03-11T20:00:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.814+0000 7f7ab5600640 1 --2- 172.21.15.97:0/3709492084 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ab00fffa0 0x7f7ab010c4b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.814+0000 7f7ab5600640 1 --2- 172.21.15.97:0/3709492084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ab00ff5e0 0x7f7ab00ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.814+0000 7f7ab5600640 1 --2- 172.21.15.97:0/3709492084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ab0105680 0x7f7ab0105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.814+0000 7f7ab5600640 1 -- 172.21.15.97:0/3709492084 >> 172.21.15.97:0/3709492084 conn(0x7f7ab00763c0 msgr2=0x7f7ab00fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.814+0000 7f7ab5600640 1 -- 172.21.15.97:0/3709492084 shutdown_connections 2024-03-11T20:00:51.810 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.814+0000 7f7ab5600640 1 -- 172.21.15.97:0/3709492084 wait complete. 2024-03-11T20:00:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.815+0000 7f7ab5600640 1 Processor -- start 2024-03-11T20:00:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.816+0000 7f7ab5600640 1 -- start start 2024-03-11T20:00:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.816+0000 7f7ab5600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ab00ff5e0 0x7f7ab0101280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.816+0000 7f7ab5600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ab00fffa0 0x7f7ab01017c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.816+0000 7f7ab5600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ab0105680 0x7f7ab0101d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.816+0000 7f7ab5600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab01a5940 con 0x7f7ab00ff5e0 2024-03-11T20:00:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.816+0000 7f7ab5600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab01a5ab0 con 0x7f7ab0105680 2024-03-11T20:00:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.816+0000 7f7ab5600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab01a5c70 con 0x7f7ab00fffa0 2024-03-11T20:00:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.816+0000 7f7aafe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ab00ff5e0 0x7f7ab0101280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.812 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.816+0000 7f7aafe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ab00ff5e0 0x7f7ab0101280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41454/0 (socket says 172.21.15.97:41454) 2024-03-11T20:00:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.816+0000 7f7aafe00640 1 -- 172.21.15.97:0/619171021 learned_addr learned my addr 172.21.15.97:0/619171021 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.816+0000 7f7aaf400640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ab00fffa0 0x7f7ab01017c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.816+0000 7f7ab4c00640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ab0105680 0x7f7ab0101d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.817+0000 7f7aafe00640 1 -- 172.21.15.97:0/619171021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ab00fffa0 msgr2=0x7f7ab01017c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.817+0000 7f7aafe00640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ab00fffa0 0x7f7ab01017c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.817+0000 7f7aafe00640 1 -- 172.21.15.97:0/619171021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ab0105680 msgr2=0x7f7ab0101d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.817+0000 7f7aafe00640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ab0105680 0x7f7ab0101d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.813 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.817+0000 7f7aafe00640 1 -- 172.21.15.97:0/619171021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa800aa10 con 0x7f7ab00ff5e0 2024-03-11T20:00:51.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.817+0000 7f7aafe00640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ab00ff5e0 0x7f7ab0101280 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000e9c0 tx=0x7f7aa000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:51.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.817+0000 7f7aacc00640 1 -- 172.21.15.97:0/619171021 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa00096e0 con 0x7f7ab00ff5e0 2024-03-11T20:00:51.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.817+0000 7f7ab5600640 1 -- 172.21.15.97:0/619171021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab01a5f50 con 0x7f7ab00ff5e0 2024-03-11T20:00:51.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.817+0000 7f7aacc00640 1 -- 172.21.15.97:0/619171021 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7aa000cd80 con 0x7f7ab00ff5e0 2024-03-11T20:00:51.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.817+0000 7f7aacc00640 1 -- 172.21.15.97:0/619171021 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0010430 con 0x7f7ab00ff5e0 2024-03-11T20:00:51.814 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.818+0000 7f7ab5600640 1 -- 172.21.15.97:0/619171021 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab01a6380 con 0x7f7ab00ff5e0 2024-03-11T20:00:51.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.821+0000 7f7aacc00640 1 -- 172.21.15.97:0/619171021 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f7aa0003680 con 0x7f7ab00ff5e0 2024-03-11T20:00:51.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.821+0000 7f7ab99e1740 1 Processor -- start 2024-03-11T20:00:51.816 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.821+0000 7f7ab99e1740 1 -- start start 2024-03-11T20:00:51.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.821+0000 7f7ab99e1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a1ed60 0x557d31a1dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.821+0000 7f7ab99e1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1dff0 0x557d31a1e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.821+0000 7f7ab99e1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a27570 0x557d31a29960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7ab99e1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d319d9620 con 0x557d31a27570 2024-03-11T20:00:51.817 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7ab99e1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3198b8c0 con 0x557d31a1ed60 2024-03-11T20:00:51.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7ab99e1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d31988df0 con 0x557d31a1dff0 2024-03-11T20:00:51.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7aaf400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a27570 0x557d31a29960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7aafe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1dff0 0x557d31a1e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7ab4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a1ed60 0x557d31a1dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7aafe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1dff0 0x557d31a1e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:54166/0 (socket says 172.21.15.97:54166) 2024-03-11T20:00:51.818 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7aafe00640 1 -- 172.21.15.97:0/354608222 learned_addr learned my addr 172.21.15.97:0/354608222 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:51.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7ab4c00640 1 -- 172.21.15.97:0/354608222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1dff0 msgr2=0x557d31a1e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:51.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7ab4c00640 1 --2- 172.21.15.97:0/354608222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1dff0 0x557d31a1e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7ab4c00640 1 -- 172.21.15.97:0/354608222 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a27570 msgr2=0x557d31a29960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:51.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7ab4c00640 1 --2- 172.21.15.97:0/354608222 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a27570 0x557d31a29960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7ab4c00640 1 -- 172.21.15.97:0/354608222 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d31905b00 con 0x557d31a1ed60 2024-03-11T20:00:51.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.822+0000 7f7aaf400640 1 --2- 172.21.15.97:0/354608222 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a27570 0x557d31a29960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:00:51.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.823+0000 7f7aacc00640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f7a940a1b90 0x7f7a940a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.820 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.823+0000 7f7ab4c00640 1 --2- 172.21.15.97:0/354608222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a1ed60 0x557d31a1dab0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7aa802d510 tx=0x7f7aa8005c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=288525ee58e66da8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.823+0000 7f7aacc00640 1 -- 172.21.15.97:0/619171021 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f7aa0014070 con 0x7f7ab00ff5e0 2024-03-11T20:00:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.823+0000 7f7aafe00640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f7a940a1b90 0x7f7a940a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.823+0000 7f7aa5200640 1 -- 172.21.15.97:0/354608222 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8033c50 con 0x557d31a1ed60 2024-03-11T20:00:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.823+0000 7f7aa5200640 1 -- 172.21.15.97:0/354608222 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7aa8033db0 con 0x557d31a1ed60 2024-03-11T20:00:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.823+0000 7f7aa5200640 1 -- 172.21.15.97:0/354608222 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8033590 con 0x557d31a1ed60 2024-03-11T20:00:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.824+0000 7f7ab99e1740 1 -- 172.21.15.97:0/354608222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a1ed60 msgr2=0x557d31a1dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.824+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/354608222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a1ed60 0x557d31a1dab0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7aa802d510 tx=0x7f7aa8005c50 comp rx=0 tx=0).stop 2024-03-11T20:00:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.824+0000 7f7aafe00640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f7a940a1b90 0x7f7a940a4050 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0096ba0 tx=0x7f7aa00d17c0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.824+0000 7f7ab99e1740 1 -- 172.21.15.97:0/354608222 shutdown_connections 2024-03-11T20:00:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.824+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/354608222 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a27570 0x557d31a29960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.824+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/354608222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1dff0 0x557d31a1e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.824+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/354608222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a1ed60 0x557d31a1dab0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.821 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.824+0000 7f7ab99e1740 1 -- 172.21.15.97:0/354608222 >> 172.21.15.97:0/354608222 conn(0x557d319e54b0 msgr2=0x557d31a18ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:51.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.824+0000 7f7ab99e1740 1 -- 172.21.15.97:0/354608222 shutdown_connections 2024-03-11T20:00:51.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.824+0000 7f7ab99e1740 1 -- 172.21.15.97:0/354608222 wait complete. 2024-03-11T20:00:51.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.824+0000 7f7ab99e1740 1 -- start start 2024-03-11T20:00:51.822 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.825+0000 7f7ab99e1740 1 Processor -- start 2024-03-11T20:00:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.825+0000 7f7ab99e1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a1dff0 0x557d31a24630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.825+0000 7f7ab99e1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a24b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.825+0000 7f7ab99e1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a27570 0x557d31a250b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.825+0000 7f7ab99e1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d319845d0 con 0x557d31a27570 2024-03-11T20:00:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.825+0000 7f7ab99e1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d31929b50 con 0x557d31a1dff0 2024-03-11T20:00:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.825+0000 7f7ab99e1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d319297f0 con 0x557d31a1ed60 2024-03-11T20:00:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.825+0000 7f7ab99e1740 10 client.0 _ll_register_callbacks cb 0x557d3193d550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:00:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.825+0000 7f7ab4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a1dff0 0x557d31a24630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.826+0000 7f7ab4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a27570 0x557d31a250b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.826+0000 7f7aaf400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a24b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.826+0000 7f7ab4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a27570 0x557d31a250b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41470/0 (socket says 172.21.15.97:41470) 2024-03-11T20:00:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.826+0000 7f7ab4c00640 1 -- 172.21.15.97:0/652160585 learned_addr learned my addr 172.21.15.97:0/652160585 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.826+0000 7f7aaf400640 1 -- 172.21.15.97:0/652160585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a1dff0 msgr2=0x557d31a24630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.826+0000 7f7aaf400640 1 --2- 172.21.15.97:0/652160585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a1dff0 0x557d31a24630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.826+0000 7f7aaf400640 1 -- 172.21.15.97:0/652160585 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a27570 msgr2=0x557d31a250b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.826+0000 7f7aaf400640 1 --2- 172.21.15.97:0/652160585 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a27570 0x557d31a250b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.826+0000 7f7aaf400640 1 -- 172.21.15.97:0/652160585 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab010cdb0 con 0x557d31a1ed60 2024-03-11T20:00:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.826+0000 7f7ab4c00640 1 --2- 172.21.15.97:0/652160585 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a27570 0x557d31a250b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:00:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.826+0000 7f7aaf400640 1 --2- 172.21.15.97:0/652160585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a24b70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x557d31a1ea90 tx=0x7f7a9c012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=161325d12de37b5b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.826+0000 7f7a99600640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-11T20:00:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.827+0000 7f7a99600640 1 -- 172.21.15.97:0/652160585 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c017070 con 0x557d31a1ed60 2024-03-11T20:00:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.827+0000 7f7ab99e1740 1 -- 172.21.15.97:0/652160585 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d31956cd0 con 0x557d31a1ed60 2024-03-11T20:00:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.827+0000 7f7a99600640 1 -- 172.21.15.97:0/652160585 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f7a9c004030 con 0x557d31a1ed60 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.827+0000 7f7a99600640 1 -- 172.21.15.97:0/652160585 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a9c0041f0 con 0x557d31a1ed60 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.827+0000 7f7ab99e1740 10 client.25693 fetch_fsmap learned FSMap version 29 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.827+0000 7f7ab99e1740 1 -- 172.21.15.97:0/652160585 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557d31982c50 con 0x557d31a1ed60 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.827+0000 7f7a99600640 1 -- 172.21.15.97:0/652160585 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557d31956cd0 con 0x557d31a1ed60 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.827+0000 7f7a99600640 1 -- 172.21.15.97:0/652160585 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7a9c003940 con 0x557d31a1ed60 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.828+0000 7f7ab99e1740 10 client.25693 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.828+0000 7f7ab99e1740 10 client.25693 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.828+0000 7f7ab99e1740 20 client.25693 populate_metadata read hostname 'smithi097' 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.828+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/652160585 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557d31a376e0 0x557d31a39b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.828+0000 7f7ab99e1740 4 client.25693 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.828+0000 7f7ab99e1740 1 -- 172.21.15.97:0/652160585 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d3197ace0 con 0x557d31a376e0 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.828+0000 7f7ab4c00640 1 --2- 172.21.15.97:0/652160585 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557d31a376e0 0x557d31a39b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.829+0000 7f7ab4c00640 1 --2- 172.21.15.97:0/652160585 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557d31a376e0 0x557d31a39b00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ea88d9b3d30f98e server_cookie=8278d037897ae7eb in_seq=0 out_seq=0 2024-03-11T20:00:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.829+0000 7f7a99600640 10 client.25693 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.829+0000 7f7a99600640 1 -- 172.21.15.97:0/652160585 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557d3197ace0 con 0x557d31a376e0 2024-03-11T20:00:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.829+0000 7f7a99600640 10 client.25693 handle_command_reply: tid=0 2024-03-11T20:00:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.829+0000 7f7ab99e1740 1 client.25693 shutdown 2024-03-11T20:00:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.829+0000 7f7ab99e1740 1 -- 172.21.15.97:0/652160585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 msgr2=0x557d31a24b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.829+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/652160585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a24b70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x557d31a1ea90 tx=0x7f7a9c012b60 comp rx=0 tx=0).stop 2024-03-11T20:00:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.830+0000 7f7ab99e1740 1 -- 172.21.15.97:0/652160585 shutdown_connections 2024-03-11T20:00:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.830+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/652160585 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557d31a376e0 0x557d31a39b00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.830+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/652160585 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a27570 0x557d31a250b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.830+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/652160585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a24b70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.830+0000 7f7a99600640 0 client.25693 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.830+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/652160585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a1dff0 0x557d31a24630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.830+0000 7f7ab99e1740 1 -- 172.21.15.97:0/652160585 >> 172.21.15.97:0/652160585 conn(0x557d319e54b0 msgr2=0x557d31a1aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.830+0000 7f7ab99e1740 1 -- 172.21.15.97:0/652160585 shutdown_connections 2024-03-11T20:00:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.830+0000 7f7ab99e1740 1 -- 172.21.15.97:0/652160585 wait complete. 2024-03-11T20:00:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.830+0000 7f7ab99e1740 20 client.25693 trim_cache size 0 max 16384 2024-03-11T20:00:51.840 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.844+0000 7f7ab99e1740 1 Processor -- start 2024-03-11T20:00:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.844+0000 7f7ab99e1740 1 -- start start 2024-03-11T20:00:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.844+0000 7f7ab99e1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a1dff0 0x557d31a26d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.844+0000 7f7ab99e1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a24ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.844+0000 7f7ab99e1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a27570 0x557d31a22f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.844+0000 7f7ab99e1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d316cce60 con 0x557d31a1dff0 2024-03-11T20:00:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.844+0000 7f7ab99e1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d31a145a0 con 0x557d31a27570 2024-03-11T20:00:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.844+0000 7f7ab99e1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d31a1d2c0 con 0x557d31a1ed60 2024-03-11T20:00:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.845+0000 7f7ab4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a1dff0 0x557d31a26d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.845+0000 7f7aaf400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a24ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.845+0000 7f7ab4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a1dff0 0x557d31a26d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41476/0 (socket says 172.21.15.97:41476) 2024-03-11T20:00:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.845+0000 7f7ab4c00640 1 -- 172.21.15.97:0/3093698693 learned_addr learned my addr 172.21.15.97:0/3093698693 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.845+0000 7f7ab4c00640 1 --2- 172.21.15.97:0/3093698693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a27570 0x557d31a22f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.845+0000 7f7aaf400640 1 -- 172.21.15.97:0/3093698693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a27570 msgr2=0x557d31a22f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.845+0000 7f7aaf400640 1 --2- 172.21.15.97:0/3093698693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a27570 0x557d31a22f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.845+0000 7f7aaf400640 1 -- 172.21.15.97:0/3093698693 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a1dff0 msgr2=0x557d31a26d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.845+0000 7f7aaf400640 1 --2- 172.21.15.97:0/3093698693 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a1dff0 0x557d31a26d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.845+0000 7f7aaf400640 1 -- 172.21.15.97:0/3093698693 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d3191f530 con 0x557d31a1ed60 2024-03-11T20:00:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.845+0000 7f7aaf400640 1 --2- 172.21.15.97:0/3093698693 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a24ed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c012420 tx=0x7f7a9c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c32a45b98be96ccd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:51.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.846+0000 7f7a9be00640 1 -- 172.21.15.97:0/3093698693 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c017040 con 0x557d31a1ed60 2024-03-11T20:00:51.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.846+0000 7f7a9be00640 1 -- 172.21.15.97:0/3093698693 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a9c00fdd0 con 0x557d31a1ed60 2024-03-11T20:00:51.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.846+0000 7f7a9be00640 1 -- 172.21.15.97:0/3093698693 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c005320 con 0x557d31a1ed60 2024-03-11T20:00:51.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.846+0000 7f7ab99e1740 1 -- 172.21.15.97:0/3093698693 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 msgr2=0x557d31a24ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:51.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.846+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/3093698693 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a24ed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c012420 tx=0x7f7a9c00f790 comp rx=0 tx=0).stop 2024-03-11T20:00:51.846 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.846+0000 7f7ab99e1740 1 -- 172.21.15.97:0/3093698693 shutdown_connections 2024-03-11T20:00:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.846+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/3093698693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a27570 0x557d31a22f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.846+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/3093698693 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a24ed0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.847+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/3093698693 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a1dff0 0x557d31a26d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.847+0000 7f7ab99e1740 1 -- 172.21.15.97:0/3093698693 >> 172.21.15.97:0/3093698693 conn(0x557d319e54b0 msgr2=0x557d31a18810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.847+0000 7f7ab99e1740 1 -- 172.21.15.97:0/3093698693 shutdown_connections 2024-03-11T20:00:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.847+0000 7f7ab99e1740 1 -- 172.21.15.97:0/3093698693 wait complete. 2024-03-11T20:00:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.847+0000 7f7ab99e1740 1 -- start start 2024-03-11T20:00:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.847+0000 7f7ab99e1740 1 Processor -- start 2024-03-11T20:00:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.847+0000 7f7ab99e1740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a1dff0 0x557d31a2ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab99e1740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a2cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab99e1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a27570 0x557d31a2ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab99e1740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d31a2e3c0 con 0x557d31a1dff0 2024-03-11T20:00:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab99e1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d31a2e530 con 0x557d31a27570 2024-03-11T20:00:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab99e1740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d31a2e6f0 con 0x557d31a1ed60 2024-03-11T20:00:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab99e1740 10 client.0 _ll_register_callbacks cb 0x7f7a9c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:00:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a1dff0 0x557d31a2ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab4c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a1dff0 0x557d31a2ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:41480/0 (socket says 172.21.15.97:41480) 2024-03-11T20:00:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab4c00640 1 -- 172.21.15.97:0/1307673728 learned_addr learned my addr 172.21.15.97:0/1307673728 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:00:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7aaf400640 1 --2- 172.21.15.97:0/1307673728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a2cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab4c00640 1 --2- 172.21.15.97:0/1307673728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a27570 0x557d31a2ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab4c00640 1 -- 172.21.15.97:0/1307673728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 msgr2=0x557d31a2cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab4c00640 1 --2- 172.21.15.97:0/1307673728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a2cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab4c00640 1 -- 172.21.15.97:0/1307673728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a27570 msgr2=0x557d31a2ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:00:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab4c00640 1 --2- 172.21.15.97:0/1307673728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a27570 0x557d31a2ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.848+0000 7f7ab4c00640 1 -- 172.21.15.97:0/1307673728 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9c021040 con 0x557d31a1dff0 2024-03-11T20:00:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.849+0000 7f7ab4c00640 1 --2- 172.21.15.97:0/1307673728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a1dff0 0x557d31a2ca00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7aa802d5e0 tx=0x7f7aa8041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c8f6b2e0574098 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:00:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.849+0000 7f7a99600640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T20:00:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.849+0000 7f7a99600640 1 -- 172.21.15.97:0/1307673728 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa805d020 con 0x557d31a1dff0 2024-03-11T20:00:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.849+0000 7f7ab99e1740 1 -- 172.21.15.97:0/1307673728 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d319a9d30 con 0x557d31a1dff0 2024-03-11T20:00:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.849+0000 7f7a99600640 1 -- 172.21.15.97:0/1307673728 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f7aa8035b70 con 0x557d31a1dff0 2024-03-11T20:00:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.849+0000 7f7a99600640 1 -- 172.21.15.97:0/1307673728 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7aa804d070 con 0x557d31a1dff0 2024-03-11T20:00:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.849+0000 7f7ab99e1740 10 client.15594 fetch_fsmap learned FSMap version 29 2024-03-11T20:00:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.849+0000 7f7ab99e1740 1 -- 172.21.15.97:0/1307673728 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557d319452c0 con 0x557d31a1dff0 2024-03-11T20:00:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.849+0000 7f7a99600640 1 -- 172.21.15.97:0/1307673728 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557d319a9d30 con 0x557d31a1dff0 2024-03-11T20:00:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.849+0000 7f7a99600640 1 -- 172.21.15.97:0/1307673728 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7aa8058d20 con 0x557d31a1dff0 2024-03-11T20:00:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.850+0000 7f7ab99e1740 10 client.15594 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:00:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.850+0000 7f7ab99e1740 10 client.15594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:00:51.850 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.850+0000 7f7ab99e1740 20 client.15594 populate_metadata read hostname 'smithi097' 2024-03-11T20:00:51.850 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.850+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/1307673728 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557d31a8eca0 0x557d31a91090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:00:51.850 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.850+0000 7f7ab99e1740 4 client.15594 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T20:00:51.850 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.850+0000 7f7ab99e1740 1 -- 172.21.15.97:0/1307673728 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557d31684600 con 0x557d31a8eca0 2024-03-11T20:00:51.850 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.850+0000 7f7aaf400640 1 --2- 172.21.15.97:0/1307673728 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557d31a8eca0 0x557d31a91090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:00:51.850 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.851+0000 7f7aaf400640 1 --2- 172.21.15.97:0/1307673728 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557d31a8eca0 0x557d31a91090 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db9d689f880d4e5d server_cookie=2112fb1845fa8229 in_seq=0 out_seq=0 2024-03-11T20:00:51.850 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.851+0000 7f7a99600640 10 client.15594 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:51.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.861+0000 7f7a99600640 1 -- 172.21.15.97:0/1307673728 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557d31684600 con 0x557d31a8eca0 2024-03-11T20:00:51.857 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.861+0000 7f7a99600640 10 client.15594 handle_command_reply: tid=0 2024-03-11T20:00:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.861+0000 7f7ab99e1740 1 client.15594 shutdown 2024-03-11T20:00:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.861+0000 7f7ab99e1740 1 -- 172.21.15.97:0/1307673728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a1dff0 msgr2=0x557d31a2ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.861+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/1307673728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a1dff0 0x557d31a2ca00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7aa802d5e0 tx=0x7f7aa8041040 comp rx=0 tx=0).stop 2024-03-11T20:00:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.862+0000 7f7ab99e1740 1 -- 172.21.15.97:0/1307673728 shutdown_connections 2024-03-11T20:00:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.862+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/1307673728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d31a27570 0x557d31a2ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.862+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/1307673728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x557d31a1ed60 0x557d31a2cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.862+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/1307673728 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x557d31a8eca0 0x557d31a91090 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.862+0000 7f7ab99e1740 1 --2- 172.21.15.97:0/1307673728 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x557d31a1dff0 0x557d31a2ca00 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.862+0000 7f7ab99e1740 1 -- 172.21.15.97:0/1307673728 >> 172.21.15.97:0/1307673728 conn(0x557d319e54b0 msgr2=0x557d31a22f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.862+0000 7f7a99600640 0 client.15594 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:00:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.862+0000 7f7ab99e1740 1 -- 172.21.15.97:0/1307673728 shutdown_connections 2024-03-11T20:00:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.862+0000 7f7ab99e1740 1 -- 172.21.15.97:0/1307673728 wait complete. 2024-03-11T20:00:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.862+0000 7f7ab99e1740 20 client.15594 trim_cache size 0 max 16384 2024-03-11T20:00:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-03-11T20:00:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-03-11T20:00:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-03-11T20:00:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-03-11T20:00:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.862+0000 7f7ab5600640 1 -- 172.21.15.97:0/619171021 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f7a940a1b90 msgr2=0x7f7a940a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.862+0000 7f7ab5600640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f7a940a1b90 0x7f7a940a4050 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0096ba0 tx=0x7f7aa00d17c0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.863+0000 7f7ab5600640 1 -- 172.21.15.97:0/619171021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ab00ff5e0 msgr2=0x7f7ab0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:00:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.863+0000 7f7ab5600640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ab00ff5e0 0x7f7ab0101280 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7aa000e9c0 tx=0x7f7aa000ee80 comp rx=0 tx=0).stop 2024-03-11T20:00:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.863+0000 7f7ab5600640 1 -- 172.21.15.97:0/619171021 shutdown_connections 2024-03-11T20:00:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.863+0000 7f7ab5600640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f7a940a1b90 0x7f7a940a4050 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.863+0000 7f7ab5600640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ab0105680 0x7f7ab0101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.863+0000 7f7ab5600640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ab00fffa0 0x7f7ab01017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.863+0000 7f7ab5600640 1 --2- 172.21.15.97:0/619171021 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ab00ff5e0 0x7f7ab0101280 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:00:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.863+0000 7f7ab5600640 1 -- 172.21.15.97:0/619171021 >> 172.21.15.97:0/619171021 conn(0x7f7ab00763c0 msgr2=0x7f7ab00fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:00:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.863+0000 7f7ab5600640 1 -- 172.21.15.97:0/619171021 shutdown_connections 2024-03-11T20:00:51.861 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:00:51.863+0000 7f7ab5600640 1 -- 172.21.15.97:0/619171021 wait complete. 2024-03-11T20:00:52.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:51 smithi097 ceph-mon[31898]: pgmap v50: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 864 KiB/s wr, 15 op/s 2024-03-11T20:00:52.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:51 smithi114 ceph-mon[36376]: pgmap v50: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 864 KiB/s wr, 15 op/s 2024-03-11T20:00:52.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:51 smithi161 ceph-mon[36332]: pgmap v50: 129 pgs: 129 active+clean; 2.5 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 864 KiB/s wr, 15 op/s 2024-03-11T20:00:54.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:53 smithi097 ceph-mon[31898]: pgmap v51: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.3 MiB/s wr, 23 op/s 2024-03-11T20:00:54.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:53 smithi114 ceph-mon[36376]: pgmap v51: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.3 MiB/s wr, 23 op/s 2024-03-11T20:00:54.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:53 smithi161 ceph-mon[36332]: pgmap v51: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.3 MiB/s wr, 23 op/s 2024-03-11T20:00:56.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:55 smithi097 ceph-mon[31898]: pgmap v52: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 864 KiB/s wr, 14 op/s 2024-03-11T20:00:56.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:55 smithi114 ceph-mon[36376]: pgmap v52: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 864 KiB/s wr, 14 op/s 2024-03-11T20:00:56.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:55 smithi161 ceph-mon[36332]: pgmap v52: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 864 KiB/s wr, 14 op/s 2024-03-11T20:00:58.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:57 smithi097 ceph-mon[31898]: pgmap v53: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 864 KiB/s wr, 14 op/s 2024-03-11T20:00:58.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:57 smithi114 ceph-mon[36376]: pgmap v53: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 864 KiB/s wr, 14 op/s 2024-03-11T20:00:58.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:57 smithi161 ceph-mon[36332]: pgmap v53: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 864 KiB/s wr, 14 op/s 2024-03-11T20:01:00.082 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:00:59 smithi161 ceph-mon[36332]: pgmap v54: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.3 MiB/s wr, 20 op/s 2024-03-11T20:01:00.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:00:59 smithi097 ceph-mon[31898]: pgmap v54: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.3 MiB/s wr, 20 op/s 2024-03-11T20:01:00.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:00:59 smithi114 ceph-mon[36376]: pgmap v54: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.3 MiB/s wr, 20 op/s 2024-03-11T20:01:02.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:01 smithi097 ceph-mon[31898]: pgmap v55: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 863 KiB/s wr, 13 op/s 2024-03-11T20:01:02.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:01 smithi114 ceph-mon[36376]: pgmap v55: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 863 KiB/s wr, 13 op/s 2024-03-11T20:01:02.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:01 smithi161 ceph-mon[36332]: pgmap v55: 129 pgs: 129 active+clean; 2.6 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 863 KiB/s wr, 13 op/s 2024-03-11T20:01:04.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:03 smithi097 ceph-mon[31898]: pgmap v56: 129 pgs: 129 active+clean; 213 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 1.4 MiB/s wr, 63 op/s 2024-03-11T20:01:04.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:03 smithi114 ceph-mon[36376]: pgmap v56: 129 pgs: 129 active+clean; 213 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 1.4 MiB/s wr, 63 op/s 2024-03-11T20:01:04.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:03 smithi161 ceph-mon[36332]: pgmap v56: 129 pgs: 129 active+clean; 213 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 1.4 MiB/s wr, 63 op/s 2024-03-11T20:01:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.023+0000 7f365b200640 1 -- 172.21.15.97:0/2825659995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c103d90 msgr2=0x7f365c107dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.023+0000 7f365b200640 1 --2- 172.21.15.97:0/2825659995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c103d90 0x7f365c107dd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3648009890 tx=0x7f364802bb70 comp rx=0 tx=0).stop 2024-03-11T20:01:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.023+0000 7f365b200640 1 -- 172.21.15.97:0/2825659995 shutdown_connections 2024-03-11T20:01:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.023+0000 7f365b200640 1 --2- 172.21.15.97:0/2825659995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c1084b0 0x7f365c110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.023+0000 7f365b200640 1 --2- 172.21.15.97:0/2825659995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c103d90 0x7f365c107dd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.023+0000 7f365b200640 1 --2- 172.21.15.97:0/2825659995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c1033e0 0x7f365c1037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.023+0000 7f365b200640 1 -- 172.21.15.97:0/2825659995 >> 172.21.15.97:0/2825659995 conn(0x7f365c0fec60 msgr2=0x7f365c101080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.024+0000 7f365b200640 1 -- 172.21.15.97:0/2825659995 shutdown_connections 2024-03-11T20:01:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.024+0000 7f365b200640 1 -- 172.21.15.97:0/2825659995 wait complete. 2024-03-11T20:01:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.025+0000 7f365b200640 1 Processor -- start 2024-03-11T20:01:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.025+0000 7f365b200640 1 -- start start 2024-03-11T20:01:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.025+0000 7f365b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c1033e0 0x7f365c19f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.025+0000 7f365b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c1084b0 0x7f365c19fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.025+0000 7f365b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c199690 0x7f365c199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.025+0000 7f365b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c19a160 con 0x7f365c199690 2024-03-11T20:01:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.025+0000 7f365b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c19a2d0 con 0x7f365c1084b0 2024-03-11T20:01:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.025+0000 7f365b200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c19a490 con 0x7f365c1033e0 2024-03-11T20:01:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.025+0000 7f3659400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c1084b0 0x7f365c19fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.025+0000 7f3659400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c1084b0 0x7f365c19fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:54896/0 (socket says 172.21.15.97:54896) 2024-03-11T20:01:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.025+0000 7f3659e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c1033e0 0x7f365c19f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.025+0000 7f3659400640 1 -- 172.21.15.97:0/766901774 learned_addr learned my addr 172.21.15.97:0/766901774 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.026+0000 7f365a800640 1 --2- 172.21.15.97:0/766901774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c199690 0x7f365c199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.026+0000 7f3659400640 1 -- 172.21.15.97:0/766901774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c1033e0 msgr2=0x7f365c19f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.026+0000 7f3659400640 1 --2- 172.21.15.97:0/766901774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c1033e0 0x7f365c19f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.026+0000 7f3659400640 1 -- 172.21.15.97:0/766901774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c199690 msgr2=0x7f365c199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.026+0000 7f3659400640 1 --2- 172.21.15.97:0/766901774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c199690 0x7f365c199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.026+0000 7f3659400640 1 -- 172.21.15.97:0/766901774 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3648009490 con 0x7f365c1084b0 2024-03-11T20:01:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.026+0000 7f3659400640 1 --2- 172.21.15.97:0/766901774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c1084b0 0x7f365c19fb10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3648000c00 tx=0x7f36480045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.027+0000 7f3652a00640 1 -- 172.21.15.97:0/766901774 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3648034a10 con 0x7f365c1084b0 2024-03-11T20:01:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.027+0000 7f3652a00640 1 -- 172.21.15.97:0/766901774 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f364802c9f0 con 0x7f365c1084b0 2024-03-11T20:01:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.027+0000 7f365b200640 1 -- 172.21.15.97:0/766901774 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f365c1a63d0 con 0x7f365c1084b0 2024-03-11T20:01:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.027+0000 7f3652a00640 1 -- 172.21.15.97:0/766901774 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36480035f0 con 0x7f365c1084b0 2024-03-11T20:01:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.027+0000 7f365b200640 1 -- 172.21.15.97:0/766901774 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365c1a6790 con 0x7f365c1084b0 2024-03-11T20:01:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.031+0000 7f3652a00640 1 -- 172.21.15.97:0/766901774 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f364803b040 con 0x7f365c1084b0 2024-03-11T20:01:05.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.032+0000 7f36637f7740 1 Processor -- start 2024-03-11T20:01:05.027 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.032+0000 7f36637f7740 1 -- start start 2024-03-11T20:01:05.028 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.032+0000 7f36637f7740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c84d60 0x5588d8c83ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.028 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.032+0000 7f36637f7740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c83ff0 0x5588d8c84470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.028 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.032+0000 7f36637f7740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c8f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.032+0000 7f36637f7740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d8c3f620 con 0x5588d8c84d60 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.032+0000 7f36637f7740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d8bf18c0 con 0x5588d8c83ff0 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.032+0000 7f36637f7740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d8beedf0 con 0x5588d8c8d570 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f3659e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c83ff0 0x5588d8c84470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f3659400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c8f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f3659e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c83ff0 0x5588d8c84470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:54908/0 (socket says 172.21.15.97:54908) 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f3659e00640 1 -- 172.21.15.97:0/152455374 learned_addr learned my addr 172.21.15.97:0/152455374 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f3652a00640 1 --2- 172.21.15.97:0/766901774 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f36400a1c10 0x7f36400a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f3659e00640 1 -- 172.21.15.97:0/152455374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 msgr2=0x5588d8c8f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f3659e00640 1 --2- 172.21.15.97:0/152455374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c8f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f3659e00640 1 -- 172.21.15.97:0/152455374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c84d60 msgr2=0x5588d8c83ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f365a800640 1 --2- 172.21.15.97:0/152455374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c84d60 0x5588d8c83ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f3652a00640 1 -- 172.21.15.97:0/766901774 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f36480ef4a0 con 0x7f365c1084b0 2024-03-11T20:01:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f3659e00640 1 --2- 172.21.15.97:0/152455374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c84d60 0x5588d8c83ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f3659e00640 1 -- 172.21.15.97:0/152455374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588d8b6bb00 con 0x5588d8c83ff0 2024-03-11T20:01:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f365a800640 1 --2- 172.21.15.97:0/152455374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c84d60 0x5588d8c83ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T20:01:05.030 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.033+0000 7f365a800640 1 --2- 172.21.15.97:0/766901774 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f36400a1c10 0x7f36400a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.034+0000 7f3659e00640 1 --2- 172.21.15.97:0/152455374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c83ff0 0x5588d8c84470 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f364c009cf0 tx=0x7f364c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5f930185da7dc8e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.034+0000 7f365a800640 1 --2- 172.21.15.97:0/766901774 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f36400a1c10 0x7f36400a40d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3654005fd0 tx=0x7f3654004a40 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.034+0000 7f3647000640 1 -- 172.21.15.97:0/152455374 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c009ea0 con 0x5588d8c83ff0 2024-03-11T20:01:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.034+0000 7f3647000640 1 -- 172.21.15.97:0/152455374 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f364c004230 con 0x5588d8c83ff0 2024-03-11T20:01:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.034+0000 7f3647000640 1 -- 172.21.15.97:0/152455374 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c019880 con 0x5588d8c83ff0 2024-03-11T20:01:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.035+0000 7f36637f7740 1 -- 172.21.15.97:0/152455374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c83ff0 msgr2=0x5588d8c84470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.035+0000 7f36637f7740 1 --2- 172.21.15.97:0/152455374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c83ff0 0x5588d8c84470 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f364c009cf0 tx=0x7f364c011810 comp rx=0 tx=0).stop 2024-03-11T20:01:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.035+0000 7f36637f7740 1 -- 172.21.15.97:0/152455374 shutdown_connections 2024-03-11T20:01:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.035+0000 7f36637f7740 1 --2- 172.21.15.97:0/152455374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c8f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.035+0000 7f36637f7740 1 --2- 172.21.15.97:0/152455374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c83ff0 0x5588d8c84470 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.035+0000 7f36637f7740 1 --2- 172.21.15.97:0/152455374 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c84d60 0x5588d8c83ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.035+0000 7f36637f7740 1 -- 172.21.15.97:0/152455374 >> 172.21.15.97:0/152455374 conn(0x5588d8c4b4b0 msgr2=0x5588d8c7ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.036+0000 7f36637f7740 1 -- 172.21.15.97:0/152455374 shutdown_connections 2024-03-11T20:01:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.036+0000 7f36637f7740 1 -- 172.21.15.97:0/152455374 wait complete. 2024-03-11T20:01:05.032 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.036+0000 7f36637f7740 1 -- start start 2024-03-11T20:01:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.036+0000 7f36637f7740 1 Processor -- start 2024-03-11T20:01:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f36637f7740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c83ff0 0x5588d8c8a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f36637f7740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c84d60 0x5588d8c8ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f36637f7740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c8b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f36637f7740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d8bea5d0 con 0x5588d8c84d60 2024-03-11T20:01:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f36637f7740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d8b8fb50 con 0x5588d8c83ff0 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f36637f7740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d8b8f7f0 con 0x5588d8c8d570 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f36637f7740 10 client.0 _ll_register_callbacks cb 0x5588d8ba3550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f3659e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c83ff0 0x5588d8c8a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f365a800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c8b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f3659e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c83ff0 0x5588d8c8a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:54912/0 (socket says 172.21.15.97:54912) 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f3659e00640 1 -- 172.21.15.97:0/458719002 learned_addr learned my addr 172.21.15.97:0/458719002 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f3659400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c84d60 0x5588d8c8ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f365a800640 1 -- 172.21.15.97:0/458719002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c83ff0 msgr2=0x5588d8c8a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f365a800640 1 --2- 172.21.15.97:0/458719002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c83ff0 0x5588d8c8a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f365a800640 1 -- 172.21.15.97:0/458719002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c84d60 msgr2=0x5588d8c8ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f365a800640 1 --2- 172.21.15.97:0/458719002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c84d60 0x5588d8c8ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.037+0000 7f365a800640 1 -- 172.21.15.97:0/458719002 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c009950 con 0x5588d8c8d570 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.038+0000 7f365a800640 1 --2- 172.21.15.97:0/458719002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c8b0b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f365400ed50 tx=0x7f3654004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9861b2668a8a6cfa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:05.035 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.038+0000 7f363b400640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-11T20:01:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.038+0000 7f363b400640 1 -- 172.21.15.97:0/458719002 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3654025600 con 0x5588d8c8d570 2024-03-11T20:01:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.038+0000 7f36637f7740 1 -- 172.21.15.97:0/458719002 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5588d8bbccd0 con 0x5588d8c8d570 2024-03-11T20:01:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.038+0000 7f363b400640 1 -- 172.21.15.97:0/458719002 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f3654025760 con 0x5588d8c8d570 2024-03-11T20:01:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.039+0000 7f363b400640 1 -- 172.21.15.97:0/458719002 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f365400ce50 con 0x5588d8c8d570 2024-03-11T20:01:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.039+0000 7f36637f7740 10 client.25711 fetch_fsmap learned FSMap version 29 2024-03-11T20:01:05.036 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.039+0000 7f36637f7740 1 -- 172.21.15.97:0/458719002 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5588d8be8c50 con 0x5588d8c8d570 2024-03-11T20:01:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.039+0000 7f363b400640 1 -- 172.21.15.97:0/458719002 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5588d8bbccd0 con 0x5588d8c8d570 2024-03-11T20:01:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.040+0000 7f363b400640 1 -- 172.21.15.97:0/458719002 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3654018aa0 con 0x5588d8c8d570 2024-03-11T20:01:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.040+0000 7f36637f7740 10 client.25711 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:01:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.040+0000 7f36637f7740 10 client.25711 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:01:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.040+0000 7f36637f7740 20 client.25711 populate_metadata read hostname 'smithi097' 2024-03-11T20:01:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.040+0000 7f36637f7740 1 --2- 172.21.15.97:0/458719002 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5588d8c9d5d0 0x5588d8c9f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.040+0000 7f36637f7740 4 client.25711 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T20:01:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.040+0000 7f36637f7740 1 -- 172.21.15.97:0/458719002 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588d8be0ce0 con 0x5588d8c9d5d0 2024-03-11T20:01:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.040+0000 7f3659e00640 1 --2- 172.21.15.97:0/458719002 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5588d8c9d5d0 0x5588d8c9f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.041+0000 7f363b400640 10 client.25711 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:01:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.041+0000 7f3659e00640 1 --2- 172.21.15.97:0/458719002 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5588d8c9d5d0 0x5588d8c9f9f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2731adfac0642569 server_cookie=c1a8564a1ed42686 in_seq=0 out_seq=0 2024-03-11T20:01:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.041+0000 7f363b400640 1 -- 172.21.15.97:0/458719002 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5588d8be0ce0 con 0x5588d8c9d5d0 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.041+0000 7f363b400640 10 client.25711 handle_command_reply: tid=0 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.041+0000 7f36637f7740 1 client.25711 shutdown 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.042+0000 7f36637f7740 1 -- 172.21.15.97:0/458719002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 msgr2=0x5588d8c8b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.042+0000 7f36637f7740 1 --2- 172.21.15.97:0/458719002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c8b0b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f365400ed50 tx=0x7f3654004ab0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.042+0000 7f36637f7740 1 -- 172.21.15.97:0/458719002 shutdown_connections 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.042+0000 7f36637f7740 1 --2- 172.21.15.97:0/458719002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c8b0b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.042+0000 7f36637f7740 1 --2- 172.21.15.97:0/458719002 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5588d8c9d5d0 0x5588d8c9f9f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.042+0000 7f36637f7740 1 --2- 172.21.15.97:0/458719002 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c84d60 0x5588d8c8ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.042+0000 7f36637f7740 1 --2- 172.21.15.97:0/458719002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c83ff0 0x5588d8c8a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.042+0000 7f36637f7740 1 -- 172.21.15.97:0/458719002 >> 172.21.15.97:0/458719002 conn(0x5588d8c4b4b0 msgr2=0x5588d8c80a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.042+0000 7f363b400640 0 client.25711 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.042+0000 7f36637f7740 1 -- 172.21.15.97:0/458719002 shutdown_connections 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.042+0000 7f36637f7740 1 -- 172.21.15.97:0/458719002 wait complete. 2024-03-11T20:01:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.042+0000 7f36637f7740 20 client.25711 trim_cache size 0 max 16384 2024-03-11T20:01:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.060+0000 7f36637f7740 1 Processor -- start 2024-03-11T20:01:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.060+0000 7f36637f7740 1 -- start start 2024-03-11T20:01:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.060+0000 7f36637f7740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c83ff0 0x5588d8c8cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.060+0000 7f36637f7740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c84d60 0x5588d8c8aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.060+0000 7f36637f7740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c94390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.055 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.060+0000 7f36637f7740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d8932e60 con 0x5588d8c83ff0 2024-03-11T20:01:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.060+0000 7f36637f7740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d8c83540 con 0x5588d8c84d60 2024-03-11T20:01:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.060+0000 7f36637f7740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d8c88b80 con 0x5588d8c8d570 2024-03-11T20:01:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.061+0000 7f365a800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c94390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.061+0000 7f3659400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c84d60 0x5588d8c8aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.061+0000 7f365a800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c94390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:38446/0 (socket says 172.21.15.97:38446) 2024-03-11T20:01:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.061+0000 7f365a800640 1 -- 172.21.15.97:0/1175937930 learned_addr learned my addr 172.21.15.97:0/1175937930 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.061+0000 7f3659e00640 1 --2- 172.21.15.97:0/1175937930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c83ff0 0x5588d8c8cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.061+0000 7f365a800640 1 -- 172.21.15.97:0/1175937930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c84d60 msgr2=0x5588d8c8aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.061+0000 7f365a800640 1 --2- 172.21.15.97:0/1175937930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c84d60 0x5588d8c8aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.061+0000 7f365a800640 1 -- 172.21.15.97:0/1175937930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c83ff0 msgr2=0x5588d8c8cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.061+0000 7f365a800640 1 --2- 172.21.15.97:0/1175937930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c83ff0 0x5588d8c8cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.061+0000 7f365a800640 1 -- 172.21.15.97:0/1175937930 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588d8b85530 con 0x5588d8c8d570 2024-03-11T20:01:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.061+0000 7f365a800640 1 --2- 172.21.15.97:0/1175937930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c94390 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3654018eb0 tx=0x7f365400edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=661b2edcecd853dd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:05.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.062+0000 7f3646600640 1 -- 172.21.15.97:0/1175937930 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3654018aa0 con 0x5588d8c8d570 2024-03-11T20:01:05.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.062+0000 7f3646600640 1 -- 172.21.15.97:0/1175937930 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3654014040 con 0x5588d8c8d570 2024-03-11T20:01:05.058 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.062+0000 7f3646600640 1 -- 172.21.15.97:0/1175937930 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36540247f0 con 0x5588d8c8d570 2024-03-11T20:01:05.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.063+0000 7f36637f7740 1 -- 172.21.15.97:0/1175937930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 msgr2=0x5588d8c94390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:05.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.063+0000 7f36637f7740 1 --2- 172.21.15.97:0/1175937930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c94390 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3654018eb0 tx=0x7f365400edd0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.059 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.063+0000 7f36637f7740 1 -- 172.21.15.97:0/1175937930 shutdown_connections 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.063+0000 7f36637f7740 1 --2- 172.21.15.97:0/1175937930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c8d570 0x5588d8c94390 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.063+0000 7f36637f7740 1 --2- 172.21.15.97:0/1175937930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c84d60 0x5588d8c8aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.063+0000 7f36637f7740 1 --2- 172.21.15.97:0/1175937930 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c83ff0 0x5588d8c8cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.063+0000 7f36637f7740 1 -- 172.21.15.97:0/1175937930 >> 172.21.15.97:0/1175937930 conn(0x5588d8c4b4b0 msgr2=0x5588d8c7e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.063+0000 7f36637f7740 1 -- 172.21.15.97:0/1175937930 shutdown_connections 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.063+0000 7f36637f7740 1 -- 172.21.15.97:0/1175937930 wait complete. 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.063+0000 7f36637f7740 1 -- start start 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.064+0000 7f36637f7740 1 Processor -- start 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.064+0000 7f36637f7740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c83ff0 0x5588d8c870d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.064+0000 7f36637f7740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c84d60 0x5588d8c87610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.064+0000 7f36637f7740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c8d570 0x5588d8c87b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.064+0000 7f36637f7740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d8c94c50 con 0x5588d8c83ff0 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.064+0000 7f36637f7740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d8cf3b10 con 0x5588d8c8d570 2024-03-11T20:01:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.064+0000 7f36637f7740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588d8cf3c50 con 0x5588d8c84d60 2024-03-11T20:01:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.064+0000 7f36637f7740 10 client.0 _ll_register_callbacks cb 0x7f3654009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:01:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.064+0000 7f365a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c8d570 0x5588d8c87b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.064+0000 7f3659e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c83ff0 0x5588d8c870d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.064+0000 7f365a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c8d570 0x5588d8c87b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:54936/0 (socket says 172.21.15.97:54936) 2024-03-11T20:01:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.064+0000 7f365a800640 1 -- 172.21.15.97:0/1004535745 learned_addr learned my addr 172.21.15.97:0/1004535745 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.065+0000 7f3659400640 1 --2- 172.21.15.97:0/1004535745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c84d60 0x5588d8c87610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.065+0000 7f365a800640 1 -- 172.21.15.97:0/1004535745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c84d60 msgr2=0x5588d8c87610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.065+0000 7f365a800640 1 --2- 172.21.15.97:0/1004535745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c84d60 0x5588d8c87610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.065+0000 7f365a800640 1 -- 172.21.15.97:0/1004535745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c83ff0 msgr2=0x5588d8c870d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.065+0000 7f365a800640 1 --2- 172.21.15.97:0/1004535745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c83ff0 0x5588d8c870d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.065+0000 7f365a800640 1 -- 172.21.15.97:0/1004535745 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588d8be8c50 con 0x5588d8c8d570 2024-03-11T20:01:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.065+0000 7f365a800640 1 --2- 172.21.15.97:0/1004535745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c8d570 0x5588d8c87b50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f365400ed50 tx=0x7f3654018570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e06b30d3b9121dc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.065+0000 7f363b400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-11T20:01:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.066+0000 7f363b400640 1 -- 172.21.15.97:0/1004535745 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3654018810 con 0x5588d8c8d570 2024-03-11T20:01:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.066+0000 7f363b400640 1 -- 172.21.15.97:0/1004535745 <== mon.2 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f3654018970 con 0x5588d8c8d570 2024-03-11T20:01:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.066+0000 7f36637f7740 1 -- 172.21.15.97:0/1004535745 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5588d8c0fd30 con 0x5588d8c8d570 2024-03-11T20:01:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.066+0000 7f363b400640 1 -- 172.21.15.97:0/1004535745 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3654018b70 con 0x5588d8c8d570 2024-03-11T20:01:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.066+0000 7f36637f7740 10 client.25730 fetch_fsmap learned FSMap version 29 2024-03-11T20:01:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.066+0000 7f36637f7740 1 -- 172.21.15.97:0/1004535745 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5588d8bab2c0 con 0x5588d8c8d570 2024-03-11T20:01:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.066+0000 7f363b400640 1 -- 172.21.15.97:0/1004535745 <== mon.2 v2:172.21.15.114:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5588d8c0fd30 con 0x5588d8c8d570 2024-03-11T20:01:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.067+0000 7f363b400640 1 -- 172.21.15.97:0/1004535745 <== mon.2 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f365402d070 con 0x5588d8c8d570 2024-03-11T20:01:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.067+0000 7f36637f7740 10 client.25730 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:01:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.067+0000 7f36637f7740 10 client.25730 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:01:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.067+0000 7f36637f7740 20 client.25730 populate_metadata read hostname 'smithi097' 2024-03-11T20:01:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.067+0000 7f36637f7740 1 --2- 172.21.15.97:0/1004535745 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5588d8cf5010 0x5588d8cf7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.067+0000 7f36637f7740 4 client.25730 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T20:01:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.067+0000 7f36637f7740 1 -- 172.21.15.97:0/1004535745 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5588d88ea600 con 0x5588d8cf5010 2024-03-11T20:01:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.067+0000 7f3659e00640 1 --2- 172.21.15.97:0/1004535745 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5588d8cf5010 0x5588d8cf7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.068+0000 7f363b400640 10 client.25730 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:01:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.068+0000 7f3659e00640 1 --2- 172.21.15.97:0/1004535745 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5588d8cf5010 0x5588d8cf7400 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b720ed8ac936ea9 server_cookie=a67b61dc42c0958d in_seq=0 out_seq=0 2024-03-11T20:01:05.072 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.077+0000 7f363b400640 1 -- 172.21.15.97:0/1004535745 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5588d88ea600 con 0x5588d8cf5010 2024-03-11T20:01:05.072 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.077+0000 7f363b400640 10 client.25730 handle_command_reply: tid=0 2024-03-11T20:01:05.072 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.077+0000 7f36637f7740 1 client.25730 shutdown 2024-03-11T20:01:05.072 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.077+0000 7f36637f7740 1 -- 172.21.15.97:0/1004535745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c8d570 msgr2=0x5588d8c87b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:05.072 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.077+0000 7f36637f7740 1 --2- 172.21.15.97:0/1004535745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c8d570 0x5588d8c87b50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f365400ed50 tx=0x7f3654018570 comp rx=0 tx=0).stop 2024-03-11T20:01:05.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.077+0000 7f36637f7740 1 -- 172.21.15.97:0/1004535745 shutdown_connections 2024-03-11T20:01:05.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.077+0000 7f36637f7740 1 --2- 172.21.15.97:0/1004535745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588d8c8d570 0x5588d8c87b50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.077+0000 7f36637f7740 1 --2- 172.21.15.97:0/1004535745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5588d8c84d60 0x5588d8c87610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.077+0000 7f36637f7740 1 --2- 172.21.15.97:0/1004535745 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5588d8cf5010 0x5588d8cf7400 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.073 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.078+0000 7f36637f7740 1 --2- 172.21.15.97:0/1004535745 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5588d8c83ff0 0x5588d8c870d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.078+0000 7f36637f7740 1 -- 172.21.15.97:0/1004535745 >> 172.21.15.97:0/1004535745 conn(0x5588d8c4b4b0 msgr2=0x5588d8c88cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.078+0000 7f363b400640 0 client.25730 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:01:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.078+0000 7f36637f7740 1 -- 172.21.15.97:0/1004535745 shutdown_connections 2024-03-11T20:01:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.078+0000 7f36637f7740 1 -- 172.21.15.97:0/1004535745 wait complete. 2024-03-11T20:01:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.078+0000 7f36637f7740 20 client.25730 trim_cache size 0 max 16384 2024-03-11T20:01:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-03-11T20:01:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-03-11T20:01:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-03-11T20:01:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-03-11T20:01:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.079+0000 7f365b200640 1 -- 172.21.15.97:0/766901774 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f36400a1c10 msgr2=0x7f36400a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.079+0000 7f365b200640 1 --2- 172.21.15.97:0/766901774 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f36400a1c10 0x7f36400a40d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3654005fd0 tx=0x7f3654004a40 comp rx=0 tx=0).stop 2024-03-11T20:01:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.079+0000 7f365b200640 1 -- 172.21.15.97:0/766901774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c1084b0 msgr2=0x7f365c19fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.079+0000 7f365b200640 1 --2- 172.21.15.97:0/766901774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c1084b0 0x7f365c19fb10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3648000c00 tx=0x7f36480045b0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.079+0000 7f365b200640 1 -- 172.21.15.97:0/766901774 shutdown_connections 2024-03-11T20:01:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.079+0000 7f365b200640 1 --2- 172.21.15.97:0/766901774 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f36400a1c10 0x7f36400a40d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.079+0000 7f365b200640 1 --2- 172.21.15.97:0/766901774 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f365c199690 0x7f365c199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.079+0000 7f365b200640 1 --2- 172.21.15.97:0/766901774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f365c1084b0 0x7f365c19fb10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.079+0000 7f365b200640 1 --2- 172.21.15.97:0/766901774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f365c1033e0 0x7f365c19f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.079+0000 7f365b200640 1 -- 172.21.15.97:0/766901774 >> 172.21.15.97:0/766901774 conn(0x7f365c0fec60 msgr2=0x7f365c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.079+0000 7f365b200640 1 -- 172.21.15.97:0/766901774 shutdown_connections 2024-03-11T20:01:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:05.079+0000 7f365b200640 1 -- 172.21.15.97:0/766901774 wait complete. 2024-03-11T20:01:06.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:05 smithi097 ceph-mon[31898]: pgmap v57: 129 pgs: 129 active+clean; 213 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 1002 KiB/s wr, 55 op/s 2024-03-11T20:01:06.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:05 smithi114 ceph-mon[36376]: pgmap v57: 129 pgs: 129 active+clean; 213 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 1002 KiB/s wr, 55 op/s 2024-03-11T20:01:06.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:05 smithi161 ceph-mon[36332]: pgmap v57: 129 pgs: 129 active+clean; 213 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 1002 KiB/s wr, 55 op/s 2024-03-11T20:01:08.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:07 smithi097 ceph-mon[31898]: pgmap v58: 129 pgs: 129 active+clean; 213 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 1002 KiB/s wr, 55 op/s 2024-03-11T20:01:08.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:07 smithi114 ceph-mon[36376]: pgmap v58: 129 pgs: 129 active+clean; 213 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 1002 KiB/s wr, 55 op/s 2024-03-11T20:01:08.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:07 smithi161 ceph-mon[36332]: pgmap v58: 129 pgs: 129 active+clean; 213 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 1002 KiB/s wr, 55 op/s 2024-03-11T20:01:10.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:09 smithi097 ceph-mon[31898]: pgmap v59: 129 pgs: 129 active+clean; 258 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 15 MiB/s wr, 210 op/s 2024-03-11T20:01:10.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:09 smithi114 ceph-mon[36376]: pgmap v59: 129 pgs: 129 active+clean; 258 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 15 MiB/s wr, 210 op/s 2024-03-11T20:01:10.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:09 smithi161 ceph-mon[36332]: pgmap v59: 129 pgs: 129 active+clean; 258 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 15 MiB/s wr, 210 op/s 2024-03-11T20:01:12.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:11 smithi097 ceph-mon[31898]: pgmap v60: 129 pgs: 129 active+clean; 258 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 14 MiB/s wr, 205 op/s 2024-03-11T20:01:12.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:11 smithi114 ceph-mon[36376]: pgmap v60: 129 pgs: 129 active+clean; 258 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 14 MiB/s wr, 205 op/s 2024-03-11T20:01:12.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:11 smithi161 ceph-mon[36332]: pgmap v60: 129 pgs: 129 active+clean; 258 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 14 MiB/s wr, 205 op/s 2024-03-11T20:01:14.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:13 smithi097 ceph-mon[31898]: pgmap v61: 129 pgs: 129 active+clean; 259 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 15 MiB/s wr, 298 op/s 2024-03-11T20:01:14.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:13 smithi114 ceph-mon[36376]: pgmap v61: 129 pgs: 129 active+clean; 259 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 15 MiB/s wr, 298 op/s 2024-03-11T20:01:14.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:13 smithi161 ceph-mon[36332]: pgmap v61: 129 pgs: 129 active+clean; 259 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 15 MiB/s wr, 298 op/s 2024-03-11T20:01:16.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:15 smithi097 ceph-mon[31898]: pgmap v62: 129 pgs: 129 active+clean; 259 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 15 MiB/s wr, 249 op/s 2024-03-11T20:01:16.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:15 smithi114 ceph-mon[36376]: pgmap v62: 129 pgs: 129 active+clean; 259 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 15 MiB/s wr, 249 op/s 2024-03-11T20:01:16.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:15 smithi161 ceph-mon[36332]: pgmap v62: 129 pgs: 129 active+clean; 259 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 15 MiB/s wr, 249 op/s 2024-03-11T20:01:18.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:17 smithi097 ceph-mon[31898]: pgmap v63: 129 pgs: 129 active+clean; 259 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 15 MiB/s wr, 249 op/s 2024-03-11T20:01:18.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:17 smithi114 ceph-mon[36376]: pgmap v63: 129 pgs: 129 active+clean; 259 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 15 MiB/s wr, 249 op/s 2024-03-11T20:01:18.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.246+0000 7fe6d5400640 1 -- 172.21.15.97:0/513250573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0102a70 msgr2=0x7fe6d010af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:18.241 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.246+0000 7fe6d5400640 1 --2- 172.21.15.97:0/513250573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0102a70 0x7fe6d010af50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009900 tx=0x7fe6bc02bcd0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.246+0000 7fe6d5400640 1 -- 172.21.15.97:0/513250573 shutdown_connections 2024-03-11T20:01:18.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.246+0000 7fe6d5400640 1 --2- 172.21.15.97:0/513250573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6d010b570 0x7fe6d010da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.246+0000 7fe6d5400640 1 --2- 172.21.15.97:0/513250573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0102a70 0x7fe6d010af50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.246+0000 7fe6d5400640 1 --2- 172.21.15.97:0/513250573 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6d0102150 0x7fe6d0102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.246+0000 7fe6d5400640 1 -- 172.21.15.97:0/513250573 >> 172.21.15.97:0/513250573 conn(0x7fe6d00fb990 msgr2=0x7fe6d00fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:18.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.246+0000 7fe6d5400640 1 -- 172.21.15.97:0/513250573 shutdown_connections 2024-03-11T20:01:18.242 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.246+0000 7fe6d5400640 1 -- 172.21.15.97:0/513250573 wait complete. 2024-03-11T20:01:18.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.247+0000 7fe6d5400640 1 Processor -- start 2024-03-11T20:01:18.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.248+0000 7fe6d5400640 1 -- start start 2024-03-11T20:01:18.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.248+0000 7fe6d5400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6d0102150 0x7fe6d019f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.248+0000 7fe6d5400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0102a70 0x7fe6d019fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.248+0000 7fe6d5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6d010b570 0x7fe6d01996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.248+0000 7fe6d5400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d0199e80 con 0x7fe6d0102150 2024-03-11T20:01:18.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.248+0000 7fe6d5400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d0199ff0 con 0x7fe6d010b570 2024-03-11T20:01:18.243 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.248+0000 7fe6d5400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d019a1b0 con 0x7fe6d0102a70 2024-03-11T20:01:18.244 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.248+0000 7fe6cf400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0102a70 0x7fe6d019fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.248+0000 7fe6d4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6d010b570 0x7fe6d01996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.248+0000 7fe6cf400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0102a70 0x7fe6d019fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:53596/0 (socket says 172.21.15.97:53596) 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.248+0000 7fe6cf400640 1 -- 172.21.15.97:0/2291879971 learned_addr learned my addr 172.21.15.97:0/2291879971 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.248+0000 7fe6cfe00640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6d0102150 0x7fe6d019f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.249+0000 7fe6cf400640 1 -- 172.21.15.97:0/2291879971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6d010b570 msgr2=0x7fe6d01996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.249+0000 7fe6cf400640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6d010b570 0x7fe6d01996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.249+0000 7fe6cf400640 1 -- 172.21.15.97:0/2291879971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6d0102150 msgr2=0x7fe6d019f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.249+0000 7fe6cf400640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6d0102150 0x7fe6d019f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.249+0000 7fe6cf400640 1 -- 172.21.15.97:0/2291879971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c8009840 con 0x7fe6d0102a70 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.249+0000 7fe6cfe00640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6d0102150 0x7fe6d019f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.249+0000 7fe6cf400640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0102a70 0x7fe6d019fb70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc000c00 tx=0x7fe6bc0045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.249+0000 7fe6ccc00640 1 -- 172.21.15.97:0/2291879971 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc02ca90 con 0x7fe6d0102a70 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.249+0000 7fe6ccc00640 1 -- 172.21.15.97:0/2291879971 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6bc02cbf0 con 0x7fe6d0102a70 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.249+0000 7fe6d5400640 1 -- 172.21.15.97:0/2291879971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6bc009560 con 0x7fe6d0102a70 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.249+0000 7fe6ccc00640 1 -- 172.21.15.97:0/2291879971 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0039e0 con 0x7fe6d0102a70 2024-03-11T20:01:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.250+0000 7fe6d5400640 1 -- 172.21.15.97:0/2291879971 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d01a65b0 con 0x7fe6d0102a70 2024-03-11T20:01:18.248 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.253+0000 7fe6ccc00640 1 -- 172.21.15.97:0/2291879971 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe6bc005360 con 0x7fe6d0102a70 2024-03-11T20:01:18.248 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.253+0000 7fe6d95e0740 1 Processor -- start 2024-03-11T20:01:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.254+0000 7fe6d95e0740 1 -- start start 2024-03-11T20:01:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.254+0000 7fe6d95e0740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 0x55fdc02fdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.254+0000 7fe6d95e0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc02fdff0 0x55fdc02fe470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.254+0000 7fe6d95e0740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc0307570 0x55fdc0309960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.254+0000 7fe6d95e0740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc02b9620 con 0x55fdc0307570 2024-03-11T20:01:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.254+0000 7fe6d95e0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc026b8c0 con 0x55fdc02fdff0 2024-03-11T20:01:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.254+0000 7fe6d95e0740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc0268df0 con 0x55fdc02fed60 2024-03-11T20:01:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.254+0000 7fe6d4a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 0x55fdc02fdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.254+0000 7fe6d4a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 0x55fdc02fdab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:53602/0 (socket says 172.21.15.97:53602) 2024-03-11T20:01:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.254+0000 7fe6d4a00640 1 -- 172.21.15.97:0/674650606 learned_addr learned my addr 172.21.15.97:0/674650606 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.254+0000 7fe6cfe00640 1 --2- 172.21.15.97:0/674650606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc02fdff0 0x55fdc02fe470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.255+0000 7fe6d4a00640 1 -- 172.21.15.97:0/674650606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc02fdff0 msgr2=0x55fdc02fe470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.255+0000 7fe6d4a00640 1 --2- 172.21.15.97:0/674650606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc02fdff0 0x55fdc02fe470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.255+0000 7fe6d4a00640 1 -- 172.21.15.97:0/674650606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc0307570 msgr2=0x55fdc0309960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:18.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.255+0000 7fe6d4a00640 1 --2- 172.21.15.97:0/674650606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc0307570 0x55fdc0309960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.255+0000 7fe6ccc00640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe6b40a1c10 0x7fe6b40a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.255+0000 7fe6d4a00640 1 -- 172.21.15.97:0/674650606 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdc01e5b00 con 0x55fdc02fed60 2024-03-11T20:01:18.251 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.255+0000 7fe6ccc00640 1 -- 172.21.15.97:0/2291879971 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7fe6bc0efc00 con 0x7fe6d0102a70 2024-03-11T20:01:18.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.255+0000 7fe6d4a00640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe6b40a1c10 0x7fe6b40a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.255+0000 7fe6d4a00640 1 --2- 172.21.15.97:0/674650606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 0x55fdc02fdab0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8011570 tx=0x7fe6c8011a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d000eb3a98b1de7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:18.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.256+0000 7fe6c5200640 1 -- 172.21.15.97:0/674650606 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8003970 con 0x55fdc02fed60 2024-03-11T20:01:18.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.256+0000 7fe6c5200640 1 -- 172.21.15.97:0/674650606 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6c800fb20 con 0x55fdc02fed60 2024-03-11T20:01:18.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.256+0000 7fe6c5200640 1 -- 172.21.15.97:0/674650606 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8019a20 con 0x55fdc02fed60 2024-03-11T20:01:18.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.257+0000 7fe6d95e0740 1 -- 172.21.15.97:0/674650606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 msgr2=0x55fdc02fdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:18.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.257+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/674650606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 0x55fdc02fdab0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8011570 tx=0x7fe6c8011a30 comp rx=0 tx=0).stop 2024-03-11T20:01:18.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.257+0000 7fe6d4a00640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe6b40a1c10 0x7fe6b40a40d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8021000 tx=0x7fe6c8022000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:18.252 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.257+0000 7fe6d95e0740 1 -- 172.21.15.97:0/674650606 shutdown_connections 2024-03-11T20:01:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.257+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/674650606 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc0307570 0x55fdc0309960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.257+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/674650606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc02fdff0 0x55fdc02fe470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.257+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/674650606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 0x55fdc02fdab0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.257+0000 7fe6d95e0740 1 -- 172.21.15.97:0/674650606 >> 172.21.15.97:0/674650606 conn(0x55fdc02c54b0 msgr2=0x55fdc02f8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.257+0000 7fe6d95e0740 1 -- 172.21.15.97:0/674650606 shutdown_connections 2024-03-11T20:01:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.257+0000 7fe6d95e0740 1 -- 172.21.15.97:0/674650606 wait complete. 2024-03-11T20:01:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.258+0000 7fe6d95e0740 1 -- start start 2024-03-11T20:01:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.258+0000 7fe6d95e0740 1 Processor -- start 2024-03-11T20:01:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.258+0000 7fe6d95e0740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fdff0 0x55fdc03045f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.258+0000 7fe6d95e0740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 0x55fdc0304b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.258+0000 7fe6d95e0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc0307570 0x55fdc0305070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.258+0000 7fe6d95e0740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc02645d0 con 0x55fdc02fdff0 2024-03-11T20:01:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6d95e0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc0209b50 con 0x55fdc0307570 2024-03-11T20:01:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6d95e0740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc02097f0 con 0x55fdc02fed60 2024-03-11T20:01:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6d95e0740 10 client.0 _ll_register_callbacks cb 0x55fdc021d550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:01:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6cf400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 0x55fdc0304b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6cf400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 0x55fdc0304b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:53604/0 (socket says 172.21.15.97:53604) 2024-03-11T20:01:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6cf400640 1 -- 172.21.15.97:0/2108984601 learned_addr learned my addr 172.21.15.97:0/2108984601 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6cfe00640 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fdff0 0x55fdc03045f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6d4a00640 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc0307570 0x55fdc0305070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6cf400640 1 -- 172.21.15.97:0/2108984601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc0307570 msgr2=0x55fdc0305070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6cf400640 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc0307570 0x55fdc0305070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6cf400640 1 -- 172.21.15.97:0/2108984601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fdff0 msgr2=0x55fdc03045f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6cf400640 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fdff0 0x55fdc03045f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6cf400640 1 -- 172.21.15.97:0/2108984601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c8009da0 con 0x55fdc02fed60 2024-03-11T20:01:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.259+0000 7fe6d4a00640 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc0307570 0x55fdc0305070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T20:01:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.260+0000 7fe6cf400640 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 0x55fdc0304b30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc005ec0 tx=0x7fe6bc0bbf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=519b54ac933c19ab server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.260+0000 7fe6b9600640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-11T20:01:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.260+0000 7fe6b9600640 1 -- 172.21.15.97:0/2108984601 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0c06f0 con 0x55fdc02fed60 2024-03-11T20:01:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.260+0000 7fe6b9600640 1 -- 172.21.15.97:0/2108984601 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7fe6bc0c4d40 con 0x55fdc02fed60 2024-03-11T20:01:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.260+0000 7fe6d95e0740 1 -- 172.21.15.97:0/2108984601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fdc0236cd0 con 0x55fdc02fed60 2024-03-11T20:01:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.260+0000 7fe6b9600640 1 -- 172.21.15.97:0/2108984601 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6bc0ed230 con 0x55fdc02fed60 2024-03-11T20:01:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.260+0000 7fe6d95e0740 10 client.25729 fetch_fsmap learned FSMap version 29 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.260+0000 7fe6d95e0740 1 -- 172.21.15.97:0/2108984601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55fdc0262c50 con 0x55fdc02fed60 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.260+0000 7fe6b9600640 1 -- 172.21.15.97:0/2108984601 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6bc09f020 con 0x55fdc02fed60 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.261+0000 7fe6b9600640 1 -- 172.21.15.97:0/2108984601 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe6bc03c070 con 0x55fdc02fed60 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.261+0000 7fe6d95e0740 10 client.25729 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.261+0000 7fe6d95e0740 10 client.25729 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.261+0000 7fe6d95e0740 20 client.25729 populate_metadata read hostname 'smithi097' 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.261+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55fdc0330b10 0x55fdc0332f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.261+0000 7fe6d95e0740 4 client.25729 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.261+0000 7fe6d95e0740 1 -- 172.21.15.97:0/2108984601 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fdc025ace0 con 0x55fdc0330b10 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.261+0000 7fe6cfe00640 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55fdc0330b10 0x55fdc0332f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.262+0000 7fe6b9600640 10 client.25729 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.262+0000 7fe6cfe00640 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55fdc0330b10 0x55fdc0332f30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=189c264104b6acaf server_cookie=7943e57b4c104cd5 in_seq=0 out_seq=0 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.262+0000 7fe6b9600640 1 -- 172.21.15.97:0/2108984601 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fdc025ace0 con 0x55fdc0330b10 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.262+0000 7fe6b9600640 10 client.25729 handle_command_reply: tid=0 2024-03-11T20:01:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.262+0000 7fe6d95e0740 1 client.25729 shutdown 2024-03-11T20:01:18.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.262+0000 7fe6d95e0740 1 -- 172.21.15.97:0/2108984601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 msgr2=0x55fdc0304b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:18.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.262+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 0x55fdc0304b30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc005ec0 tx=0x7fe6bc0bbf00 comp rx=0 tx=0).stop 2024-03-11T20:01:18.259 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.263+0000 7fe6d95e0740 1 -- 172.21.15.97:0/2108984601 shutdown_connections 2024-03-11T20:01:18.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.263+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc0307570 0x55fdc0305070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.263+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fed60 0x55fdc0304b30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.263+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55fdc0330b10 0x55fdc0332f30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.263+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/2108984601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fdff0 0x55fdc03045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.263+0000 7fe6d95e0740 1 -- 172.21.15.97:0/2108984601 >> 172.21.15.97:0/2108984601 conn(0x55fdc02c54b0 msgr2=0x55fdc02faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:18.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.263+0000 7fe6b9600640 0 client.25729 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:01:18.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.263+0000 7fe6d95e0740 1 -- 172.21.15.97:0/2108984601 shutdown_connections 2024-03-11T20:01:18.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.263+0000 7fe6d95e0740 1 -- 172.21.15.97:0/2108984601 wait complete. 2024-03-11T20:01:18.260 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.263+0000 7fe6d95e0740 20 client.25729 trim_cache size 0 max 16384 2024-03-11T20:01:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.281+0000 7fe6d95e0740 1 Processor -- start 2024-03-11T20:01:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.282+0000 7fe6d95e0740 1 -- start start 2024-03-11T20:01:18.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.282+0000 7fe6d95e0740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fdff0 0x55fdc0306d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.282+0000 7fe6d95e0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc02fed60 0x55fdc0304e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.282+0000 7fe6d95e0740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc0307570 0x55fdc031f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.282+0000 7fe6d95e0740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdbfface60 con 0x55fdc02fdff0 2024-03-11T20:01:18.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.282+0000 7fe6d95e0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc025ad10 con 0x55fdc02fed60 2024-03-11T20:01:18.277 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.282+0000 7fe6d95e0740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc02f84c0 con 0x55fdc0307570 2024-03-11T20:01:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.282+0000 7fe6cf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc02fed60 0x55fdc0304e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.282+0000 7fe6cf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc02fed60 0x55fdc0304e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:51064/0 (socket says 172.21.15.97:51064) 2024-03-11T20:01:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.282+0000 7fe6cf400640 1 -- 172.21.15.97:0/3337278715 learned_addr learned my addr 172.21.15.97:0/3337278715 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.283+0000 7fe6cfe00640 1 --2- 172.21.15.97:0/3337278715 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fdff0 0x55fdc0306d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.283+0000 7fe6d4a00640 1 --2- 172.21.15.97:0/3337278715 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc0307570 0x55fdc031f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.283+0000 7fe6cf400640 1 -- 172.21.15.97:0/3337278715 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc0307570 msgr2=0x55fdc031f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.283+0000 7fe6cf400640 1 --2- 172.21.15.97:0/3337278715 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc0307570 0x55fdc031f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.283+0000 7fe6cf400640 1 -- 172.21.15.97:0/3337278715 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fdff0 msgr2=0x55fdc0306d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.283+0000 7fe6cf400640 1 --2- 172.21.15.97:0/3337278715 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fdff0 0x55fdc0306d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.283+0000 7fe6cf400640 1 -- 172.21.15.97:0/3337278715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdc01ff530 con 0x55fdc02fed60 2024-03-11T20:01:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.283+0000 7fe6cf400640 1 --2- 172.21.15.97:0/3337278715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc02fed60 0x55fdc0304e90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0c4810 tx=0x7fe6bc0bc720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4970acc9e300af9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.283+0000 7fe6d4a00640 1 --2- 172.21.15.97:0/3337278715 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc0307570 0x55fdc031f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T20:01:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.284+0000 7fe6bbe00640 1 -- 172.21.15.97:0/3337278715 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0bcca0 con 0x55fdc02fed60 2024-03-11T20:01:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.284+0000 7fe6bbe00640 1 -- 172.21.15.97:0/3337278715 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6bc09f030 con 0x55fdc02fed60 2024-03-11T20:01:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.284+0000 7fe6bbe00640 1 -- 172.21.15.97:0/3337278715 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0c0780 con 0x55fdc02fed60 2024-03-11T20:01:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.285+0000 7fe6d95e0740 1 -- 172.21.15.97:0/3337278715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc02fed60 msgr2=0x55fdc0304e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.285+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/3337278715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc02fed60 0x55fdc0304e90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0c4810 tx=0x7fe6bc0bc720 comp rx=0 tx=0).stop 2024-03-11T20:01:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.285+0000 7fe6d95e0740 1 -- 172.21.15.97:0/3337278715 shutdown_connections 2024-03-11T20:01:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.285+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/3337278715 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc0307570 0x55fdc031f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.285+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/3337278715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc02fed60 0x55fdc0304e90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.285+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/3337278715 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fdff0 0x55fdc0306d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.285+0000 7fe6d95e0740 1 -- 172.21.15.97:0/3337278715 >> 172.21.15.97:0/3337278715 conn(0x55fdc02c54b0 msgr2=0x55fdc0302f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:18.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:17 smithi161 ceph-mon[36332]: pgmap v63: 129 pgs: 129 active+clean; 259 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 15 MiB/s wr, 249 op/s 2024-03-11T20:01:18.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.286+0000 7fe6d95e0740 1 -- 172.21.15.97:0/3337278715 shutdown_connections 2024-03-11T20:01:18.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.286+0000 7fe6d95e0740 1 -- 172.21.15.97:0/3337278715 wait complete. 2024-03-11T20:01:18.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.286+0000 7fe6d95e0740 1 -- start start 2024-03-11T20:01:18.281 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.287+0000 7fe6d95e0740 1 Processor -- start 2024-03-11T20:01:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.287+0000 7fe6d95e0740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fdff0 0x55fdc03013a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.287+0000 7fe6d95e0740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fed60 0x55fdc03018e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.287+0000 7fe6d95e0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc0307570 0x55fdc0301e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.287+0000 7fe6d95e0740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc0386fd0 con 0x55fdc02fed60 2024-03-11T20:01:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.287+0000 7fe6d95e0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc0387110 con 0x55fdc0307570 2024-03-11T20:01:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.287+0000 7fe6d95e0740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdc03872a0 con 0x55fdc02fdff0 2024-03-11T20:01:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.287+0000 7fe6d95e0740 10 client.0 _ll_register_callbacks cb 0x7fe6bc0bc370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:01:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.287+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fdff0 0x55fdc03013a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.287+0000 7fe6d4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc0307570 0x55fdc0301e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.287+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fdff0 0x55fdc03013a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:53618/0 (socket says 172.21.15.97:53618) 2024-03-11T20:01:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.288+0000 7fe6cfe00640 1 -- 172.21.15.97:0/1459816825 learned_addr learned my addr 172.21.15.97:0/1459816825 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.288+0000 7fe6cf400640 1 --2- 172.21.15.97:0/1459816825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fed60 0x55fdc03018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.288+0000 7fe6d4a00640 1 -- 172.21.15.97:0/1459816825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fdff0 msgr2=0x55fdc03013a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.288+0000 7fe6d4a00640 1 --2- 172.21.15.97:0/1459816825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fdff0 0x55fdc03013a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.288+0000 7fe6d4a00640 1 -- 172.21.15.97:0/1459816825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fed60 msgr2=0x55fdc03018e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.288+0000 7fe6d4a00640 1 --2- 172.21.15.97:0/1459816825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fed60 0x55fdc03018e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.288+0000 7fe6d4a00640 1 -- 172.21.15.97:0/1459816825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fdc0262c50 con 0x55fdc0307570 2024-03-11T20:01:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.289+0000 7fe6d4a00640 1 --2- 172.21.15.97:0/1459816825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc0307570 0x55fdc0301e20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8011510 tx=0x7fe6c80049e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=262ae6e27c8d4f6c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.289+0000 7fe6b9600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-11T20:01:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.289+0000 7fe6b9600640 1 -- 172.21.15.97:0/1459816825 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8019d10 con 0x55fdc0307570 2024-03-11T20:01:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.289+0000 7fe6b9600640 1 -- 172.21.15.97:0/1459816825 <== mon.2 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7fe6c8019e70 con 0x55fdc0307570 2024-03-11T20:01:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.289+0000 7fe6d95e0740 1 -- 172.21.15.97:0/1459816825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fdc0289d30 con 0x55fdc0307570 2024-03-11T20:01:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.289+0000 7fe6b9600640 1 -- 172.21.15.97:0/1459816825 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6c80193e0 con 0x55fdc0307570 2024-03-11T20:01:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.290+0000 7fe6d95e0740 10 client.25745 fetch_fsmap learned FSMap version 29 2024-03-11T20:01:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.290+0000 7fe6d95e0740 1 -- 172.21.15.97:0/1459816825 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55fdc02252c0 con 0x55fdc0307570 2024-03-11T20:01:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.290+0000 7fe6b9600640 1 -- 172.21.15.97:0/1459816825 <== mon.2 v2:172.21.15.114:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6c8019750 con 0x55fdc0307570 2024-03-11T20:01:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.290+0000 7fe6b9600640 1 -- 172.21.15.97:0/1459816825 <== mon.2 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe6c8023020 con 0x55fdc0307570 2024-03-11T20:01:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.290+0000 7fe6d95e0740 10 client.25745 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:01:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.290+0000 7fe6d95e0740 10 client.25745 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:01:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.290+0000 7fe6d95e0740 20 client.25745 populate_metadata read hostname 'smithi097' 2024-03-11T20:01:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.290+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/1459816825 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55fdc0325c60 0x55fdc0328080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.290+0000 7fe6d95e0740 4 client.25745 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T20:01:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.290+0000 7fe6d95e0740 1 -- 172.21.15.97:0/1459816825 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fdc0289d30 con 0x55fdc0325c60 2024-03-11T20:01:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.291+0000 7fe6cfe00640 1 --2- 172.21.15.97:0/1459816825 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55fdc0325c60 0x55fdc0328080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.291+0000 7fe6cfe00640 1 --2- 172.21.15.97:0/1459816825 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55fdc0325c60 0x55fdc0328080 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdc834d7ceeda4e4 server_cookie=51711ebd6d6f356a in_seq=0 out_seq=0 2024-03-11T20:01:18.287 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.291+0000 7fe6b9600640 10 client.25745 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:01:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.300+0000 7fe6b9600640 1 -- 172.21.15.97:0/1459816825 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fdc0289d30 con 0x55fdc0325c60 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.300+0000 7fe6b9600640 10 client.25745 handle_command_reply: tid=0 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.300+0000 7fe6d95e0740 1 client.25745 shutdown 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.300+0000 7fe6d95e0740 1 -- 172.21.15.97:0/1459816825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc0307570 msgr2=0x55fdc0301e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.300+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/1459816825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc0307570 0x55fdc0301e20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8011510 tx=0x7fe6c80049e0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.300+0000 7fe6d95e0740 1 -- 172.21.15.97:0/1459816825 shutdown_connections 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.300+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/1459816825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fdc0307570 0x55fdc0301e20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.300+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/1459816825 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x55fdc0325c60 0x55fdc0328080 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.300+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/1459816825 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55fdc02fed60 0x55fdc03018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.300+0000 7fe6d95e0740 1 --2- 172.21.15.97:0/1459816825 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55fdc02fdff0 0x55fdc03013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.300+0000 7fe6b9600640 0 client.25745 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.300+0000 7fe6d95e0740 1 -- 172.21.15.97:0/1459816825 >> 172.21.15.97:0/1459816825 conn(0x55fdc02c54b0 msgr2=0x55fdc02f7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.301+0000 7fe6d95e0740 1 -- 172.21.15.97:0/1459816825 shutdown_connections 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.301+0000 7fe6d95e0740 1 -- 172.21.15.97:0/1459816825 wait complete. 2024-03-11T20:01:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.301+0000 7fe6d95e0740 20 client.25745 trim_cache size 0 max 16384 2024-03-11T20:01:18.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-03-11T20:01:18.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-03-11T20:01:18.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-03-11T20:01:18.297 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-03-11T20:01:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.301+0000 7fe6d5400640 1 -- 172.21.15.97:0/2291879971 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe6b40a1c10 msgr2=0x7fe6b40a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.301+0000 7fe6d5400640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe6b40a1c10 0x7fe6b40a40d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8021000 tx=0x7fe6c8022000 comp rx=0 tx=0).stop 2024-03-11T20:01:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.301+0000 7fe6d5400640 1 -- 172.21.15.97:0/2291879971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0102a70 msgr2=0x7fe6d019fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.301+0000 7fe6d5400640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0102a70 0x7fe6d019fb70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc000c00 tx=0x7fe6bc0045b0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.302+0000 7fe6d5400640 1 -- 172.21.15.97:0/2291879971 shutdown_connections 2024-03-11T20:01:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.302+0000 7fe6d5400640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe6b40a1c10 0x7fe6b40a40d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.302+0000 7fe6d5400640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6d010b570 0x7fe6d01996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.302+0000 7fe6d5400640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6d0102a70 0x7fe6d019fb70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.302+0000 7fe6d5400640 1 --2- 172.21.15.97:0/2291879971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe6d0102150 0x7fe6d019f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.302+0000 7fe6d5400640 1 -- 172.21.15.97:0/2291879971 >> 172.21.15.97:0/2291879971 conn(0x7fe6d00fb990 msgr2=0x7fe6d010c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.302+0000 7fe6d5400640 1 -- 172.21.15.97:0/2291879971 shutdown_connections 2024-03-11T20:01:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:18.302+0000 7fe6d5400640 1 -- 172.21.15.97:0/2291879971 wait complete. 2024-03-11T20:01:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:19 smithi097 ceph-mon[31898]: pgmap v64: 129 pgs: 129 active+clean; 260 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 MiB/s wr, 342 op/s 2024-03-11T20:01:20.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:19 smithi114 ceph-mon[36376]: pgmap v64: 129 pgs: 129 active+clean; 260 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 MiB/s wr, 342 op/s 2024-03-11T20:01:20.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:19 smithi161 ceph-mon[36332]: pgmap v64: 129 pgs: 129 active+clean; 260 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 MiB/s wr, 342 op/s 2024-03-11T20:01:22.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:21 smithi097 ceph-mon[31898]: pgmap v65: 129 pgs: 129 active+clean; 260 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.5 MiB/s wr, 187 op/s 2024-03-11T20:01:22.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:21 smithi114 ceph-mon[36376]: pgmap v65: 129 pgs: 129 active+clean; 260 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.5 MiB/s wr, 187 op/s 2024-03-11T20:01:22.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:21 smithi161 ceph-mon[36332]: pgmap v65: 129 pgs: 129 active+clean; 260 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.5 MiB/s wr, 187 op/s 2024-03-11T20:01:24.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:23 smithi097 ceph-mon[31898]: pgmap v66: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-03-11T20:01:24.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:23 smithi114 ceph-mon[36376]: pgmap v66: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-03-11T20:01:24.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:23 smithi161 ceph-mon[36332]: pgmap v66: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-03-11T20:01:26.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:25 smithi097 ceph-mon[31898]: pgmap v67: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-11T20:01:26.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:25 smithi114 ceph-mon[36376]: pgmap v67: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-11T20:01:26.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:25 smithi161 ceph-mon[36332]: pgmap v67: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-11T20:01:27.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:26 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:01:27.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:26 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:01:27.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:26 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:01:28.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:27 smithi097 ceph-mon[31898]: pgmap v68: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-11T20:01:28.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:27 smithi114 ceph-mon[36376]: pgmap v68: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-11T20:01:28.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:27 smithi161 ceph-mon[36332]: pgmap v68: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-03-11T20:01:29.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:28 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:01:29.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:28 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:01:29.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:28 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:01:29.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:28 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:01:29.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:28 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:01:29.235 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:28 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:01:29.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:28 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:01:29.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:28 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:01:29.281 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:28 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:01:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:29 smithi097 ceph-mon[31898]: pgmap v69: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-03-11T20:01:30.182 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:29 smithi161 ceph-mon[36332]: pgmap v69: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-03-11T20:01:30.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:29 smithi114 ceph-mon[36376]: pgmap v69: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-03-11T20:01:31.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:30 smithi097 ceph-mon[31898]: pgmap v70: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 924 op/s 2024-03-11T20:01:31.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:30 smithi114 ceph-mon[36376]: pgmap v70: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 924 op/s 2024-03-11T20:01:31.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:30 smithi161 ceph-mon[36332]: pgmap v70: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 924 op/s 2024-03-11T20:01:31.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.436+0000 7f50c7c00640 1 -- 172.21.15.97:0/3812664794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50c0103f70 msgr2=0x7f50c0107fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:31.432 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.436+0000 7f50c7c00640 1 --2- 172.21.15.97:0/3812664794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50c0103f70 0x7f50c0107fb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009930 tx=0x7f50ac02bc20 comp rx=0 tx=0).stop 2024-03-11T20:01:31.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.436+0000 7f50c7c00640 1 -- 172.21.15.97:0/3812664794 shutdown_connections 2024-03-11T20:01:31.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.436+0000 7f50c7c00640 1 --2- 172.21.15.97:0/3812664794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c0108690 0x7f50c0110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.436+0000 7f50c7c00640 1 --2- 172.21.15.97:0/3812664794 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50c0103f70 0x7f50c0107fb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.436+0000 7f50c7c00640 1 --2- 172.21.15.97:0/3812664794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50c01035c0 0x7f50c01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.436+0000 7f50c7c00640 1 -- 172.21.15.97:0/3812664794 >> 172.21.15.97:0/3812664794 conn(0x7f50c00fee40 msgr2=0x7f50c0101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:31.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.436+0000 7f50c7c00640 1 -- 172.21.15.97:0/3812664794 shutdown_connections 2024-03-11T20:01:31.433 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.436+0000 7f50c7c00640 1 -- 172.21.15.97:0/3812664794 wait complete. 2024-03-11T20:01:31.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.437+0000 7f50c7c00640 1 Processor -- start 2024-03-11T20:01:31.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.437+0000 7f50c7c00640 1 -- start start 2024-03-11T20:01:31.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.438+0000 7f50c7c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c01035c0 0x7f50c019f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.438+0000 7f50c7c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50c0108690 0x7f50c019fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.438+0000 7f50c7c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50c0199880 0x7f50c0199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.435 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.438+0000 7f50c7c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c019a330 con 0x7f50c0108690 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.438+0000 7f50c7c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c019a4a0 con 0x7f50c01035c0 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.438+0000 7f50c7c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c019a610 con 0x7f50c0199880 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.438+0000 7f50bfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50c0108690 0x7f50c019fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.438+0000 7f50c5c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50c0199880 0x7f50c0199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.438+0000 7f50c5c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50c0199880 0x7f50c0199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:59782/0 (socket says 172.21.15.97:59782) 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.438+0000 7f50c5c00640 1 -- 172.21.15.97:0/3966571862 learned_addr learned my addr 172.21.15.97:0/3966571862 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.438+0000 7f50c5200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c01035c0 0x7f50c019f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.439+0000 7f50c5c00640 1 -- 172.21.15.97:0/3966571862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c01035c0 msgr2=0x7f50c019f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.439+0000 7f50c5c00640 1 --2- 172.21.15.97:0/3966571862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c01035c0 0x7f50c019f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.439+0000 7f50c5c00640 1 -- 172.21.15.97:0/3966571862 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50c0108690 msgr2=0x7f50c019fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.439+0000 7f50c5c00640 1 --2- 172.21.15.97:0/3966571862 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50c0108690 0x7f50c019fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.439+0000 7f50c5c00640 1 -- 172.21.15.97:0/3966571862 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac009560 con 0x7f50c0199880 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.439+0000 7f50bfe00640 1 --2- 172.21.15.97:0/3966571862 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50c0108690 0x7f50c019fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.439+0000 7f50c5c00640 1 --2- 172.21.15.97:0/3966571862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50c0199880 0x7f50c0199d30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f50b800eb10 tx=0x7f50b800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.439+0000 7f50bd600640 1 -- 172.21.15.97:0/3966571862 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b8004b20 con 0x7f50c0199880 2024-03-11T20:01:31.436 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.439+0000 7f50bd600640 1 -- 172.21.15.97:0/3966571862 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50b8004c80 con 0x7f50c0199880 2024-03-11T20:01:31.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.439+0000 7f50c7c00640 1 -- 172.21.15.97:0/3966571862 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c01a6570 con 0x7f50c0199880 2024-03-11T20:01:31.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.439+0000 7f50bd600640 1 -- 172.21.15.97:0/3966571862 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b8010650 con 0x7f50c0199880 2024-03-11T20:01:31.437 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.439+0000 7f50c7c00640 1 -- 172.21.15.97:0/3966571862 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c01a6a00 con 0x7f50c0199880 2024-03-11T20:01:31.438 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.442+0000 7f50ca87e740 1 Processor -- start 2024-03-11T20:01:31.438 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.442+0000 7f50bd600640 1 -- 172.21.15.97:0/3966571862 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f50b8005020 con 0x7f50c0199880 2024-03-11T20:01:31.438 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.443+0000 7f50ca87e740 1 -- start start 2024-03-11T20:01:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.443+0000 7f50ca87e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40c7b20 0x5618f40c6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.443+0000 7f50ca87e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c6db0 0x5618f40c7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.443+0000 7f50ca87e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40d0330 0x5618f40d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.443+0000 7f50ca87e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618f4082620 con 0x5618f40c6db0 2024-03-11T20:01:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.443+0000 7f50ca87e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618f40348c0 con 0x5618f40c7b20 2024-03-11T20:01:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.443+0000 7f50ca87e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618f4031df0 con 0x5618f40d0330 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.443+0000 7f50c5200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c6db0 0x5618f40c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.443+0000 7f50bfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40c7b20 0x5618f40c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.443+0000 7f50bfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40c7b20 0x5618f40c6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:34536/0 (socket says 172.21.15.97:34536) 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.443+0000 7f50bfe00640 1 -- 172.21.15.97:0/1729001639 learned_addr learned my addr 172.21.15.97:0/1729001639 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.443+0000 7f50bfe00640 1 --2- 172.21.15.97:0/1729001639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40d0330 0x5618f40d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.444+0000 7f50c5200640 1 -- 172.21.15.97:0/1729001639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40d0330 msgr2=0x5618f40d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.444+0000 7f50c5200640 1 --2- 172.21.15.97:0/1729001639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40d0330 0x5618f40d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.444+0000 7f50c5200640 1 -- 172.21.15.97:0/1729001639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40c7b20 msgr2=0x5618f40c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.444+0000 7f50c5200640 1 --2- 172.21.15.97:0/1729001639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40c7b20 0x5618f40c6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.444+0000 7f50c5200640 1 -- 172.21.15.97:0/1729001639 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618f3faeb00 con 0x5618f40c6db0 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.444+0000 7f50bd600640 1 --2- 172.21.15.97:0/3966571862 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f50a80a1c40 0x7f50a80a4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.444+0000 7f50c5200640 1 --2- 172.21.15.97:0/1729001639 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c6db0 0x5618f40c7230 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009c00 tx=0x7f50b4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0ed8d9d34614c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.444+0000 7f50bd600640 1 -- 172.21.15.97:0/3966571862 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f50b8014070 con 0x7f50c0199880 2024-03-11T20:01:31.441 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.445+0000 7f50b2000640 1 -- 172.21.15.97:0/1729001639 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b4004050 con 0x5618f40c6db0 2024-03-11T20:01:31.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.445+0000 7f50bfe00640 1 --2- 172.21.15.97:0/3966571862 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f50a80a1c40 0x7f50a80a4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.445+0000 7f50b2000640 1 -- 172.21.15.97:0/1729001639 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50b4005500 con 0x5618f40c6db0 2024-03-11T20:01:31.442 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.445+0000 7f50b2000640 1 -- 172.21.15.97:0/1729001639 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b4004cc0 con 0x5618f40c6db0 2024-03-11T20:01:31.445 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.445+0000 7f50ca87e740 1 -- 172.21.15.97:0/1729001639 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c6db0 msgr2=0x5618f40c7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:31.445 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.445+0000 7f50ca87e740 1 --2- 172.21.15.97:0/1729001639 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c6db0 0x5618f40c7230 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009c00 tx=0x7f50b4011810 comp rx=0 tx=0).stop 2024-03-11T20:01:31.445 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.445+0000 7f50bfe00640 1 --2- 172.21.15.97:0/3966571862 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f50a80a1c40 0x7f50a80a4100 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x5618f40c7850 tx=0x7f50ac02bc20 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:31.445 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.445+0000 7f50ca87e740 1 -- 172.21.15.97:0/1729001639 shutdown_connections 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.445+0000 7f50ca87e740 1 --2- 172.21.15.97:0/1729001639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40d0330 0x5618f40d2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.445+0000 7f50ca87e740 1 --2- 172.21.15.97:0/1729001639 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c6db0 0x5618f40c7230 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.445+0000 7f50ca87e740 1 --2- 172.21.15.97:0/1729001639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40c7b20 0x5618f40c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.445+0000 7f50ca87e740 1 -- 172.21.15.97:0/1729001639 >> 172.21.15.97:0/1729001639 conn(0x5618f408e4b0 msgr2=0x5618f40c1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.446+0000 7f50ca87e740 1 -- 172.21.15.97:0/1729001639 shutdown_connections 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.446+0000 7f50ca87e740 1 -- 172.21.15.97:0/1729001639 wait complete. 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.446+0000 7f50ca87e740 1 -- start start 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.446+0000 7f50ca87e740 1 Processor -- start 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50ca87e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40c6db0 0x5618f40cd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50ca87e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c7b20 0x5618f40cd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50ca87e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40d0330 0x5618f40cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50ca87e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618f402d5d0 con 0x5618f40d0330 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50ca87e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618f3fd2b50 con 0x5618f40c6db0 2024-03-11T20:01:31.446 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50ca87e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618f3fd27f0 con 0x5618f40c7b20 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50ca87e740 10 client.0 _ll_register_callbacks cb 0x5618f3fe6550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50c5c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40d0330 0x5618f40cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50bfe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c7b20 0x5618f40cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50c5c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40d0330 0x5618f40cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45964/0 (socket says 172.21.15.97:45964) 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50bfe00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c7b20 0x5618f40cd940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:59802/0 (socket says 172.21.15.97:59802) 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50c5c00640 1 -- 172.21.15.97:0/527249243 learned_addr learned my addr 172.21.15.97:0/527249243 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50c5200640 1 --2- 172.21.15.97:0/527249243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40c6db0 0x5618f40cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50c5c00640 1 -- 172.21.15.97:0/527249243 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c7b20 msgr2=0x5618f40cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50c5c00640 1 --2- 172.21.15.97:0/527249243 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c7b20 0x5618f40cd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50c5c00640 1 -- 172.21.15.97:0/527249243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40c6db0 msgr2=0x5618f40cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50c5c00640 1 --2- 172.21.15.97:0/527249243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40c6db0 0x5618f40cd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.447+0000 7f50c5c00640 1 -- 172.21.15.97:0/527249243 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b4009830 con 0x5618f40d0330 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.448+0000 7f50c5c00640 1 --2- 172.21.15.97:0/527249243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40d0330 0x5618f40cde80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f50b80a2bf0 tx=0x7f50b809b080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3707d8a46c2188cc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.448+0000 7f50a6a00640 10 client.0 ms_handle_connect on v2:172.21.15.97:3300/0 2024-03-11T20:01:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.448+0000 7f50a6a00640 1 -- 172.21.15.97:0/527249243 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b80b2070 con 0x5618f40d0330 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.448+0000 7f50a6a00640 1 -- 172.21.15.97:0/527249243 <== mon.0 v2:172.21.15.97:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f50b80a3120 con 0x5618f40d0330 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.448+0000 7f50ca87e740 1 -- 172.21.15.97:0/527249243 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5618f3fffcd0 con 0x5618f40d0330 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.448+0000 7f50a6a00640 1 -- 172.21.15.97:0/527249243 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50b8096ff0 con 0x5618f40d0330 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.448+0000 7f50ca87e740 10 client.15636 fetch_fsmap learned FSMap version 29 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.448+0000 7f50ca87e740 1 -- 172.21.15.97:0/527249243 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5618f402bc50 con 0x5618f40d0330 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.448+0000 7f50a6a00640 1 -- 172.21.15.97:0/527249243 <== mon.0 v2:172.21.15.97:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50b807d020 con 0x5618f40d0330 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.448+0000 7f50a6a00640 1 -- 172.21.15.97:0/527249243 <== mon.0 v2:172.21.15.97:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f50b801f060 con 0x5618f40d0330 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.449+0000 7f50ca87e740 10 client.15636 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.449+0000 7f50ca87e740 10 client.15636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.449+0000 7f50ca87e740 20 client.15636 populate_metadata read hostname 'smithi097' 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.449+0000 7f50ca87e740 1 --2- 172.21.15.97:0/527249243 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5618f40e0370 0x5618f40e2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.449+0000 7f50ca87e740 4 client.15636 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.449+0000 7f50ca87e740 1 -- 172.21.15.97:0/527249243 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5618f4023ce0 con 0x5618f40e0370 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.449+0000 7f50c5200640 1 --2- 172.21.15.97:0/527249243 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5618f40e0370 0x5618f40e2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.450+0000 7f50c5200640 1 --2- 172.21.15.97:0/527249243 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5618f40e0370 0x5618f40e2790 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=275da9b24ee9cf21 server_cookie=11f9ef17179835bd in_seq=0 out_seq=0 2024-03-11T20:01:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.450+0000 7f50a6a00640 10 client.15636 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:01:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.450+0000 7f50a6a00640 1 -- 172.21.15.97:0/527249243 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5618f4023ce0 con 0x5618f40e0370 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.450+0000 7f50a6a00640 10 client.15636 handle_command_reply: tid=0 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.450+0000 7f50ca87e740 1 client.15636 shutdown 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.451+0000 7f50ca87e740 1 -- 172.21.15.97:0/527249243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40d0330 msgr2=0x5618f40cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.451+0000 7f50ca87e740 1 --2- 172.21.15.97:0/527249243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40d0330 0x5618f40cde80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f50b80a2bf0 tx=0x7f50b809b080 comp rx=0 tx=0).stop 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.451+0000 7f50ca87e740 1 -- 172.21.15.97:0/527249243 shutdown_connections 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.451+0000 7f50ca87e740 1 --2- 172.21.15.97:0/527249243 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5618f40e0370 0x5618f40e2790 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.451+0000 7f50ca87e740 1 --2- 172.21.15.97:0/527249243 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40d0330 0x5618f40cde80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.451+0000 7f50ca87e740 1 --2- 172.21.15.97:0/527249243 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c7b20 0x5618f40cd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.451+0000 7f50a6a00640 0 client.15636 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.451+0000 7f50ca87e740 1 --2- 172.21.15.97:0/527249243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40c6db0 0x5618f40cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.451+0000 7f50ca87e740 1 -- 172.21.15.97:0/527249243 >> 172.21.15.97:0/527249243 conn(0x5618f408e4b0 msgr2=0x5618f40c37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.451+0000 7f50ca87e740 1 -- 172.21.15.97:0/527249243 shutdown_connections 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.451+0000 7f50ca87e740 1 -- 172.21.15.97:0/527249243 wait complete. 2024-03-11T20:01:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.451+0000 7f50ca87e740 20 client.15636 trim_cache size 0 max 16384 2024-03-11T20:01:31.461 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.465+0000 7f50ca87e740 1 Processor -- start 2024-03-11T20:01:31.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.465+0000 7f50ca87e740 1 -- start start 2024-03-11T20:01:31.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50ca87e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c6db0 0x5618f40cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50ca87e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c7b20 0x5618f40cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50ca87e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40d0330 0x5618f40d7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50ca87e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618f3d75e60 con 0x5618f40c7b20 2024-03-11T20:01:31.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50ca87e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618f40e79a0 con 0x5618f40d0330 2024-03-11T20:01:31.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50ca87e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618f40bd370 con 0x5618f40c6db0 2024-03-11T20:01:31.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50c5c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40d0330 0x5618f40d7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50c5200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c6db0 0x5618f40cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50bfe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c7b20 0x5618f40cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50c5200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c6db0 0x5618f40cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:59810/0 (socket says 172.21.15.97:59810) 2024-03-11T20:01:31.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50c5c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40d0330 0x5618f40d7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:34568/0 (socket says 172.21.15.97:34568) 2024-03-11T20:01:31.462 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50c5200640 1 -- 172.21.15.97:0/2960299872 learned_addr learned my addr 172.21.15.97:0/2960299872 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:31.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50c5200640 1 -- 172.21.15.97:0/2960299872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40d0330 msgr2=0x5618f40d7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:31.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50c5200640 1 --2- 172.21.15.97:0/2960299872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40d0330 0x5618f40d7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50c5200640 1 -- 172.21.15.97:0/2960299872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c7b20 msgr2=0x5618f40cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:31.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50c5200640 1 --2- 172.21.15.97:0/2960299872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c7b20 0x5618f40cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50c5200640 1 -- 172.21.15.97:0/2960299872 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618f3fc8530 con 0x5618f40c6db0 2024-03-11T20:01:31.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.466+0000 7f50c5c00640 1 --2- 172.21.15.97:0/2960299872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40d0330 0x5618f40d7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:01:31.464 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.467+0000 7f50c5200640 1 --2- 172.21.15.97:0/2960299872 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c6db0 0x5618f40cfb30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009b20 tx=0x7f50b4004840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a7f9aa17504ba9c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.467+0000 7f50b1600640 1 -- 172.21.15.97:0/2960299872 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b402a040 con 0x5618f40c6db0 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.467+0000 7f50b1600640 1 -- 172.21.15.97:0/2960299872 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50b402d820 con 0x5618f40c6db0 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.467+0000 7f50b1600640 1 -- 172.21.15.97:0/2960299872 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b40197b0 con 0x5618f40c6db0 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.468+0000 7f50ca87e740 1 -- 172.21.15.97:0/2960299872 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c6db0 msgr2=0x5618f40cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.468+0000 7f50ca87e740 1 --2- 172.21.15.97:0/2960299872 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c6db0 0x5618f40cfb30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009b20 tx=0x7f50b4004840 comp rx=0 tx=0).stop 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.468+0000 7f50ca87e740 1 -- 172.21.15.97:0/2960299872 shutdown_connections 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.468+0000 7f50ca87e740 1 --2- 172.21.15.97:0/2960299872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40d0330 0x5618f40d7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.468+0000 7f50ca87e740 1 --2- 172.21.15.97:0/2960299872 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c7b20 0x5618f40cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.468+0000 7f50ca87e740 1 --2- 172.21.15.97:0/2960299872 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c6db0 0x5618f40cfb30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.468+0000 7f50ca87e740 1 -- 172.21.15.97:0/2960299872 >> 172.21.15.97:0/2960299872 conn(0x5618f408e4b0 msgr2=0x5618f40cbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.468+0000 7f50ca87e740 1 -- 172.21.15.97:0/2960299872 shutdown_connections 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.468+0000 7f50ca87e740 1 -- 172.21.15.97:0/2960299872 wait complete. 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.468+0000 7f50ca87e740 1 -- start start 2024-03-11T20:01:31.465 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.469+0000 7f50ca87e740 1 Processor -- start 2024-03-11T20:01:31.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.469+0000 7f50ca87e740 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c6db0 0x5618f40ca180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.469+0000 7f50ca87e740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c7b20 0x5618f40ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.469+0000 7f50ca87e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40d0330 0x5618f40cac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.469+0000 7f50ca87e740 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618f41368c0 con 0x5618f40c6db0 2024-03-11T20:01:31.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.469+0000 7f50ca87e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618f4136a30 con 0x5618f40d0330 2024-03-11T20:01:31.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.469+0000 7f50ca87e740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618f4136bf0 con 0x5618f40c7b20 2024-03-11T20:01:31.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.469+0000 7f50ca87e740 10 client.0 _ll_register_callbacks cb 0x7f50b8004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T20:01:31.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.469+0000 7f50c5200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c6db0 0x5618f40ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.468 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.469+0000 7f50c5200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c6db0 0x5618f40ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45986/0 (socket says 172.21.15.97:45986) 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.469+0000 7f50c5200640 1 -- 172.21.15.97:0/43856775 learned_addr learned my addr 172.21.15.97:0/43856775 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.469+0000 7f50c5c00640 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40d0330 0x5618f40cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.470+0000 7f50bfe00640 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c7b20 0x5618f40ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.470+0000 7f50c5c00640 1 -- 172.21.15.97:0/43856775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c7b20 msgr2=0x5618f40ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.470+0000 7f50c5c00640 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c7b20 0x5618f40ca6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.470+0000 7f50c5c00640 1 -- 172.21.15.97:0/43856775 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c6db0 msgr2=0x5618f40ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.470+0000 7f50c5c00640 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c6db0 0x5618f40ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.470+0000 7f50c5c00640 1 -- 172.21.15.97:0/43856775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618f402bc50 con 0x5618f40d0330 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.470+0000 7f50c5200640 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c6db0 0x5618f40ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.470+0000 7f50c5c00640 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40d0330 0x5618f40cac00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f50b80a2e90 tx=0x7f50b800cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb4e688f442f7be5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.470+0000 7f50a6a00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.470+0000 7f50a6a00640 1 -- 172.21.15.97:0/43856775 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b80b2070 con 0x5618f40d0330 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.470+0000 7f50ca87e740 1 -- 172.21.15.97:0/43856775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5618f4052d30 con 0x5618f40d0330 2024-03-11T20:01:31.469 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.470+0000 7f50a6a00640 1 -- 172.21.15.97:0/43856775 <== mon.2 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9445+0+0 (secure 0 0 0) 0x7f50b807e080 con 0x5618f40d0330 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.471+0000 7f50a6a00640 1 -- 172.21.15.97:0/43856775 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50b809a4c0 con 0x5618f40d0330 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.471+0000 7f50ca87e740 10 client.25763 fetch_fsmap learned FSMap version 29 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.471+0000 7f50ca87e740 1 -- 172.21.15.97:0/43856775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5618f3faeb00 con 0x5618f40d0330 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.471+0000 7f50a6a00640 1 -- 172.21.15.97:0/43856775 <== mon.2 v2:172.21.15.114:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5618f4052d30 con 0x5618f40d0330 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.471+0000 7f50a6a00640 1 -- 172.21.15.97:0/43856775 <== mon.2 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f50b801f040 con 0x5618f40d0330 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.471+0000 7f50ca87e740 10 client.25763 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.471+0000 7f50ca87e740 10 client.25763 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.471+0000 7f50ca87e740 20 client.25763 populate_metadata read hostname 'smithi097' 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.471+0000 7f50ca87e740 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5618f4138170 0x5618f413a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.471+0000 7f50ca87e740 4 client.25763 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.471+0000 7f50ca87e740 1 -- 172.21.15.97:0/43856775 --> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5618f3d2d600 con 0x5618f4138170 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.471+0000 7f50c5200640 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5618f4138170 0x5618f413a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.472+0000 7f50c5200640 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5618f4138170 0x5618f413a590 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b305cd3437b1f380 server_cookie=b0e133d1dea03354 in_seq=0 out_seq=0 2024-03-11T20:01:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.472+0000 7f50a6a00640 10 client.25763 ms_handle_connect on v2:172.21.15.114:6838/1624532248 2024-03-11T20:01:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.484+0000 7f50a6a00640 1 -- 172.21.15.97:0/43856775 <== mds.0 v2:172.21.15.114:6838/1624532248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5618f3d2d600 con 0x5618f4138170 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.484+0000 7f50a6a00640 10 client.25763 handle_command_reply: tid=0 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.484+0000 7f50ca87e740 1 client.25763 shutdown 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.484+0000 7f50ca87e740 1 -- 172.21.15.97:0/43856775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40d0330 msgr2=0x5618f40cac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.484+0000 7f50ca87e740 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40d0330 0x5618f40cac00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f50b80a2e90 tx=0x7f50b800cf70 comp rx=0 tx=0).stop 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.485+0000 7f50ca87e740 1 -- 172.21.15.97:0/43856775 shutdown_connections 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.485+0000 7f50ca87e740 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5618f40d0330 0x5618f40cac00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.485+0000 7f50ca87e740 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5618f40c7b20 0x5618f40ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.485+0000 7f50ca87e740 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.114:6838/1624532248,v1:172.21.15.114:6839/1624532248] conn(0x5618f4138170 0x5618f413a590 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.485+0000 7f50ca87e740 1 --2- 172.21.15.97:0/43856775 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5618f40c6db0 0x5618f40ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.485+0000 7f50a6a00640 0 client.25763 ms_handle_reset on v2:172.21.15.114:6838/1624532248 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.485+0000 7f50ca87e740 1 -- 172.21.15.97:0/43856775 >> 172.21.15.97:0/43856775 conn(0x5618f408e4b0 msgr2=0x5618f40c04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.485+0000 7f50ca87e740 1 -- 172.21.15.97:0/43856775 shutdown_connections 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.485+0000 7f50ca87e740 1 -- 172.21.15.97:0/43856775 wait complete. 2024-03-11T20:01:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.485+0000 7f50ca87e740 20 client.25763 trim_cache size 0 max 16384 2024-03-11T20:01:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:{ 2024-03-11T20:01:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "message": "", 2024-03-11T20:01:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout: "return_code": 0 2024-03-11T20:01:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stdout:} 2024-03-11T20:01:31.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.486+0000 7f50c7c00640 1 -- 172.21.15.97:0/3966571862 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f50a80a1c40 msgr2=0x7f50a80a4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:31.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.486+0000 7f50c7c00640 1 --2- 172.21.15.97:0/3966571862 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f50a80a1c40 0x7f50a80a4100 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x5618f40c7850 tx=0x7f50ac02bc20 comp rx=0 tx=0).stop 2024-03-11T20:01:31.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.486+0000 7f50c7c00640 1 -- 172.21.15.97:0/3966571862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50c0199880 msgr2=0x7f50c0199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:01:31.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.486+0000 7f50c7c00640 1 --2- 172.21.15.97:0/3966571862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50c0199880 0x7f50c0199d30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f50b800eb10 tx=0x7f50b800c2d0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.486+0000 7f50c7c00640 1 -- 172.21.15.97:0/3966571862 shutdown_connections 2024-03-11T20:01:31.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.486+0000 7f50c7c00640 1 --2- 172.21.15.97:0/3966571862 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f50a80a1c40 0x7f50a80a4100 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.486+0000 7f50c7c00640 1 --2- 172.21.15.97:0/3966571862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50c0199880 0x7f50c0199d30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.486+0000 7f50c7c00640 1 --2- 172.21.15.97:0/3966571862 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f50c0108690 0x7f50c019fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.486+0000 7f50c7c00640 1 --2- 172.21.15.97:0/3966571862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c01035c0 0x7f50c019f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:01:31.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.486+0000 7f50c7c00640 1 -- 172.21.15.97:0/3966571862 >> 172.21.15.97:0/3966571862 conn(0x7f50c00fee40 msgr2=0x7f50c0101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:01:31.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.486+0000 7f50c7c00640 1 -- 172.21.15.97:0/3966571862 shutdown_connections 2024-03-11T20:01:31.483 INFO:teuthology.task.background_exec.ubuntu@smithi097.front.sepia.ceph.com.smithi097.stderr:2024-03-11T20:01:31.486+0000 7f50c7c00640 1 -- 172.21.15.97:0/3966571862 wait complete. 2024-03-11T20:01:34.030 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:33 smithi161 ceph-mon[36332]: pgmap v71: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 MiB/s wr, 932 op/s 2024-03-11T20:01:34.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:33 smithi097 ceph-mon[31898]: pgmap v71: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 MiB/s wr, 932 op/s 2024-03-11T20:01:34.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:33 smithi114 ceph-mon[36376]: pgmap v71: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 MiB/s wr, 932 op/s 2024-03-11T20:01:35.684 INFO:tasks.workunit.client.0.smithi097.stdout:setting up random write file 2024-03-11T20:01:35.684 INFO:tasks.workunit.client.0.smithi097.stdout:done setting up random write file 2024-03-11T20:01:35.684 INFO:tasks.workunit.client.0.smithi097.stdout:starting fsync run 2024-03-11T20:01:35.684 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0010s fsync time: 0.0324s 2024-03-11T20:01:35.684 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0006s fsync time: 0.0184s 2024-03-11T20:01:35.684 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0012s fsync time: 0.0172s 2024-03-11T20:01:35.684 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0208s 2024-03-11T20:01:35.684 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0241s 2024-03-11T20:01:35.684 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0008s fsync time: 0.0218s 2024-03-11T20:01:35.684 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0263s 2024-03-11T20:01:35.684 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0222s 2024-03-11T20:01:35.684 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0014s fsync time: 0.0229s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0248s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0245s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0252s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0014s fsync time: 0.0183s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0252s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0242s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0173s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0241s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0243s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0221s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0230s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0235s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0256s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0012s fsync time: 0.0238s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0244s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0253s 2024-03-11T20:01:35.685 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0014s fsync time: 0.0156s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0008s fsync time: 0.0254s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0018s fsync time: 0.0253s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0020s fsync time: 0.0198s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0018s fsync time: 0.0196s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0006s fsync time: 0.0170s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0014s fsync time: 0.0137s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0027s fsync time: 0.0254s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0011s fsync time: 0.0224s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0012s fsync time: 0.0245s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0014s fsync time: 0.0332s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0239s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0009s fsync time: 0.0164s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0010s fsync time: 0.0145s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0009s fsync time: 0.0199s 2024-03-11T20:01:35.686 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0159s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0017s fsync time: 0.0197s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0020s fsync time: 0.0214s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0015s fsync time: 0.0261s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0008s fsync time: 0.3906s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0233s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0242s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0008s fsync time: 0.0127s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0250s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0012s fsync time: 0.0243s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0256s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0005s fsync time: 0.0066s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0017s fsync time: 0.0247s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0008s fsync time: 0.0100s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0008s fsync time: 0.0149s 2024-03-11T20:01:35.687 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0172s 2024-03-11T20:01:35.688 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0014s fsync time: 0.0248s 2024-03-11T20:01:35.688 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0220s 2024-03-11T20:01:35.688 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0238s 2024-03-11T20:01:35.688 INFO:tasks.workunit.client.0.smithi097.stdout:write time: 0.0013s fsync time: 0.0257s 2024-03-11T20:01:35.688 INFO:tasks.workunit.client.0.smithi097.stdout:run done 60 fsyncs total, killing random writer 2024-03-11T20:01:35.688 INFO:tasks.workunit.client.0.smithi097.stderr:+ echo /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-11T20:01:35.688 INFO:tasks.workunit.client.0.smithi097.stdout:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-11T20:01:35.689 INFO:tasks.workunit.client.0.smithi097.stderr:+ whereis lsof 2024-03-11T20:01:35.728 INFO:tasks.workunit.client.0.smithi097.stdout:lsof: /usr/bin/lsof /usr/share/man/man1/lsof.1.gz 2024-03-11T20:01:35.728 INFO:tasks.workunit.client.0.smithi097.stderr:+ lsof 2024-03-11T20:01:35.753 INFO:tasks.workunit.client.0.smithi097.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/fa8621052974e23c9d8a56ff33761b9bd3dcad3a9e4b3bb3ed04e2c2649c9558/merged 2024-03-11T20:01:35.754 INFO:tasks.workunit.client.0.smithi097.stderr: Output information may be incomplete. 2024-03-11T20:01:35.754 INFO:tasks.workunit.client.0.smithi097.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/db10f39827646906b4622ef908fdf616d6b8e5658e008631616b52cb62f8a56d/merged 2024-03-11T20:01:35.754 INFO:tasks.workunit.client.0.smithi097.stderr: Output information may be incomplete. 2024-03-11T20:01:35.760 INFO:tasks.workunit.client.0.smithi097.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/f51d1f776cbb0e99d3850027c762f5e4cbaa3cb498e936b04c471a84fbd7fcbe/merged 2024-03-11T20:01:35.760 INFO:tasks.workunit.client.0.smithi097.stderr: Output information may be incomplete. 2024-03-11T20:01:35.760 INFO:tasks.workunit.client.0.smithi097.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/ec6b946d4c60bbe62468c69b4d50ed152000e7db099441e0970ee121e4fda4c3/merged 2024-03-11T20:01:35.761 INFO:tasks.workunit.client.0.smithi097.stderr: Output information may be incomplete. 2024-03-11T20:01:35.761 INFO:tasks.workunit.client.0.smithi097.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/267036aea224cf6f0771e4018b35f3c7d3c255b17526536c4ce4e7d07bbd5f18/merged 2024-03-11T20:01:35.761 INFO:tasks.workunit.client.0.smithi097.stderr: Output information may be incomplete. 2024-03-11T20:01:35.761 INFO:tasks.workunit.client.0.smithi097.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/6f2750daaa6b7ad719826904c69fe43dd873c083ae19a4ab00ef467697ba4ca1/merged 2024-03-11T20:01:35.761 INFO:tasks.workunit.client.0.smithi097.stderr: Output information may be incomplete. 2024-03-11T20:01:35.761 INFO:tasks.workunit.client.0.smithi097.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/2cc05717523036ea4bed32b1d08d3dd9102f2dc8588a30fef9aae040470e3a25/merged 2024-03-11T20:01:35.761 INFO:tasks.workunit.client.0.smithi097.stderr: Output information may be incomplete. 2024-03-11T20:01:35.761 INFO:tasks.workunit.client.0.smithi097.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/323711713fbc1b75913406a7a8faef5099ca51845c2fe828c65c818b405a116b/merged 2024-03-11T20:01:35.761 INFO:tasks.workunit.client.0.smithi097.stderr: Output information may be incomplete. 2024-03-11T20:01:35.761 INFO:tasks.workunit.client.0.smithi097.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/00a927362f891155cf8a8d7417fbd5aafe7090e0f54a27fac0706cccdbdce4e1/merged 2024-03-11T20:01:35.761 INFO:tasks.workunit.client.0.smithi097.stderr: Output information may be incomplete. 2024-03-11T20:01:35.761 INFO:tasks.workunit.client.0.smithi097.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/59f8364e95708ab6dc8efd5f3643dbbb449329bf274aaec909a0db8874c68def/merged 2024-03-11T20:01:35.761 INFO:tasks.workunit.client.0.smithi097.stderr: Output information may be incomplete. 2024-03-11T20:01:35.892 INFO:tasks.workunit.client.0.smithi097.stdout:COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME 2024-03-11T20:01:35.892 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1 root cwd unknown /proc/1/cwd (readlink: Permission denied) 2024-03-11T20:01:35.892 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1 root rtd unknown /proc/1/root (readlink: Permission denied) 2024-03-11T20:01:35.892 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1 root txt unknown /proc/1/exe (readlink: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1 root NOFD /proc/1/fd (opendir: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:kthreadd 2 root cwd unknown /proc/2/cwd (readlink: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:kthreadd 2 root rtd unknown /proc/2/root (readlink: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:kthreadd 2 root txt unknown /proc/2/exe (readlink: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:kthreadd 2 root NOFD /proc/2/fd (opendir: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:pool_work 3 root cwd unknown /proc/3/cwd (readlink: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:pool_work 3 root rtd unknown /proc/3/root (readlink: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:pool_work 3 root txt unknown /proc/3/exe (readlink: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:pool_work 3 root NOFD /proc/3/fd (opendir: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 4 root cwd unknown /proc/4/cwd (readlink: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 4 root rtd unknown /proc/4/root (readlink: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 4 root txt unknown /proc/4/exe (readlink: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 4 root NOFD /proc/4/fd (opendir: Permission denied) 2024-03-11T20:01:35.893 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 5 root cwd unknown /proc/5/cwd (readlink: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 5 root rtd unknown /proc/5/root (readlink: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 5 root txt unknown /proc/5/exe (readlink: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 5 root NOFD /proc/5/fd (opendir: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 6 root cwd unknown /proc/6/cwd (readlink: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 6 root rtd unknown /proc/6/root (readlink: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 6 root txt unknown /proc/6/exe (readlink: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 6 root NOFD /proc/6/fd (opendir: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 7 root cwd unknown /proc/7/cwd (readlink: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 7 root rtd unknown /proc/7/root (readlink: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 7 root txt unknown /proc/7/exe (readlink: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 7 root NOFD /proc/7/fd (opendir: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 10 root cwd unknown /proc/10/cwd (readlink: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 10 root rtd unknown /proc/10/root (readlink: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 10 root txt unknown /proc/10/exe (readlink: Permission denied) 2024-03-11T20:01:35.894 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 10 root NOFD /proc/10/fd (opendir: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 12 root cwd unknown /proc/12/cwd (readlink: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 12 root rtd unknown /proc/12/root (readlink: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 12 root txt unknown /proc/12/exe (readlink: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 12 root NOFD /proc/12/fd (opendir: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_tasks 13 root cwd unknown /proc/13/cwd (readlink: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_tasks 13 root rtd unknown /proc/13/root (readlink: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_tasks 13 root txt unknown /proc/13/exe (readlink: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_tasks 13 root NOFD /proc/13/fd (opendir: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_tasks 14 root cwd unknown /proc/14/cwd (readlink: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_tasks 14 root rtd unknown /proc/14/root (readlink: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_tasks 14 root txt unknown /proc/14/exe (readlink: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_tasks 14 root NOFD /proc/14/fd (opendir: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_tasks 15 root cwd unknown /proc/15/cwd (readlink: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_tasks 15 root rtd unknown /proc/15/root (readlink: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_tasks 15 root txt unknown /proc/15/exe (readlink: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_tasks 15 root NOFD /proc/15/fd (opendir: Permission denied) 2024-03-11T20:01:35.895 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 16 root cwd unknown /proc/16/cwd (readlink: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 16 root rtd unknown /proc/16/root (readlink: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 16 root txt unknown /proc/16/exe (readlink: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 16 root NOFD /proc/16/fd (opendir: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_preem 17 root cwd unknown /proc/17/cwd (readlink: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_preem 17 root rtd unknown /proc/17/root (readlink: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_preem 17 root txt unknown /proc/17/exe (readlink: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:rcu_preem 17 root NOFD /proc/17/fd (opendir: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:migration 18 root cwd unknown /proc/18/cwd (readlink: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:migration 18 root rtd unknown /proc/18/root (readlink: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:migration 18 root txt unknown /proc/18/exe (readlink: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:migration 18 root NOFD /proc/18/fd (opendir: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/0 19 root cwd unknown /proc/19/cwd (readlink: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/0 19 root rtd unknown /proc/19/root (readlink: Permission denied) 2024-03-11T20:01:35.896 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/0 19 root txt unknown /proc/19/exe (readlink: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/0 19 root NOFD /proc/19/fd (opendir: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/1 20 root cwd unknown /proc/20/cwd (readlink: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/1 20 root rtd unknown /proc/20/root (readlink: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/1 20 root txt unknown /proc/20/exe (readlink: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/1 20 root NOFD /proc/20/fd (opendir: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:migration 21 root cwd unknown /proc/21/cwd (readlink: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:migration 21 root rtd unknown /proc/21/root (readlink: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:migration 21 root txt unknown /proc/21/exe (readlink: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:migration 21 root NOFD /proc/21/fd (opendir: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 22 root cwd unknown /proc/22/cwd (readlink: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 22 root rtd unknown /proc/22/root (readlink: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 22 root txt unknown /proc/22/exe (readlink: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 22 root NOFD /proc/22/fd (opendir: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 24 root cwd unknown /proc/24/cwd (readlink: Permission denied) 2024-03-11T20:01:35.897 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 24 root rtd unknown /proc/24/root (readlink: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 24 root txt unknown /proc/24/exe (readlink: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 24 root NOFD /proc/24/fd (opendir: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/2 25 root cwd unknown /proc/25/cwd (readlink: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/2 25 root rtd unknown /proc/25/root (readlink: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/2 25 root txt unknown /proc/25/exe (readlink: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/2 25 root NOFD /proc/25/fd (opendir: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:migration 26 root cwd unknown /proc/26/cwd (readlink: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:migration 26 root rtd unknown /proc/26/root (readlink: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:migration 26 root txt unknown /proc/26/exe (readlink: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:migration 26 root NOFD /proc/26/fd (opendir: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 27 root cwd unknown /proc/27/cwd (readlink: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 27 root rtd unknown /proc/27/root (readlink: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 27 root txt unknown /proc/27/exe (readlink: Permission denied) 2024-03-11T20:01:35.898 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 27 root NOFD /proc/27/fd (opendir: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 28 root cwd unknown /proc/28/cwd (readlink: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 28 root rtd unknown /proc/28/root (readlink: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 28 root txt unknown /proc/28/exe (readlink: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 28 root NOFD /proc/28/fd (opendir: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 29 root cwd unknown /proc/29/cwd (readlink: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 29 root rtd unknown /proc/29/root (readlink: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 29 root txt unknown /proc/29/exe (readlink: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 29 root NOFD /proc/29/fd (opendir: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/3 30 root cwd unknown /proc/30/cwd (readlink: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/3 30 root rtd unknown /proc/30/root (readlink: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/3 30 root txt unknown /proc/30/exe (readlink: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/3 30 root NOFD /proc/30/fd (opendir: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:migration 31 root cwd unknown /proc/31/cwd (readlink: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:migration 31 root rtd unknown /proc/31/root (readlink: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:migration 31 root txt unknown /proc/31/exe (readlink: Permission denied) 2024-03-11T20:01:35.899 INFO:tasks.workunit.client.0.smithi097.stdout:migration 31 root NOFD /proc/31/fd (opendir: Permission denied) 2024-03-11T20:01:35.900 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 32 root cwd unknown /proc/32/cwd (readlink: Permission denied) 2024-03-11T20:01:35.900 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 32 root rtd unknown /proc/32/root (readlink: Permission denied) 2024-03-11T20:01:35.900 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 32 root txt unknown /proc/32/exe (readlink: Permission denied) 2024-03-11T20:01:35.900 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 32 root NOFD /proc/32/fd (opendir: Permission denied) 2024-03-11T20:01:35.900 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 34 root cwd unknown /proc/34/cwd (readlink: Permission denied) 2024-03-11T20:01:35.900 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 34 root rtd unknown /proc/34/root (readlink: Permission denied) 2024-03-11T20:01:35.900 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 34 root txt unknown /proc/34/exe (readlink: Permission denied) 2024-03-11T20:01:35.900 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 34 root NOFD /proc/34/fd (opendir: Permission denied) 2024-03-11T20:01:35.900 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/4 35 root cwd unknown /proc/35/cwd (readlink: Permission denied) 2024-03-11T20:01:35.900 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/4 35 root rtd unknown /proc/35/root (readlink: Permission denied) 2024-03-11T20:01:35.900 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/4 35 root txt unknown /proc/35/exe (readlink: Permission denied) 2024-03-11T20:01:35.900 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/4 35 root NOFD /proc/35/fd (opendir: Permission denied) 2024-03-11T20:01:35.900 INFO:tasks.workunit.client.0.smithi097.stdout:migration 36 root cwd unknown /proc/36/cwd (readlink: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:migration 36 root rtd unknown /proc/36/root (readlink: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:migration 36 root txt unknown /proc/36/exe (readlink: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:migration 36 root NOFD /proc/36/fd (opendir: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 37 root cwd unknown /proc/37/cwd (readlink: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 37 root rtd unknown /proc/37/root (readlink: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 37 root txt unknown /proc/37/exe (readlink: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 37 root NOFD /proc/37/fd (opendir: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 39 root cwd unknown /proc/39/cwd (readlink: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 39 root rtd unknown /proc/39/root (readlink: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 39 root txt unknown /proc/39/exe (readlink: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 39 root NOFD /proc/39/fd (opendir: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/5 40 root cwd unknown /proc/40/cwd (readlink: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/5 40 root rtd unknown /proc/40/root (readlink: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/5 40 root txt unknown /proc/40/exe (readlink: Permission denied) 2024-03-11T20:01:35.901 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/5 40 root NOFD /proc/40/fd (opendir: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:migration 41 root cwd unknown /proc/41/cwd (readlink: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:migration 41 root rtd unknown /proc/41/root (readlink: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:migration 41 root txt unknown /proc/41/exe (readlink: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:migration 41 root NOFD /proc/41/fd (opendir: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 42 root cwd unknown /proc/42/cwd (readlink: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 42 root rtd unknown /proc/42/root (readlink: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 42 root txt unknown /proc/42/exe (readlink: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 42 root NOFD /proc/42/fd (opendir: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 44 root cwd unknown /proc/44/cwd (readlink: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 44 root rtd unknown /proc/44/root (readlink: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 44 root txt unknown /proc/44/exe (readlink: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 44 root NOFD /proc/44/fd (opendir: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/6 45 root cwd unknown /proc/45/cwd (readlink: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/6 45 root rtd unknown /proc/45/root (readlink: Permission denied) 2024-03-11T20:01:35.902 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/6 45 root txt unknown /proc/45/exe (readlink: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/6 45 root NOFD /proc/45/fd (opendir: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:migration 46 root cwd unknown /proc/46/cwd (readlink: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:migration 46 root rtd unknown /proc/46/root (readlink: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:migration 46 root txt unknown /proc/46/exe (readlink: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:migration 46 root NOFD /proc/46/fd (opendir: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 47 root cwd unknown /proc/47/cwd (readlink: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 47 root rtd unknown /proc/47/root (readlink: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 47 root txt unknown /proc/47/exe (readlink: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 47 root NOFD /proc/47/fd (opendir: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 49 root cwd unknown /proc/49/cwd (readlink: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 49 root rtd unknown /proc/49/root (readlink: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 49 root txt unknown /proc/49/exe (readlink: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 49 root NOFD /proc/49/fd (opendir: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/7 50 root cwd unknown /proc/50/cwd (readlink: Permission denied) 2024-03-11T20:01:35.903 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/7 50 root rtd unknown /proc/50/root (readlink: Permission denied) 2024-03-11T20:01:35.904 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/7 50 root txt unknown /proc/50/exe (readlink: Permission denied) 2024-03-11T20:01:35.904 INFO:tasks.workunit.client.0.smithi097.stdout:cpuhp/7 50 root NOFD /proc/50/fd (opendir: Permission denied) 2024-03-11T20:01:35.904 INFO:tasks.workunit.client.0.smithi097.stdout:migration 51 root cwd unknown /proc/51/cwd (readlink: Permission denied) 2024-03-11T20:01:35.904 INFO:tasks.workunit.client.0.smithi097.stdout:migration 51 root rtd unknown /proc/51/root (readlink: Permission denied) 2024-03-11T20:01:35.904 INFO:tasks.workunit.client.0.smithi097.stdout:migration 51 root txt unknown /proc/51/exe (readlink: Permission denied) 2024-03-11T20:01:35.904 INFO:tasks.workunit.client.0.smithi097.stdout:migration 51 root NOFD /proc/51/fd (opendir: Permission denied) 2024-03-11T20:01:35.904 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 52 root cwd unknown /proc/52/cwd (readlink: Permission denied) 2024-03-11T20:01:35.904 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 52 root rtd unknown /proc/52/root (readlink: Permission denied) 2024-03-11T20:01:35.904 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 52 root txt unknown /proc/52/exe (readlink: Permission denied) 2024-03-11T20:01:35.904 INFO:tasks.workunit.client.0.smithi097.stdout:ksoftirqd 52 root NOFD /proc/52/fd (opendir: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 54 root cwd unknown /proc/54/cwd (readlink: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 54 root rtd unknown /proc/54/root (readlink: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 54 root txt unknown /proc/54/exe (readlink: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 54 root NOFD /proc/54/fd (opendir: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kdevtmpfs 55 root cwd unknown /proc/55/cwd (readlink: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kdevtmpfs 55 root rtd unknown /proc/55/root (readlink: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kdevtmpfs 55 root txt unknown /proc/55/exe (readlink: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kdevtmpfs 55 root NOFD /proc/55/fd (opendir: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 56 root cwd unknown /proc/56/cwd (readlink: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 56 root rtd unknown /proc/56/root (readlink: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 56 root txt unknown /proc/56/exe (readlink: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 56 root NOFD /proc/56/fd (opendir: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kauditd 57 root cwd unknown /proc/57/cwd (readlink: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kauditd 57 root rtd unknown /proc/57/root (readlink: Permission denied) 2024-03-11T20:01:35.905 INFO:tasks.workunit.client.0.smithi097.stdout:kauditd 57 root txt unknown /proc/57/exe (readlink: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:kauditd 57 root NOFD /proc/57/fd (opendir: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:khungtask 58 root cwd unknown /proc/58/cwd (readlink: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:khungtask 58 root rtd unknown /proc/58/root (readlink: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:khungtask 58 root txt unknown /proc/58/exe (readlink: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:khungtask 58 root NOFD /proc/58/fd (opendir: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:oom_reape 60 root cwd unknown /proc/60/cwd (readlink: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:oom_reape 60 root rtd unknown /proc/60/root (readlink: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:oom_reape 60 root txt unknown /proc/60/exe (readlink: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:oom_reape 60 root NOFD /proc/60/fd (opendir: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 62 root cwd unknown /proc/62/cwd (readlink: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 62 root rtd unknown /proc/62/root (readlink: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 62 root txt unknown /proc/62/exe (readlink: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 62 root NOFD /proc/62/fd (opendir: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:kcompactd 63 root cwd unknown /proc/63/cwd (readlink: Permission denied) 2024-03-11T20:01:35.906 INFO:tasks.workunit.client.0.smithi097.stdout:kcompactd 63 root rtd unknown /proc/63/root (readlink: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:kcompactd 63 root txt unknown /proc/63/exe (readlink: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:kcompactd 63 root NOFD /proc/63/fd (opendir: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:ksmd 64 root cwd unknown /proc/64/cwd (readlink: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:ksmd 64 root rtd unknown /proc/64/root (readlink: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:ksmd 64 root txt unknown /proc/64/exe (readlink: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:ksmd 64 root NOFD /proc/64/fd (opendir: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:khugepage 65 root cwd unknown /proc/65/cwd (readlink: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:khugepage 65 root rtd unknown /proc/65/root (readlink: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:khugepage 65 root txt unknown /proc/65/exe (readlink: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:khugepage 65 root NOFD /proc/65/fd (opendir: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 66 root cwd unknown /proc/66/cwd (readlink: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 66 root rtd unknown /proc/66/root (readlink: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 66 root txt unknown /proc/66/exe (readlink: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 66 root NOFD /proc/66/fd (opendir: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 67 root cwd unknown /proc/67/cwd (readlink: Permission denied) 2024-03-11T20:01:35.907 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 67 root rtd unknown /proc/67/root (readlink: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 67 root txt unknown /proc/67/exe (readlink: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 67 root NOFD /proc/67/fd (opendir: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 68 root cwd unknown /proc/68/cwd (readlink: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 68 root rtd unknown /proc/68/root (readlink: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 68 root txt unknown /proc/68/exe (readlink: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 68 root NOFD /proc/68/fd (opendir: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 69 root cwd unknown /proc/69/cwd (readlink: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 69 root rtd unknown /proc/69/root (readlink: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 69 root txt unknown /proc/69/exe (readlink: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 69 root NOFD /proc/69/fd (opendir: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:irq/9-acp 77 root cwd unknown /proc/77/cwd (readlink: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:irq/9-acp 77 root rtd unknown /proc/77/root (readlink: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:irq/9-acp 77 root txt unknown /proc/77/exe (readlink: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:irq/9-acp 77 root NOFD /proc/77/fd (opendir: Permission denied) 2024-03-11T20:01:35.908 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 78 root cwd unknown /proc/78/cwd (readlink: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 78 root rtd unknown /proc/78/root (readlink: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 78 root txt unknown /proc/78/exe (readlink: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 78 root NOFD /proc/78/fd (opendir: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 79 root cwd unknown /proc/79/cwd (readlink: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 79 root rtd unknown /proc/79/root (readlink: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 79 root txt unknown /proc/79/exe (readlink: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 79 root NOFD /proc/79/fd (opendir: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 80 root cwd unknown /proc/80/cwd (readlink: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 80 root rtd unknown /proc/80/root (readlink: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 80 root txt unknown /proc/80/exe (readlink: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 80 root NOFD /proc/80/fd (opendir: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 81 root cwd unknown /proc/81/cwd (readlink: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 81 root rtd unknown /proc/81/root (readlink: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 81 root txt unknown /proc/81/exe (readlink: Permission denied) 2024-03-11T20:01:35.909 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 81 root NOFD /proc/81/fd (opendir: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 82 root cwd unknown /proc/82/cwd (readlink: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 82 root rtd unknown /proc/82/root (readlink: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 82 root txt unknown /proc/82/exe (readlink: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 82 root NOFD /proc/82/fd (opendir: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:watchdogd 83 root cwd unknown /proc/83/cwd (readlink: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:watchdogd 83 root rtd unknown /proc/83/root (readlink: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:watchdogd 83 root txt unknown /proc/83/exe (readlink: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:watchdogd 83 root NOFD /proc/83/fd (opendir: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 84 root cwd unknown /proc/84/cwd (readlink: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 84 root rtd unknown /proc/84/root (readlink: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 84 root txt unknown /proc/84/exe (readlink: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 84 root NOFD /proc/84/fd (opendir: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:kswapd0 85 root cwd unknown /proc/85/cwd (readlink: Permission denied) 2024-03-11T20:01:35.910 INFO:tasks.workunit.client.0.smithi097.stdout:kswapd0 85 root rtd unknown /proc/85/root (readlink: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kswapd0 85 root txt unknown /proc/85/exe (readlink: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kswapd0 85 root NOFD /proc/85/fd (opendir: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87 root cwd unknown /proc/87/cwd (readlink: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87 root rtd unknown /proc/87/root (readlink: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87 root txt unknown /proc/87/exe (readlink: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87 root NOFD /proc/87/fd (opendir: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 91 root cwd unknown /proc/91/cwd (readlink: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 91 root rtd unknown /proc/91/root (readlink: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 91 root txt unknown /proc/91/exe (readlink: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 91 root NOFD /proc/91/fd (opendir: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 92 root cwd unknown /proc/92/cwd (readlink: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 92 root rtd unknown /proc/92/root (readlink: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 92 root txt unknown /proc/92/exe (readlink: Permission denied) 2024-03-11T20:01:35.911 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 92 root NOFD /proc/92/fd (opendir: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_0 93 root cwd unknown /proc/93/cwd (readlink: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_0 93 root rtd unknown /proc/93/root (readlink: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_0 93 root txt unknown /proc/93/exe (readlink: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_0 93 root NOFD /proc/93/fd (opendir: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 94 root cwd unknown /proc/94/cwd (readlink: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 94 root rtd unknown /proc/94/root (readlink: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 94 root txt unknown /proc/94/exe (readlink: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 94 root NOFD /proc/94/fd (opendir: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_1 95 root cwd unknown /proc/95/cwd (readlink: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_1 95 root rtd unknown /proc/95/root (readlink: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_1 95 root txt unknown /proc/95/exe (readlink: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_1 95 root NOFD /proc/95/fd (opendir: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 96 root cwd unknown /proc/96/cwd (readlink: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 96 root rtd unknown /proc/96/root (readlink: Permission denied) 2024-03-11T20:01:35.912 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 96 root txt unknown /proc/96/exe (readlink: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 96 root NOFD /proc/96/fd (opendir: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_2 97 root cwd unknown /proc/97/cwd (readlink: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_2 97 root rtd unknown /proc/97/root (readlink: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_2 97 root txt unknown /proc/97/exe (readlink: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_2 97 root NOFD /proc/97/fd (opendir: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 98 root cwd unknown /proc/98/cwd (readlink: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 98 root rtd unknown /proc/98/root (readlink: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 98 root txt unknown /proc/98/exe (readlink: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 98 root NOFD /proc/98/fd (opendir: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_3 99 root cwd unknown /proc/99/cwd (readlink: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_3 99 root rtd unknown /proc/99/root (readlink: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_3 99 root txt unknown /proc/99/exe (readlink: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_3 99 root NOFD /proc/99/fd (opendir: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 100 root cwd unknown /proc/100/cwd (readlink: Permission denied) 2024-03-11T20:01:35.913 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 100 root rtd unknown /proc/100/root (readlink: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 100 root txt unknown /proc/100/exe (readlink: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 100 root NOFD /proc/100/fd (opendir: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 102 root cwd unknown /proc/102/cwd (readlink: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 102 root rtd unknown /proc/102/root (readlink: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 102 root txt unknown /proc/102/exe (readlink: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 102 root NOFD /proc/102/fd (opendir: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 103 root cwd unknown /proc/103/cwd (readlink: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 103 root rtd unknown /proc/103/root (readlink: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 103 root txt unknown /proc/103/exe (readlink: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 103 root NOFD /proc/103/fd (opendir: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_4 104 root cwd unknown /proc/104/cwd (readlink: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_4 104 root rtd unknown /proc/104/root (readlink: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_4 104 root txt unknown /proc/104/exe (readlink: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_4 104 root NOFD /proc/104/fd (opendir: Permission denied) 2024-03-11T20:01:35.914 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 105 root cwd unknown /proc/105/cwd (readlink: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 105 root rtd unknown /proc/105/root (readlink: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 105 root txt unknown /proc/105/exe (readlink: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 105 root NOFD /proc/105/fd (opendir: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_5 106 root cwd unknown /proc/106/cwd (readlink: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_5 106 root rtd unknown /proc/106/root (readlink: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_5 106 root txt unknown /proc/106/exe (readlink: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_5 106 root NOFD /proc/106/fd (opendir: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 107 root cwd unknown /proc/107/cwd (readlink: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 107 root rtd unknown /proc/107/root (readlink: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 107 root txt unknown /proc/107/exe (readlink: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 107 root NOFD /proc/107/fd (opendir: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_6 108 root cwd unknown /proc/108/cwd (readlink: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_6 108 root rtd unknown /proc/108/root (readlink: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_6 108 root txt unknown /proc/108/exe (readlink: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_6 108 root NOFD /proc/108/fd (opendir: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 109 root cwd unknown /proc/109/cwd (readlink: Permission denied) 2024-03-11T20:01:35.915 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 109 root rtd unknown /proc/109/root (readlink: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 109 root txt unknown /proc/109/exe (readlink: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 109 root NOFD /proc/109/fd (opendir: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_7 110 root cwd unknown /proc/110/cwd (readlink: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_7 110 root rtd unknown /proc/110/root (readlink: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_7 110 root txt unknown /proc/110/exe (readlink: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_7 110 root NOFD /proc/110/fd (opendir: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 111 root cwd unknown /proc/111/cwd (readlink: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 111 root rtd unknown /proc/111/root (readlink: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 111 root txt unknown /proc/111/exe (readlink: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 111 root NOFD /proc/111/fd (opendir: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_8 112 root cwd unknown /proc/112/cwd (readlink: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_8 112 root rtd unknown /proc/112/root (readlink: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_8 112 root txt unknown /proc/112/exe (readlink: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_8 112 root NOFD /proc/112/fd (opendir: Permission denied) 2024-03-11T20:01:35.916 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 113 root cwd unknown /proc/113/cwd (readlink: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 113 root rtd unknown /proc/113/root (readlink: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 113 root txt unknown /proc/113/exe (readlink: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 113 root NOFD /proc/113/fd (opendir: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_9 114 root cwd unknown /proc/114/cwd (readlink: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_9 114 root rtd unknown /proc/114/root (readlink: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_9 114 root txt unknown /proc/114/exe (readlink: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:scsi_eh_9 114 root NOFD /proc/114/fd (opendir: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 115 root cwd unknown /proc/115/cwd (readlink: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 115 root rtd unknown /proc/115/root (readlink: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 115 root txt unknown /proc/115/exe (readlink: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 115 root NOFD /proc/115/fd (opendir: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 119 root cwd unknown /proc/119/cwd (readlink: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 119 root rtd unknown /proc/119/root (readlink: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 119 root txt unknown /proc/119/exe (readlink: Permission denied) 2024-03-11T20:01:35.917 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 119 root NOFD /proc/119/fd (opendir: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 120 root cwd unknown /proc/120/cwd (readlink: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 120 root rtd unknown /proc/120/root (readlink: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 120 root txt unknown /proc/120/exe (readlink: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 120 root NOFD /proc/120/fd (opendir: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 128 root cwd unknown /proc/128/cwd (readlink: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 128 root rtd unknown /proc/128/root (readlink: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 128 root txt unknown /proc/128/exe (readlink: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 128 root NOFD /proc/128/fd (opendir: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 129 root cwd unknown /proc/129/cwd (readlink: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 129 root rtd unknown /proc/129/root (readlink: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 129 root txt unknown /proc/129/exe (readlink: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 129 root NOFD /proc/129/fd (opendir: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 130 root cwd unknown /proc/130/cwd (readlink: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 130 root rtd unknown /proc/130/root (readlink: Permission denied) 2024-03-11T20:01:35.918 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 130 root txt unknown /proc/130/exe (readlink: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 130 root NOFD /proc/130/fd (opendir: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 131 root cwd unknown /proc/131/cwd (readlink: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 131 root rtd unknown /proc/131/root (readlink: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 131 root txt unknown /proc/131/exe (readlink: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 131 root NOFD /proc/131/fd (opendir: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 132 root cwd unknown /proc/132/cwd (readlink: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 132 root rtd unknown /proc/132/root (readlink: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 132 root txt unknown /proc/132/exe (readlink: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 132 root NOFD /proc/132/fd (opendir: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 133 root cwd unknown /proc/133/cwd (readlink: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 133 root rtd unknown /proc/133/root (readlink: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 133 root txt unknown /proc/133/exe (readlink: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 133 root NOFD /proc/133/fd (opendir: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 134 root cwd unknown /proc/134/cwd (readlink: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 134 root rtd unknown /proc/134/root (readlink: Permission denied) 2024-03-11T20:01:35.919 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 134 root txt unknown /proc/134/exe (readlink: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 134 root NOFD /proc/134/fd (opendir: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 141 root cwd unknown /proc/141/cwd (readlink: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 141 root rtd unknown /proc/141/root (readlink: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 141 root txt unknown /proc/141/exe (readlink: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 141 root NOFD /proc/141/fd (opendir: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 211 root cwd unknown /proc/211/cwd (readlink: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 211 root rtd unknown /proc/211/root (readlink: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 211 root txt unknown /proc/211/exe (readlink: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 211 root NOFD /proc/211/fd (opendir: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 251 root cwd unknown /proc/251/cwd (readlink: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 251 root rtd unknown /proc/251/root (readlink: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 251 root txt unknown /proc/251/exe (readlink: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 251 root NOFD /proc/251/fd (opendir: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 253 root cwd unknown /proc/253/cwd (readlink: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 253 root rtd unknown /proc/253/root (readlink: Permission denied) 2024-03-11T20:01:35.920 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 253 root txt unknown /proc/253/exe (readlink: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 253 root NOFD /proc/253/fd (opendir: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 258 root cwd unknown /proc/258/cwd (readlink: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 258 root rtd unknown /proc/258/root (readlink: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 258 root txt unknown /proc/258/exe (readlink: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 258 root NOFD /proc/258/fd (opendir: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 268 root cwd unknown /proc/268/cwd (readlink: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 268 root rtd unknown /proc/268/root (readlink: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 268 root txt unknown /proc/268/exe (readlink: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 268 root NOFD /proc/268/fd (opendir: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 402 root cwd unknown /proc/402/cwd (readlink: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 402 root rtd unknown /proc/402/root (readlink: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 402 root txt unknown /proc/402/exe (readlink: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 402 root NOFD /proc/402/fd (opendir: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 403 root cwd unknown /proc/403/cwd (readlink: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 403 root rtd unknown /proc/403/root (readlink: Permission denied) 2024-03-11T20:01:35.921 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 403 root txt unknown /proc/403/exe (readlink: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 403 root NOFD /proc/403/fd (opendir: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 404 root cwd unknown /proc/404/cwd (readlink: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 404 root rtd unknown /proc/404/root (readlink: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 404 root txt unknown /proc/404/exe (readlink: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 404 root NOFD /proc/404/fd (opendir: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 435 root cwd unknown /proc/435/cwd (readlink: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 435 root rtd unknown /proc/435/root (readlink: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 435 root txt unknown /proc/435/exe (readlink: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 435 root NOFD /proc/435/fd (opendir: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:jbd2/sda1 448 root cwd unknown /proc/448/cwd (readlink: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:jbd2/sda1 448 root rtd unknown /proc/448/root (readlink: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:jbd2/sda1 448 root txt unknown /proc/448/exe (readlink: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:jbd2/sda1 448 root NOFD /proc/448/fd (opendir: Permission denied) 2024-03-11T20:01:35.922 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 449 root cwd unknown /proc/449/cwd (readlink: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 449 root rtd unknown /proc/449/root (readlink: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 449 root txt unknown /proc/449/exe (readlink: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 449 root NOFD /proc/449/fd (opendir: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:systemd-j 524 root cwd unknown /proc/524/cwd (readlink: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:systemd-j 524 root rtd unknown /proc/524/root (readlink: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:systemd-j 524 root txt unknown /proc/524/exe (readlink: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:systemd-j 524 root NOFD /proc/524/fd (opendir: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:ext4lazyi 530 root cwd unknown /proc/530/cwd (readlink: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:ext4lazyi 530 root rtd unknown /proc/530/root (readlink: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:ext4lazyi 530 root txt unknown /proc/530/exe (readlink: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:ext4lazyi 530 root NOFD /proc/530/fd (opendir: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:systemd-u 544 root cwd unknown /proc/544/cwd (readlink: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:systemd-u 544 root rtd unknown /proc/544/root (readlink: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:systemd-u 544 root txt unknown /proc/544/exe (readlink: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:systemd-u 544 root NOFD /proc/544/fd (opendir: Permission denied) 2024-03-11T20:01:35.923 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 603 root cwd unknown /proc/603/cwd (readlink: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 603 root rtd unknown /proc/603/root (readlink: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 603 root txt unknown /proc/603/exe (readlink: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 603 root NOFD /proc/603/fd (opendir: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:rpcbind 611 rpc cwd unknown /proc/611/cwd (readlink: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:rpcbind 611 rpc rtd unknown /proc/611/root (readlink: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:rpcbind 611 rpc txt unknown /proc/611/exe (readlink: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:rpcbind 611 rpc NOFD /proc/611/fd (opendir: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:kipmi0 615 root cwd unknown /proc/615/cwd (readlink: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:kipmi0 615 root rtd unknown /proc/615/root (readlink: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:kipmi0 615 root txt unknown /proc/615/exe (readlink: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:kipmi0 615 root NOFD /proc/615/fd (opendir: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 617 root cwd unknown /proc/617/cwd (readlink: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 617 root rtd unknown /proc/617/root (readlink: Permission denied) 2024-03-11T20:01:35.924 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 617 root txt unknown /proc/617/exe (readlink: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 617 root NOFD /proc/617/fd (opendir: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:auditd 619 root cwd unknown /proc/619/cwd (readlink: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:auditd 619 root rtd unknown /proc/619/root (readlink: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:auditd 619 root txt unknown /proc/619/exe (readlink: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:auditd 619 root NOFD /proc/619/fd (opendir: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:auditd 619 620 auditd root cwd unknown /proc/619/task/620/cwd (readlink: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:auditd 619 620 auditd root rtd unknown /proc/619/task/620/root (readlink: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:auditd 619 620 auditd root txt unknown /proc/619/task/620/exe (readlink: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:auditd 619 620 auditd root NOFD /proc/619/task/620/fd (opendir: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:auditd 619 622 auditd root cwd unknown /proc/619/task/622/cwd (readlink: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:auditd 619 622 auditd root rtd unknown /proc/619/task/622/root (readlink: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:auditd 619 622 auditd root txt unknown /proc/619/task/622/exe (readlink: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:auditd 619 622 auditd root NOFD /proc/619/task/622/fd (opendir: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:sedispatc 621 root cwd unknown /proc/621/cwd (readlink: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:sedispatc 621 root rtd unknown /proc/621/root (readlink: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:sedispatc 621 root txt unknown /proc/621/exe (readlink: Permission denied) 2024-03-11T20:01:35.925 INFO:tasks.workunit.client.0.smithi097.stdout:sedispatc 621 root NOFD /proc/621/fd (opendir: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 624 root cwd unknown /proc/624/cwd (readlink: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 624 root rtd unknown /proc/624/root (readlink: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 624 root txt unknown /proc/624/exe (readlink: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 624 root NOFD /proc/624/fd (opendir: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 625 root cwd unknown /proc/625/cwd (readlink: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 625 root rtd unknown /proc/625/root (readlink: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 625 root txt unknown /proc/625/exe (readlink: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 625 root NOFD /proc/625/fd (opendir: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:dbus-brok 660 dbus cwd unknown /proc/660/cwd (readlink: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:dbus-brok 660 dbus rtd unknown /proc/660/root (readlink: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:dbus-brok 660 dbus txt unknown /proc/660/exe (readlink: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:dbus-brok 660 dbus NOFD /proc/660/fd (opendir: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:dbus-brok 661 dbus cwd unknown /proc/661/cwd (readlink: Permission denied) 2024-03-11T20:01:35.926 INFO:tasks.workunit.client.0.smithi097.stdout:dbus-brok 661 dbus rtd unknown /proc/661/root (readlink: Permission denied) 2024-03-11T20:01:35.927 INFO:tasks.workunit.client.0.smithi097.stdout:dbus-brok 661 dbus txt unknown /proc/661/exe (readlink: Permission denied) 2024-03-11T20:01:35.927 INFO:tasks.workunit.client.0.smithi097.stdout:dbus-brok 661 dbus NOFD /proc/661/fd (opendir: Permission denied) 2024-03-11T20:01:35.927 INFO:tasks.workunit.client.0.smithi097.stdout:NetworkMa 662 root cwd unknown /proc/662/cwd (readlink: Permission denied) 2024-03-11T20:01:35.927 INFO:tasks.workunit.client.0.smithi097.stdout:NetworkMa 662 root rtd unknown /proc/662/root (readlink: Permission denied) 2024-03-11T20:01:35.927 INFO:tasks.workunit.client.0.smithi097.stdout:NetworkMa 662 root txt unknown /proc/662/exe (readlink: Permission denied) 2024-03-11T20:01:35.927 INFO:tasks.workunit.client.0.smithi097.stdout:NetworkMa 662 root NOFD /proc/662/fd (opendir: Permission denied) 2024-03-11T20:01:35.927 INFO:tasks.workunit.client.0.smithi097.stdout:NetworkMa 662 689 gmain root cwd unknown /proc/662/task/689/cwd (readlink: Permission denied) 2024-03-11T20:01:35.927 INFO:tasks.workunit.client.0.smithi097.stdout:NetworkMa 662 689 gmain root rtd unknown /proc/662/task/689/root (readlink: Permission denied) 2024-03-11T20:01:35.927 INFO:tasks.workunit.client.0.smithi097.stdout:NetworkMa 662 689 gmain root txt unknown /proc/662/task/689/exe (readlink: Permission denied) 2024-03-11T20:01:35.927 INFO:tasks.workunit.client.0.smithi097.stdout:NetworkMa 662 689 gmain root NOFD /proc/662/task/689/fd (opendir: Permission denied) 2024-03-11T20:01:35.927 INFO:tasks.workunit.client.0.smithi097.stdout:NetworkMa 662 690 gdbus root cwd unknown /proc/662/task/690/cwd (readlink: Permission denied) 2024-03-11T20:01:35.927 INFO:tasks.workunit.client.0.smithi097.stdout:NetworkMa 662 690 gdbus root rtd unknown /proc/662/task/690/root (readlink: Permission denied) 2024-03-11T20:01:35.927 INFO:tasks.workunit.client.0.smithi097.stdout:NetworkMa 662 690 gdbus root txt unknown /proc/662/task/690/exe (readlink: Permission denied) 2024-03-11T20:01:35.929 INFO:tasks.workunit.client.0.smithi097.stdout:NetworkMa 662 690 gdbus root NOFD /proc/662/task/690/fd (opendir: Permission denied) 2024-03-11T20:01:35.929 INFO:tasks.workunit.client.0.smithi097.stdout:irqbalanc 665 root cwd unknown /proc/665/cwd (readlink: Permission denied) 2024-03-11T20:01:35.929 INFO:tasks.workunit.client.0.smithi097.stdout:irqbalanc 665 root rtd unknown /proc/665/root (readlink: Permission denied) 2024-03-11T20:01:35.929 INFO:tasks.workunit.client.0.smithi097.stdout:irqbalanc 665 root txt unknown /proc/665/exe (readlink: Permission denied) 2024-03-11T20:01:35.929 INFO:tasks.workunit.client.0.smithi097.stdout:irqbalanc 665 root NOFD /proc/665/fd (opendir: Permission denied) 2024-03-11T20:01:35.929 INFO:tasks.workunit.client.0.smithi097.stdout:irqbalanc 665 671 gmain root cwd unknown /proc/665/task/671/cwd (readlink: Permission denied) 2024-03-11T20:01:35.929 INFO:tasks.workunit.client.0.smithi097.stdout:irqbalanc 665 671 gmain root rtd unknown /proc/665/task/671/root (readlink: Permission denied) 2024-03-11T20:01:35.929 INFO:tasks.workunit.client.0.smithi097.stdout:irqbalanc 665 671 gmain root txt unknown /proc/665/task/671/exe (readlink: Permission denied) 2024-03-11T20:01:35.929 INFO:tasks.workunit.client.0.smithi097.stdout:irqbalanc 665 671 gmain root NOFD /proc/665/task/671/fd (opendir: Permission denied) 2024-03-11T20:01:35.929 INFO:tasks.workunit.client.0.smithi097.stdout:lsmd 666 libstoragemgmt cwd unknown /proc/666/cwd (readlink: Permission denied) 2024-03-11T20:01:35.929 INFO:tasks.workunit.client.0.smithi097.stdout:lsmd 666 libstoragemgmt rtd unknown /proc/666/root (readlink: Permission denied) 2024-03-11T20:01:35.929 INFO:tasks.workunit.client.0.smithi097.stdout:lsmd 666 libstoragemgmt txt unknown /proc/666/exe (readlink: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:lsmd 666 libstoragemgmt NOFD /proc/666/fd (opendir: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 polkitd cwd unknown /proc/667/cwd (readlink: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 polkitd rtd unknown /proc/667/root (readlink: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 polkitd txt unknown /proc/667/exe (readlink: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 polkitd NOFD /proc/667/fd (opendir: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 679 gmain polkitd cwd unknown /proc/667/task/679/cwd (readlink: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 679 gmain polkitd rtd unknown /proc/667/task/679/root (readlink: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 679 gmain polkitd txt unknown /proc/667/task/679/exe (readlink: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 679 gmain polkitd NOFD /proc/667/task/679/fd (opendir: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 680 gdbus polkitd cwd unknown /proc/667/task/680/cwd (readlink: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 680 gdbus polkitd rtd unknown /proc/667/task/680/root (readlink: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 680 gdbus polkitd txt unknown /proc/667/task/680/exe (readlink: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 680 gdbus polkitd NOFD /proc/667/task/680/fd (opendir: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 681 JS\x20Hel polkitd cwd unknown /proc/667/task/681/cwd (readlink: Permission denied) 2024-03-11T20:01:35.930 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 681 JS\x20Hel polkitd rtd unknown /proc/667/task/681/root (readlink: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 681 JS\x20Hel polkitd txt unknown /proc/667/task/681/exe (readlink: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 681 JS\x20Hel polkitd NOFD /proc/667/task/681/fd (opendir: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 682 JS\x20Hel polkitd cwd unknown /proc/667/task/682/cwd (readlink: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 682 JS\x20Hel polkitd rtd unknown /proc/667/task/682/root (readlink: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 682 JS\x20Hel polkitd txt unknown /proc/667/task/682/exe (readlink: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 682 JS\x20Hel polkitd NOFD /proc/667/task/682/fd (opendir: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 683 JS\x20Hel polkitd cwd unknown /proc/667/task/683/cwd (readlink: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 683 JS\x20Hel polkitd rtd unknown /proc/667/task/683/root (readlink: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 683 JS\x20Hel polkitd txt unknown /proc/667/task/683/exe (readlink: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 683 JS\x20Hel polkitd NOFD /proc/667/task/683/fd (opendir: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 684 JS\x20Hel polkitd cwd unknown /proc/667/task/684/cwd (readlink: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 684 JS\x20Hel polkitd rtd unknown /proc/667/task/684/root (readlink: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 684 JS\x20Hel polkitd txt unknown /proc/667/task/684/exe (readlink: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 684 JS\x20Hel polkitd NOFD /proc/667/task/684/fd (opendir: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 685 JS\x20Hel polkitd cwd unknown /proc/667/task/685/cwd (readlink: Permission denied) 2024-03-11T20:01:35.931 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 685 JS\x20Hel polkitd rtd unknown /proc/667/task/685/root (readlink: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 685 JS\x20Hel polkitd txt unknown /proc/667/task/685/exe (readlink: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 685 JS\x20Hel polkitd NOFD /proc/667/task/685/fd (opendir: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 686 JS\x20Hel polkitd cwd unknown /proc/667/task/686/cwd (readlink: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 686 JS\x20Hel polkitd rtd unknown /proc/667/task/686/root (readlink: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 686 JS\x20Hel polkitd txt unknown /proc/667/task/686/exe (readlink: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 686 JS\x20Hel polkitd NOFD /proc/667/task/686/fd (opendir: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 687 JS\x20Hel polkitd cwd unknown /proc/667/task/687/cwd (readlink: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 687 JS\x20Hel polkitd rtd unknown /proc/667/task/687/root (readlink: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 687 JS\x20Hel polkitd txt unknown /proc/667/task/687/exe (readlink: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 687 JS\x20Hel polkitd NOFD /proc/667/task/687/fd (opendir: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 688 JS\x20Hel polkitd cwd unknown /proc/667/task/688/cwd (readlink: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 688 JS\x20Hel polkitd rtd unknown /proc/667/task/688/root (readlink: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 688 JS\x20Hel polkitd txt unknown /proc/667/task/688/exe (readlink: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 688 JS\x20Hel polkitd NOFD /proc/667/task/688/fd (opendir: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 691 polkitd polkitd cwd unknown /proc/667/task/691/cwd (readlink: Permission denied) 2024-03-11T20:01:35.932 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 691 polkitd polkitd rtd unknown /proc/667/task/691/root (readlink: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 691 polkitd polkitd txt unknown /proc/667/task/691/exe (readlink: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:polkitd 667 691 polkitd polkitd NOFD /proc/667/task/691/fd (opendir: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:systemd-l 670 root cwd unknown /proc/670/cwd (readlink: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:systemd-l 670 root rtd unknown /proc/670/root (readlink: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:systemd-l 670 root txt unknown /proc/670/exe (readlink: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:systemd-l 670 root NOFD /proc/670/fd (opendir: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 716 root cwd unknown /proc/716/cwd (readlink: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 716 root rtd unknown /proc/716/root (readlink: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 716 root txt unknown /proc/716/exe (readlink: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 716 root NOFD /proc/716/fd (opendir: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:tuned 721 root cwd unknown /proc/721/cwd (readlink: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:tuned 721 root rtd unknown /proc/721/root (readlink: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:tuned 721 root txt unknown /proc/721/exe (readlink: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:tuned 721 root NOFD /proc/721/fd (opendir: Permission denied) 2024-03-11T20:01:35.933 INFO:tasks.workunit.client.0.smithi097.stdout:tuned 721 812 tuned root cwd unknown /proc/721/task/812/cwd (readlink: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:tuned 721 812 tuned root rtd unknown /proc/721/task/812/root (readlink: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:tuned 721 812 tuned root txt unknown /proc/721/task/812/exe (readlink: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:tuned 721 812 tuned root NOFD /proc/721/task/812/fd (opendir: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 root cwd unknown /proc/731/cwd (readlink: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 root rtd unknown /proc/731/root (readlink: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 root txt unknown /proc/731/exe (readlink: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 root NOFD /proc/731/fd (opendir: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 732 gssproxy root cwd unknown /proc/731/task/732/cwd (readlink: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 732 gssproxy root rtd unknown /proc/731/task/732/root (readlink: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 732 gssproxy root txt unknown /proc/731/task/732/exe (readlink: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 732 gssproxy root NOFD /proc/731/task/732/fd (opendir: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 733 gssproxy root cwd unknown /proc/731/task/733/cwd (readlink: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 733 gssproxy root rtd unknown /proc/731/task/733/root (readlink: Permission denied) 2024-03-11T20:01:35.934 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 733 gssproxy root txt unknown /proc/731/task/733/exe (readlink: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 733 gssproxy root NOFD /proc/731/task/733/fd (opendir: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 734 gssproxy root cwd unknown /proc/731/task/734/cwd (readlink: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 734 gssproxy root rtd unknown /proc/731/task/734/root (readlink: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 734 gssproxy root txt unknown /proc/731/task/734/exe (readlink: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 734 gssproxy root NOFD /proc/731/task/734/fd (opendir: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 735 gssproxy root cwd unknown /proc/731/task/735/cwd (readlink: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 735 gssproxy root rtd unknown /proc/731/task/735/root (readlink: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 735 gssproxy root txt unknown /proc/731/task/735/exe (readlink: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 735 gssproxy root NOFD /proc/731/task/735/fd (opendir: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 736 gssproxy root cwd unknown /proc/731/task/736/cwd (readlink: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 736 gssproxy root rtd unknown /proc/731/task/736/root (readlink: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 736 gssproxy root txt unknown /proc/731/task/736/exe (readlink: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:gssproxy 731 736 gssproxy root NOFD /proc/731/task/736/fd (opendir: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:atd 738 root cwd unknown /proc/738/cwd (readlink: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:atd 738 root rtd unknown /proc/738/root (readlink: Permission denied) 2024-03-11T20:01:35.935 INFO:tasks.workunit.client.0.smithi097.stdout:atd 738 root txt unknown /proc/738/exe (readlink: Permission denied) 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:atd 738 root NOFD /proc/738/fd (opendir: Permission denied) 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:crond 739 root cwd unknown /proc/739/cwd (readlink: Permission denied) 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:crond 739 root rtd unknown /proc/739/root (readlink: Permission denied) 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:crond 739 root txt unknown /proc/739/exe (readlink: Permission denied) 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:crond 739 root NOFD /proc/739/fd (opendir: Permission denied) 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:nrpe 827 nrpe cwd unknown /proc/827/cwd (readlink: Permission denied) 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:nrpe 827 nrpe rtd unknown /proc/827/root (readlink: Permission denied) 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:nrpe 827 nrpe txt unknown /proc/827/exe (readlink: Permission denied) 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:nrpe 827 nrpe NOFD /proc/827/fd (opendir: Permission denied) 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu cwd DIR 8,1 4096 2 / 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu rtd DIR 8,1 4096 2 / 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu txt REG 8,1 102120 20353 /usr/lib/systemd/systemd 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu DEL REG 8,1 1167 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-11T20:01:35.936 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu DEL REG 8,1 10007 /usr/lib64/libcrypto.so.3.0.7 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu DEL REG 8,1 1705 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu DEL REG 8,1 10126 /usr/lib64/libblkid.so.1.1.0 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-11T20:01:35.937 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu DEL REG 8,1 10156 /usr/lib64/libmount.so.1.1.0 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 2174088 49251 /usr/lib64/systemd/libsystemd-core-252.so 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu DEL REG 8,1 9806 /usr/lib64/libacl.so.1.1.2301 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu mem REG 0,19 0 19 /sys/fs/selinux/status 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 0r CHR 1,3 0t0 4 /dev/null 2024-03-11T20:01:35.938 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 1u unix 0x0000000000000000 0t0 10379 type=STREAM (CONNECTED) 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 2u unix 0x0000000000000000 0t0 10379 type=STREAM (CONNECTED) 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 3u unix 0x0000000000000000 0t0 6531 type=DGRAM (CONNECTED) 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 4u a_inode 0,14 0 50 [eventpoll:5,6,8,9,11,12,14,15,16,17,19,20,21] 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 5u a_inode 0,14 0 50 [signalfd] 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 6r a_inode 0,14 0 50 inotify 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 7r DIR 0,25 0 3353 /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 8u a_inode 0,14 0 50 [timerfd] 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 9u a_inode 0,14 0 50 [eventpoll:10,13] 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 10r REG 0,20 0 10409 /proc/1151/mountinfo 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 11r a_inode 0,14 0 50 inotify 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 12u unix 0x0000000000000000 0t0 10439 /run/user/1000/bus type=STREAM (LISTEN) 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 13r a_inode 0,14 0 50 inotify 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 14r REG 0,20 0 4026532078 /proc/swaps 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 15u netlink 0t0 10421 KOBJECT_UEVENT 2024-03-11T20:01:35.939 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 16u unix 0x0000000000000000 0t0 10426 /run/user/1000/systemd/notify type=DGRAM (UNCONNECTED) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 17u unix 0x0000000000000000 0t0 10427 type=DGRAM (CONNECTED) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 18u unix 0x0000000000000000 0t0 10428 type=DGRAM (CONNECTED) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 19u unix 0x0000000000000000 0t0 10429 /run/user/1000/systemd/private type=STREAM (LISTEN) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 20u unix 0x0000000000000000 0t0 10431 type=STREAM (CONNECTED) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 1151 ubuntu 21u a_inode 0,14 0 50 [timerfd] 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:(sd-pam) 1160 ubuntu cwd unknown /proc/1160/cwd (readlink: Permission denied) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:(sd-pam) 1160 ubuntu rtd unknown /proc/1160/root (readlink: Permission denied) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:(sd-pam) 1160 ubuntu txt unknown /proc/1160/exe (readlink: Permission denied) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:(sd-pam) 1160 ubuntu NOFD /proc/1160/fd (opendir: Permission denied) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:agetty 1161 root cwd unknown /proc/1161/cwd (readlink: Permission denied) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:agetty 1161 root rtd unknown /proc/1161/root (readlink: Permission denied) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:agetty 1161 root txt unknown /proc/1161/exe (readlink: Permission denied) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:agetty 1161 root NOFD /proc/1161/fd (opendir: Permission denied) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:agetty 1164 root cwd unknown /proc/1164/cwd (readlink: Permission denied) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:agetty 1164 root rtd unknown /proc/1164/root (readlink: Permission denied) 2024-03-11T20:01:35.940 INFO:tasks.workunit.client.0.smithi097.stdout:agetty 1164 root txt unknown /proc/1164/exe (readlink: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:agetty 1164 root NOFD /proc/1164/fd (opendir: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:rsyslogd 2128 root cwd unknown /proc/2128/cwd (readlink: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:rsyslogd 2128 root rtd unknown /proc/2128/root (readlink: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:rsyslogd 2128 root txt unknown /proc/2128/exe (readlink: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:rsyslogd 2128 root NOFD /proc/2128/fd (opendir: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:rsyslogd 2128 2150 in:imjour root cwd unknown /proc/2128/task/2150/cwd (readlink: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:rsyslogd 2128 2150 in:imjour root rtd unknown /proc/2128/task/2150/root (readlink: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:rsyslogd 2128 2150 in:imjour root txt unknown /proc/2128/task/2150/exe (readlink: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:rsyslogd 2128 2150 in:imjour root NOFD /proc/2128/task/2150/fd (opendir: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:rsyslogd 2128 2151 rs:main root cwd unknown /proc/2128/task/2151/cwd (readlink: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:rsyslogd 2128 2151 rs:main root rtd unknown /proc/2128/task/2151/root (readlink: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:rsyslogd 2128 2151 rs:main root txt unknown /proc/2128/task/2151/exe (readlink: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:rsyslogd 2128 2151 rs:main root NOFD /proc/2128/task/2151/fd (opendir: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:smartd 15275 root cwd unknown /proc/15275/cwd (readlink: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:smartd 15275 root rtd unknown /proc/15275/root (readlink: Permission denied) 2024-03-11T20:01:35.941 INFO:tasks.workunit.client.0.smithi097.stdout:smartd 15275 root txt unknown /proc/15275/exe (readlink: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:smartd 15275 root NOFD /proc/15275/fd (opendir: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20643 root cwd unknown /proc/20643/cwd (readlink: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20643 root rtd unknown /proc/20643/root (readlink: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20643 root txt unknown /proc/20643/exe (readlink: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20643 root NOFD /proc/20643/fd (opendir: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20673 root cwd unknown /proc/20673/cwd (readlink: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20673 root rtd unknown /proc/20673/root (readlink: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20673 root txt unknown /proc/20673/exe (readlink: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20673 root NOFD /proc/20673/fd (opendir: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20702 root cwd unknown /proc/20702/cwd (readlink: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20702 root rtd unknown /proc/20702/root (readlink: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20702 root txt unknown /proc/20702/exe (readlink: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20702 root NOFD /proc/20702/fd (opendir: Permission denied) 2024-03-11T20:01:35.942 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20731 root cwd unknown /proc/20731/cwd (readlink: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20731 root rtd unknown /proc/20731/root (readlink: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20731 root txt unknown /proc/20731/exe (readlink: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20731 root NOFD /proc/20731/fd (opendir: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20760 root cwd unknown /proc/20760/cwd (readlink: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20760 root rtd unknown /proc/20760/root (readlink: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20760 root txt unknown /proc/20760/exe (readlink: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20760 root NOFD /proc/20760/fd (opendir: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20924 root cwd unknown /proc/20924/cwd (readlink: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20924 root rtd unknown /proc/20924/root (readlink: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20924 root txt unknown /proc/20924/exe (readlink: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20924 root NOFD /proc/20924/fd (opendir: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20925 root cwd unknown /proc/20925/cwd (readlink: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20925 root rtd unknown /proc/20925/root (readlink: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20925 root txt unknown /proc/20925/exe (readlink: Permission denied) 2024-03-11T20:01:35.943 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20925 root NOFD /proc/20925/fd (opendir: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20926 root cwd unknown /proc/20926/cwd (readlink: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20926 root rtd unknown /proc/20926/root (readlink: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20926 root txt unknown /proc/20926/exe (readlink: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20926 root NOFD /proc/20926/fd (opendir: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20927 root cwd unknown /proc/20927/cwd (readlink: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20927 root rtd unknown /proc/20927/root (readlink: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20927 root txt unknown /proc/20927/exe (readlink: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20927 root NOFD /proc/20927/fd (opendir: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20928 root cwd unknown /proc/20928/cwd (readlink: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20928 root rtd unknown /proc/20928/root (readlink: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20928 root txt unknown /proc/20928/exe (readlink: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20928 root NOFD /proc/20928/fd (opendir: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20929 root cwd unknown /proc/20929/cwd (readlink: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20929 root rtd unknown /proc/20929/root (readlink: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20929 root txt unknown /proc/20929/exe (readlink: Permission denied) 2024-03-11T20:01:35.944 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20929 root NOFD /proc/20929/fd (opendir: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20930 root cwd unknown /proc/20930/cwd (readlink: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20930 root rtd unknown /proc/20930/root (readlink: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20930 root txt unknown /proc/20930/exe (readlink: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20930 root NOFD /proc/20930/fd (opendir: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20932 root cwd unknown /proc/20932/cwd (readlink: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20932 root rtd unknown /proc/20932/root (readlink: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20932 root txt unknown /proc/20932/exe (readlink: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20932 root NOFD /proc/20932/fd (opendir: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20933 root cwd unknown /proc/20933/cwd (readlink: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20933 root rtd unknown /proc/20933/root (readlink: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20933 root txt unknown /proc/20933/exe (readlink: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 20933 root NOFD /proc/20933/fd (opendir: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:xfsaild/d 20934 root cwd unknown /proc/20934/cwd (readlink: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:xfsaild/d 20934 root rtd unknown /proc/20934/root (readlink: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:xfsaild/d 20934 root txt unknown /proc/20934/exe (readlink: Permission denied) 2024-03-11T20:01:35.945 INFO:tasks.workunit.client.0.smithi097.stdout:xfsaild/d 20934 root NOFD /proc/20934/fd (opendir: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 22940 root cwd unknown /proc/22940/cwd (readlink: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 22940 root rtd unknown /proc/22940/root (readlink: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 22940 root txt unknown /proc/22940/exe (readlink: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 22940 root NOFD /proc/22940/fd (opendir: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 22943 ubuntu cwd unknown /proc/22943/cwd (readlink: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 22943 ubuntu rtd unknown /proc/22943/root (readlink: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 22943 ubuntu txt unknown /proc/22943/exe (readlink: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 22943 ubuntu NOFD /proc/22943/fd (opendir: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:chronyd 23003 chrony cwd unknown /proc/23003/cwd (readlink: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:chronyd 23003 chrony rtd unknown /proc/23003/root (readlink: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:chronyd 23003 chrony txt unknown /proc/23003/exe (readlink: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:chronyd 23003 chrony NOFD /proc/23003/fd (opendir: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-cras 23346 ceph cwd unknown /proc/23346/cwd (readlink: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-cras 23346 ceph rtd unknown /proc/23346/root (readlink: Permission denied) 2024-03-11T20:01:35.946 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-cras 23346 ceph txt unknown /proc/23346/exe (readlink: Permission denied) 2024-03-11T20:01:35.947 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-cras 23346 ceph NOFD /proc/23346/fd (opendir: Permission denied) 2024-03-11T20:01:35.947 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 24696 root cwd unknown /proc/24696/cwd (readlink: Permission denied) 2024-03-11T20:01:35.947 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 24696 root rtd unknown /proc/24696/root (readlink: Permission denied) 2024-03-11T20:01:35.947 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 24696 root txt unknown /proc/24696/exe (readlink: Permission denied) 2024-03-11T20:01:35.947 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 24696 root NOFD /proc/24696/fd (opendir: Permission denied) 2024-03-11T20:01:35.947 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 24895 root cwd unknown /proc/24895/cwd (readlink: Permission denied) 2024-03-11T20:01:35.947 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 24895 root rtd unknown /proc/24895/root (readlink: Permission denied) 2024-03-11T20:01:35.947 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 24895 root txt unknown /proc/24895/exe (readlink: Permission denied) 2024-03-11T20:01:35.947 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 24895 root NOFD /proc/24895/fd (opendir: Permission denied) 2024-03-11T20:01:35.947 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 30883 root cwd unknown /proc/30883/cwd (readlink: Permission denied) 2024-03-11T20:01:35.947 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 30883 root rtd unknown /proc/30883/root (readlink: Permission denied) 2024-03-11T20:01:35.947 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 30883 root txt unknown /proc/30883/exe (readlink: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 30883 root NOFD /proc/30883/fd (opendir: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 30899 root cwd unknown /proc/30899/cwd (readlink: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 30899 root rtd unknown /proc/30899/root (readlink: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 30899 root txt unknown /proc/30899/exe (readlink: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 30899 root NOFD /proc/30899/fd (opendir: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 30915 root cwd unknown /proc/30915/cwd (readlink: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 30915 root rtd unknown /proc/30915/root (readlink: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 30915 root txt unknown /proc/30915/exe (readlink: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 30915 root NOFD /proc/30915/fd (opendir: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 30929 root cwd unknown /proc/30929/cwd (readlink: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 30929 root rtd unknown /proc/30929/root (readlink: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 30929 root txt unknown /proc/30929/exe (readlink: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 30929 root NOFD /proc/30929/fd (opendir: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 31194 root cwd unknown /proc/31194/cwd (readlink: Permission denied) 2024-03-11T20:01:35.948 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 31194 root rtd unknown /proc/31194/root (readlink: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 31194 root txt unknown /proc/31194/exe (readlink: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 31194 root NOFD /proc/31194/fd (opendir: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 31894 root cwd unknown /proc/31894/cwd (readlink: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 31894 root rtd unknown /proc/31894/root (readlink: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 31894 root txt unknown /proc/31894/exe (readlink: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 31894 root NOFD /proc/31894/fd (opendir: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 31896 root cwd unknown /proc/31896/cwd (readlink: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 31896 root rtd unknown /proc/31896/root (readlink: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 31896 root txt unknown /proc/31896/exe (readlink: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 31896 root NOFD /proc/31896/fd (opendir: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 ceph cwd unknown /proc/31898/cwd (readlink: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 ceph rtd unknown /proc/31898/root (readlink: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 ceph txt unknown /proc/31898/exe (readlink: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 ceph NOFD /proc/31898/fd (opendir: Permission denied) 2024-03-11T20:01:35.949 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31899 log ceph cwd unknown /proc/31898/task/31899/cwd (readlink: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31899 log ceph rtd unknown /proc/31898/task/31899/root (readlink: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31899 log ceph txt unknown /proc/31898/task/31899/exe (readlink: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31899 log ceph NOFD /proc/31898/task/31899/fd (opendir: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31900 service ceph cwd unknown /proc/31898/task/31900/cwd (readlink: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31900 service ceph rtd unknown /proc/31898/task/31900/root (readlink: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31900 service ceph txt unknown /proc/31898/task/31900/exe (readlink: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31900 service ceph NOFD /proc/31898/task/31900/fd (opendir: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31901 admin_soc ceph cwd unknown /proc/31898/task/31901/cwd (readlink: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31901 admin_soc ceph rtd unknown /proc/31898/task/31901/root (readlink: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31901 admin_soc ceph txt unknown /proc/31898/task/31901/exe (readlink: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31901 admin_soc ceph NOFD /proc/31898/task/31901/fd (opendir: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31902 signal_ha ceph cwd unknown /proc/31898/task/31902/cwd (readlink: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31902 signal_ha ceph rtd unknown /proc/31898/task/31902/root (readlink: Permission denied) 2024-03-11T20:01:35.950 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31902 signal_ha ceph txt unknown /proc/31898/task/31902/exe (readlink: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31902 signal_ha ceph NOFD /proc/31898/task/31902/fd (opendir: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31903 rocksdb:l ceph cwd unknown /proc/31898/task/31903/cwd (readlink: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31903 rocksdb:l ceph rtd unknown /proc/31898/task/31903/root (readlink: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31903 rocksdb:l ceph txt unknown /proc/31898/task/31903/exe (readlink: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31903 rocksdb:l ceph NOFD /proc/31898/task/31903/fd (opendir: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31904 rocksdb:h ceph cwd unknown /proc/31898/task/31904/cwd (readlink: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31904 rocksdb:h ceph rtd unknown /proc/31898/task/31904/root (readlink: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31904 rocksdb:h ceph txt unknown /proc/31898/task/31904/exe (readlink: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31904 rocksdb:h ceph NOFD /proc/31898/task/31904/fd (opendir: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31920 ceph-mon ceph cwd unknown /proc/31898/task/31920/cwd (readlink: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31920 ceph-mon ceph rtd unknown /proc/31898/task/31920/root (readlink: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31920 ceph-mon ceph txt unknown /proc/31898/task/31920/exe (readlink: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31920 ceph-mon ceph NOFD /proc/31898/task/31920/fd (opendir: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31921 fn_monsto ceph cwd unknown /proc/31898/task/31921/cwd (readlink: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31921 fn_monsto ceph rtd unknown /proc/31898/task/31921/root (readlink: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31921 fn_monsto ceph txt unknown /proc/31898/task/31921/exe (readlink: Permission denied) 2024-03-11T20:01:35.951 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31921 fn_monsto ceph NOFD /proc/31898/task/31921/fd (opendir: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31922 msgr-work ceph cwd unknown /proc/31898/task/31922/cwd (readlink: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31922 msgr-work ceph rtd unknown /proc/31898/task/31922/root (readlink: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31922 msgr-work ceph txt unknown /proc/31898/task/31922/exe (readlink: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31922 msgr-work ceph NOFD /proc/31898/task/31922/fd (opendir: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31923 msgr-work ceph cwd unknown /proc/31898/task/31923/cwd (readlink: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31923 msgr-work ceph rtd unknown /proc/31898/task/31923/root (readlink: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31923 msgr-work ceph txt unknown /proc/31898/task/31923/exe (readlink: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31923 msgr-work ceph NOFD /proc/31898/task/31923/fd (opendir: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31924 msgr-work ceph cwd unknown /proc/31898/task/31924/cwd (readlink: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31924 msgr-work ceph rtd unknown /proc/31898/task/31924/root (readlink: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31924 msgr-work ceph txt unknown /proc/31898/task/31924/exe (readlink: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31924 msgr-work ceph NOFD /proc/31898/task/31924/fd (opendir: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31925 OpHistory ceph cwd unknown /proc/31898/task/31925/cwd (readlink: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31925 OpHistory ceph rtd unknown /proc/31898/task/31925/root (readlink: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31925 OpHistory ceph txt unknown /proc/31898/task/31925/exe (readlink: Permission denied) 2024-03-11T20:01:35.952 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31925 OpHistory ceph NOFD /proc/31898/task/31925/fd (opendir: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31926 fin ceph cwd unknown /proc/31898/task/31926/cwd (readlink: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31926 fin ceph rtd unknown /proc/31898/task/31926/root (readlink: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31926 fin ceph txt unknown /proc/31898/task/31926/exe (readlink: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31926 fin ceph NOFD /proc/31898/task/31926/fd (opendir: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31927 safe_time ceph cwd unknown /proc/31898/task/31927/cwd (readlink: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31927 safe_time ceph rtd unknown /proc/31898/task/31927/root (readlink: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31927 safe_time ceph txt unknown /proc/31898/task/31927/exe (readlink: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31927 safe_time ceph NOFD /proc/31898/task/31927/fd (opendir: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31928 cpu_tp ceph cwd unknown /proc/31898/task/31928/cwd (readlink: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31928 cpu_tp ceph rtd unknown /proc/31898/task/31928/root (readlink: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31928 cpu_tp ceph txt unknown /proc/31898/task/31928/exe (readlink: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31928 cpu_tp ceph NOFD /proc/31898/task/31928/fd (opendir: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31929 cpu_tp ceph cwd unknown /proc/31898/task/31929/cwd (readlink: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31929 cpu_tp ceph rtd unknown /proc/31898/task/31929/root (readlink: Permission denied) 2024-03-11T20:01:35.953 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31929 cpu_tp ceph txt unknown /proc/31898/task/31929/exe (readlink: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31929 cpu_tp ceph NOFD /proc/31898/task/31929/fd (opendir: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31930 cpu_tp ceph cwd unknown /proc/31898/task/31930/cwd (readlink: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31930 cpu_tp ceph rtd unknown /proc/31898/task/31930/root (readlink: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31930 cpu_tp ceph txt unknown /proc/31898/task/31930/exe (readlink: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31930 cpu_tp ceph NOFD /proc/31898/task/31930/fd (opendir: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31931 cpu_tp ceph cwd unknown /proc/31898/task/31931/cwd (readlink: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31931 cpu_tp ceph rtd unknown /proc/31898/task/31931/root (readlink: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31931 cpu_tp ceph txt unknown /proc/31898/task/31931/exe (readlink: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31931 cpu_tp ceph NOFD /proc/31898/task/31931/fd (opendir: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31932 ms_dispat ceph cwd unknown /proc/31898/task/31932/cwd (readlink: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31932 ms_dispat ceph rtd unknown /proc/31898/task/31932/root (readlink: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31932 ms_dispat ceph txt unknown /proc/31898/task/31932/exe (readlink: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31932 ms_dispat ceph NOFD /proc/31898/task/31932/fd (opendir: Permission denied) 2024-03-11T20:01:35.954 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31933 ms_local ceph cwd unknown /proc/31898/task/31933/cwd (readlink: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31933 ms_local ceph rtd unknown /proc/31898/task/31933/root (readlink: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31933 ms_local ceph txt unknown /proc/31898/task/31933/exe (readlink: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31933 ms_local ceph NOFD /proc/31898/task/31933/fd (opendir: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31934 safe_time ceph cwd unknown /proc/31898/task/31934/cwd (readlink: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31934 safe_time ceph rtd unknown /proc/31898/task/31934/root (readlink: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31934 safe_time ceph txt unknown /proc/31898/task/31934/exe (readlink: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31934 safe_time ceph NOFD /proc/31898/task/31934/fd (opendir: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31935 ms_dispat ceph cwd unknown /proc/31898/task/31935/cwd (readlink: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31935 ms_dispat ceph rtd unknown /proc/31898/task/31935/root (readlink: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31935 ms_dispat ceph txt unknown /proc/31898/task/31935/exe (readlink: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31935 ms_dispat ceph NOFD /proc/31898/task/31935/fd (opendir: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31936 ms_local ceph cwd unknown /proc/31898/task/31936/cwd (readlink: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31936 ms_local ceph rtd unknown /proc/31898/task/31936/root (readlink: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31936 ms_local ceph txt unknown /proc/31898/task/31936/exe (readlink: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 31936 ms_local ceph NOFD /proc/31898/task/31936/fd (opendir: Permission denied) 2024-03-11T20:01:35.955 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 67624 rstore_co ceph cwd unknown /proc/31898/task/67624/cwd (readlink: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 67624 rstore_co ceph rtd unknown /proc/31898/task/67624/root (readlink: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 67624 rstore_co ceph txt unknown /proc/31898/task/67624/exe (readlink: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mon 31898 67624 rstore_co ceph NOFD /proc/31898/task/67624/fd (opendir: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 32126 root cwd unknown /proc/32126/cwd (readlink: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 32126 root rtd unknown /proc/32126/root (readlink: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 32126 root txt unknown /proc/32126/exe (readlink: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 32126 root NOFD /proc/32126/fd (opendir: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 32128 root cwd unknown /proc/32128/cwd (readlink: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 32128 root rtd unknown /proc/32128/root (readlink: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 32128 root txt unknown /proc/32128/exe (readlink: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 32128 root NOFD /proc/32128/fd (opendir: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 ceph cwd unknown /proc/32130/cwd (readlink: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 ceph rtd unknown /proc/32130/root (readlink: Permission denied) 2024-03-11T20:01:35.956 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 ceph txt unknown /proc/32130/exe (readlink: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 ceph NOFD /proc/32130/fd (opendir: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87319 log ceph cwd unknown /proc/32130/task/87319/cwd (readlink: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87319 log ceph rtd unknown /proc/32130/task/87319/root (readlink: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87319 log ceph txt unknown /proc/32130/task/87319/exe (readlink: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87319 log ceph NOFD /proc/32130/task/87319/fd (opendir: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87321 msgr-work ceph cwd unknown /proc/32130/task/87321/cwd (readlink: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87321 msgr-work ceph rtd unknown /proc/32130/task/87321/root (readlink: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87321 msgr-work ceph txt unknown /proc/32130/task/87321/exe (readlink: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87321 msgr-work ceph NOFD /proc/32130/task/87321/fd (opendir: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87322 msgr-work ceph cwd unknown /proc/32130/task/87322/cwd (readlink: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87322 msgr-work ceph rtd unknown /proc/32130/task/87322/root (readlink: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87322 msgr-work ceph txt unknown /proc/32130/task/87322/exe (readlink: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87322 msgr-work ceph NOFD /proc/32130/task/87322/fd (opendir: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87323 msgr-work ceph cwd unknown /proc/32130/task/87323/cwd (readlink: Permission denied) 2024-03-11T20:01:35.957 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87323 msgr-work ceph rtd unknown /proc/32130/task/87323/root (readlink: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87323 msgr-work ceph txt unknown /proc/32130/task/87323/exe (readlink: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87323 msgr-work ceph NOFD /proc/32130/task/87323/fd (opendir: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87327 service ceph cwd unknown /proc/32130/task/87327/cwd (readlink: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87327 service ceph rtd unknown /proc/32130/task/87327/root (readlink: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87327 service ceph txt unknown /proc/32130/task/87327/exe (readlink: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87327 service ceph NOFD /proc/32130/task/87327/fd (opendir: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87328 admin_soc ceph cwd unknown /proc/32130/task/87328/cwd (readlink: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87328 admin_soc ceph rtd unknown /proc/32130/task/87328/root (readlink: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87328 admin_soc ceph txt unknown /proc/32130/task/87328/exe (readlink: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87328 admin_soc ceph NOFD /proc/32130/task/87328/fd (opendir: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87329 ceph_time ceph cwd unknown /proc/32130/task/87329/cwd (readlink: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87329 ceph_time ceph rtd unknown /proc/32130/task/87329/root (readlink: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87329 ceph_time ceph txt unknown /proc/32130/task/87329/exe (readlink: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87329 ceph_time ceph NOFD /proc/32130/task/87329/fd (opendir: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87330 fn_anonym ceph cwd unknown /proc/32130/task/87330/cwd (readlink: Permission denied) 2024-03-11T20:01:35.958 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87330 fn_anonym ceph rtd unknown /proc/32130/task/87330/root (readlink: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87330 fn_anonym ceph txt unknown /proc/32130/task/87330/exe (readlink: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87330 fn_anonym ceph NOFD /proc/32130/task/87330/fd (opendir: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87331 signal_ha ceph cwd unknown /proc/32130/task/87331/cwd (readlink: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87331 signal_ha ceph rtd unknown /proc/32130/task/87331/root (readlink: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87331 signal_ha ceph txt unknown /proc/32130/task/87331/exe (readlink: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87331 signal_ha ceph NOFD /proc/32130/task/87331/fd (opendir: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87332 mgrsb-fin ceph cwd unknown /proc/32130/task/87332/cwd (readlink: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87332 mgrsb-fin ceph rtd unknown /proc/32130/task/87332/root (readlink: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87332 mgrsb-fin ceph txt unknown /proc/32130/task/87332/exe (readlink: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87332 mgrsb-fin ceph NOFD /proc/32130/task/87332/fd (opendir: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87333 ms_dispat ceph cwd unknown /proc/32130/task/87333/cwd (readlink: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87333 ms_dispat ceph rtd unknown /proc/32130/task/87333/root (readlink: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87333 ms_dispat ceph txt unknown /proc/32130/task/87333/exe (readlink: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87333 ms_dispat ceph NOFD /proc/32130/task/87333/fd (opendir: Permission denied) 2024-03-11T20:01:35.959 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87334 ms_local ceph cwd unknown /proc/32130/task/87334/cwd (readlink: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87334 ms_local ceph rtd unknown /proc/32130/task/87334/root (readlink: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87334 ms_local ceph txt unknown /proc/32130/task/87334/exe (readlink: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87334 ms_local ceph NOFD /proc/32130/task/87334/fd (opendir: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87335 io_contex ceph cwd unknown /proc/32130/task/87335/cwd (readlink: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87335 io_contex ceph rtd unknown /proc/32130/task/87335/root (readlink: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87335 io_contex ceph txt unknown /proc/32130/task/87335/exe (readlink: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87335 io_contex ceph NOFD /proc/32130/task/87335/fd (opendir: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87336 io_contex ceph cwd unknown /proc/32130/task/87336/cwd (readlink: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87336 io_contex ceph rtd unknown /proc/32130/task/87336/root (readlink: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87336 io_contex ceph txt unknown /proc/32130/task/87336/exe (readlink: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87336 io_contex ceph NOFD /proc/32130/task/87336/fd (opendir: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87337 safe_time ceph cwd unknown /proc/32130/task/87337/cwd (readlink: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87337 safe_time ceph rtd unknown /proc/32130/task/87337/root (readlink: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87337 safe_time ceph txt unknown /proc/32130/task/87337/exe (readlink: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87337 safe_time ceph NOFD /proc/32130/task/87337/fd (opendir: Permission denied) 2024-03-11T20:01:35.960 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87338 safe_time ceph cwd unknown /proc/32130/task/87338/cwd (readlink: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87338 safe_time ceph rtd unknown /proc/32130/task/87338/root (readlink: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87338 safe_time ceph txt unknown /proc/32130/task/87338/exe (readlink: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87338 safe_time ceph NOFD /proc/32130/task/87338/fd (opendir: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87339 safe_time ceph cwd unknown /proc/32130/task/87339/cwd (readlink: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87339 safe_time ceph rtd unknown /proc/32130/task/87339/root (readlink: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87339 safe_time ceph txt unknown /proc/32130/task/87339/exe (readlink: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87339 safe_time ceph NOFD /proc/32130/task/87339/fd (opendir: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87340 fn_anonym ceph cwd unknown /proc/32130/task/87340/cwd (readlink: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87340 fn_anonym ceph rtd unknown /proc/32130/task/87340/root (readlink: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87340 fn_anonym ceph txt unknown /proc/32130/task/87340/exe (readlink: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87340 fn_anonym ceph NOFD /proc/32130/task/87340/fd (opendir: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87341 flusher ceph cwd unknown /proc/32130/task/87341/cwd (readlink: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87341 flusher ceph rtd unknown /proc/32130/task/87341/root (readlink: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87341 flusher ceph txt unknown /proc/32130/task/87341/exe (readlink: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87341 flusher ceph NOFD /proc/32130/task/87341/fd (opendir: Permission denied) 2024-03-11T20:01:35.961 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87342 safe_time ceph cwd unknown /proc/32130/task/87342/cwd (readlink: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87342 safe_time ceph rtd unknown /proc/32130/task/87342/root (readlink: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87342 safe_time ceph txt unknown /proc/32130/task/87342/exe (readlink: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87342 safe_time ceph NOFD /proc/32130/task/87342/fd (opendir: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87506 mgr-fin ceph cwd unknown /proc/32130/task/87506/cwd (readlink: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87506 mgr-fin ceph rtd unknown /proc/32130/task/87506/root (readlink: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87506 mgr-fin ceph txt unknown /proc/32130/task/87506/exe (readlink: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87506 mgr-fin ceph NOFD /proc/32130/task/87506/fd (opendir: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87507 ms_dispat ceph cwd unknown /proc/32130/task/87507/cwd (readlink: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87507 ms_dispat ceph rtd unknown /proc/32130/task/87507/root (readlink: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87507 ms_dispat ceph txt unknown /proc/32130/task/87507/exe (readlink: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87507 ms_dispat ceph NOFD /proc/32130/task/87507/fd (opendir: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87508 ms_local ceph cwd unknown /proc/32130/task/87508/cwd (readlink: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87508 ms_local ceph rtd unknown /proc/32130/task/87508/root (readlink: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87508 ms_local ceph txt unknown /proc/32130/task/87508/exe (readlink: Permission denied) 2024-03-11T20:01:35.962 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87508 ms_local ceph NOFD /proc/32130/task/87508/fd (opendir: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87509 safe_time ceph cwd unknown /proc/32130/task/87509/cwd (readlink: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87509 safe_time ceph rtd unknown /proc/32130/task/87509/root (readlink: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87509 safe_time ceph txt unknown /proc/32130/task/87509/exe (readlink: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87509 safe_time ceph NOFD /proc/32130/task/87509/fd (opendir: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87510 cmdfin ceph cwd unknown /proc/32130/task/87510/cwd (readlink: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87510 cmdfin ceph rtd unknown /proc/32130/task/87510/root (readlink: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87510 cmdfin ceph txt unknown /proc/32130/task/87510/exe (readlink: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87510 cmdfin ceph NOFD /proc/32130/task/87510/fd (opendir: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87514 io_contex ceph cwd unknown /proc/32130/task/87514/cwd (readlink: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87514 io_contex ceph rtd unknown /proc/32130/task/87514/root (readlink: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87514 io_contex ceph txt unknown /proc/32130/task/87514/exe (readlink: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87514 io_contex ceph NOFD /proc/32130/task/87514/fd (opendir: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87515 io_contex ceph cwd unknown /proc/32130/task/87515/cwd (readlink: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87515 io_contex ceph rtd unknown /proc/32130/task/87515/root (readlink: Permission denied) 2024-03-11T20:01:35.963 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87515 io_contex ceph txt unknown /proc/32130/task/87515/exe (readlink: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87515 io_contex ceph NOFD /proc/32130/task/87515/fd (opendir: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87516 ceph_time ceph cwd unknown /proc/32130/task/87516/cwd (readlink: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87516 ceph_time ceph rtd unknown /proc/32130/task/87516/root (readlink: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87516 ceph_time ceph txt unknown /proc/32130/task/87516/exe (readlink: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87516 ceph_time ceph NOFD /proc/32130/task/87516/fd (opendir: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87517 ms_dispat ceph cwd unknown /proc/32130/task/87517/cwd (readlink: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87517 ms_dispat ceph rtd unknown /proc/32130/task/87517/root (readlink: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87517 ms_dispat ceph txt unknown /proc/32130/task/87517/exe (readlink: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87517 ms_dispat ceph NOFD /proc/32130/task/87517/fd (opendir: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87518 ms_local ceph cwd unknown /proc/32130/task/87518/cwd (readlink: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87518 ms_local ceph rtd unknown /proc/32130/task/87518/root (readlink: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87518 ms_local ceph txt unknown /proc/32130/task/87518/exe (readlink: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87518 ms_local ceph NOFD /proc/32130/task/87518/fd (opendir: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87519 safe_time ceph cwd unknown /proc/32130/task/87519/cwd (readlink: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87519 safe_time ceph rtd unknown /proc/32130/task/87519/root (readlink: Permission denied) 2024-03-11T20:01:35.964 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87519 safe_time ceph txt unknown /proc/32130/task/87519/exe (readlink: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87519 safe_time ceph NOFD /proc/32130/task/87519/fd (opendir: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87520 safe_time ceph cwd unknown /proc/32130/task/87520/cwd (readlink: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87520 safe_time ceph rtd unknown /proc/32130/task/87520/root (readlink: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87520 safe_time ceph txt unknown /proc/32130/task/87520/exe (readlink: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87520 safe_time ceph NOFD /proc/32130/task/87520/fd (opendir: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87521 balancer ceph cwd unknown /proc/32130/task/87521/cwd (readlink: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87521 balancer ceph rtd unknown /proc/32130/task/87521/root (readlink: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87521 balancer ceph txt unknown /proc/32130/task/87521/exe (readlink: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87521 balancer ceph NOFD /proc/32130/task/87521/fd (opendir: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87522 m-fin-bal ceph cwd unknown /proc/32130/task/87522/cwd (readlink: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87522 m-fin-bal ceph rtd unknown /proc/32130/task/87522/root (readlink: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87522 m-fin-bal ceph txt unknown /proc/32130/task/87522/exe (readlink: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87522 m-fin-bal ceph NOFD /proc/32130/task/87522/fd (opendir: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87523 mgr-fin ceph cwd unknown /proc/32130/task/87523/cwd (readlink: Permission denied) 2024-03-11T20:01:35.965 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87523 mgr-fin ceph rtd unknown /proc/32130/task/87523/root (readlink: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87523 mgr-fin ceph txt unknown /proc/32130/task/87523/exe (readlink: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87523 mgr-fin ceph NOFD /proc/32130/task/87523/fd (opendir: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87524 mgr-fin ceph cwd unknown /proc/32130/task/87524/cwd (readlink: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87524 mgr-fin ceph rtd unknown /proc/32130/task/87524/root (readlink: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87524 mgr-fin ceph txt unknown /proc/32130/task/87524/exe (readlink: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87524 mgr-fin ceph NOFD /proc/32130/task/87524/fd (opendir: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87525 mgr-fin ceph cwd unknown /proc/32130/task/87525/cwd (readlink: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87525 mgr-fin ceph rtd unknown /proc/32130/task/87525/root (readlink: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87525 mgr-fin ceph txt unknown /proc/32130/task/87525/exe (readlink: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87525 mgr-fin ceph NOFD /proc/32130/task/87525/fd (opendir: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87526 mgr-fin ceph cwd unknown /proc/32130/task/87526/cwd (readlink: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87526 mgr-fin ceph rtd unknown /proc/32130/task/87526/root (readlink: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87526 mgr-fin ceph txt unknown /proc/32130/task/87526/exe (readlink: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87526 mgr-fin ceph NOFD /proc/32130/task/87526/fd (opendir: Permission denied) 2024-03-11T20:01:35.966 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87527 mgr-fin ceph cwd unknown /proc/32130/task/87527/cwd (readlink: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87527 mgr-fin ceph rtd unknown /proc/32130/task/87527/root (readlink: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87527 mgr-fin ceph txt unknown /proc/32130/task/87527/exe (readlink: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87527 mgr-fin ceph NOFD /proc/32130/task/87527/fd (opendir: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87528 mgr-fin ceph cwd unknown /proc/32130/task/87528/cwd (readlink: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87528 mgr-fin ceph rtd unknown /proc/32130/task/87528/root (readlink: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87528 mgr-fin ceph txt unknown /proc/32130/task/87528/exe (readlink: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87528 mgr-fin ceph NOFD /proc/32130/task/87528/fd (opendir: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87529 mgr-fin ceph cwd unknown /proc/32130/task/87529/cwd (readlink: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87529 mgr-fin ceph rtd unknown /proc/32130/task/87529/root (readlink: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87529 mgr-fin ceph txt unknown /proc/32130/task/87529/exe (readlink: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87529 mgr-fin ceph NOFD /proc/32130/task/87529/fd (opendir: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87530 mgr-fin ceph cwd unknown /proc/32130/task/87530/cwd (readlink: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87530 mgr-fin ceph rtd unknown /proc/32130/task/87530/root (readlink: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87530 mgr-fin ceph txt unknown /proc/32130/task/87530/exe (readlink: Permission denied) 2024-03-11T20:01:35.967 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87530 mgr-fin ceph NOFD /proc/32130/task/87530/fd (opendir: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87531 mgr-fin ceph cwd unknown /proc/32130/task/87531/cwd (readlink: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87531 mgr-fin ceph rtd unknown /proc/32130/task/87531/root (readlink: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87531 mgr-fin ceph txt unknown /proc/32130/task/87531/exe (readlink: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87531 mgr-fin ceph NOFD /proc/32130/task/87531/fd (opendir: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87532 mgr-fin ceph cwd unknown /proc/32130/task/87532/cwd (readlink: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87532 mgr-fin ceph rtd unknown /proc/32130/task/87532/root (readlink: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87532 mgr-fin ceph txt unknown /proc/32130/task/87532/exe (readlink: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87532 mgr-fin ceph NOFD /proc/32130/task/87532/fd (opendir: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87533 mgr-fin ceph cwd unknown /proc/32130/task/87533/cwd (readlink: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87533 mgr-fin ceph rtd unknown /proc/32130/task/87533/root (readlink: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87533 mgr-fin ceph txt unknown /proc/32130/task/87533/exe (readlink: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87533 mgr-fin ceph NOFD /proc/32130/task/87533/fd (opendir: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87534 mgr-fin ceph cwd unknown /proc/32130/task/87534/cwd (readlink: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87534 mgr-fin ceph rtd unknown /proc/32130/task/87534/root (readlink: Permission denied) 2024-03-11T20:01:35.968 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87534 mgr-fin ceph txt unknown /proc/32130/task/87534/exe (readlink: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87534 mgr-fin ceph NOFD /proc/32130/task/87534/fd (opendir: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87535 mgr-fin ceph cwd unknown /proc/32130/task/87535/cwd (readlink: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87535 mgr-fin ceph rtd unknown /proc/32130/task/87535/root (readlink: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87535 mgr-fin ceph txt unknown /proc/32130/task/87535/exe (readlink: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87535 mgr-fin ceph NOFD /proc/32130/task/87535/fd (opendir: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87536 mgr-fin ceph cwd unknown /proc/32130/task/87536/cwd (readlink: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87536 mgr-fin ceph rtd unknown /proc/32130/task/87536/root (readlink: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87536 mgr-fin ceph txt unknown /proc/32130/task/87536/exe (readlink: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87536 mgr-fin ceph NOFD /proc/32130/task/87536/fd (opendir: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87537 mgr-fin ceph cwd unknown /proc/32130/task/87537/cwd (readlink: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87537 mgr-fin ceph rtd unknown /proc/32130/task/87537/root (readlink: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87537 mgr-fin ceph txt unknown /proc/32130/task/87537/exe (readlink: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87537 mgr-fin ceph NOFD /proc/32130/task/87537/fd (opendir: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87538 mgr-fin ceph cwd unknown /proc/32130/task/87538/cwd (readlink: Permission denied) 2024-03-11T20:01:35.969 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87538 mgr-fin ceph rtd unknown /proc/32130/task/87538/root (readlink: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87538 mgr-fin ceph txt unknown /proc/32130/task/87538/exe (readlink: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87538 mgr-fin ceph NOFD /proc/32130/task/87538/fd (opendir: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87539 cephadm ceph cwd unknown /proc/32130/task/87539/cwd (readlink: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87539 cephadm ceph rtd unknown /proc/32130/task/87539/root (readlink: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87539 cephadm ceph txt unknown /proc/32130/task/87539/exe (readlink: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87539 cephadm ceph NOFD /proc/32130/task/87539/fd (opendir: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87540 m-fin-cep ceph cwd unknown /proc/32130/task/87540/cwd (readlink: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87540 m-fin-cep ceph rtd unknown /proc/32130/task/87540/root (readlink: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87540 m-fin-cep ceph txt unknown /proc/32130/task/87540/exe (readlink: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87540 m-fin-cep ceph NOFD /proc/32130/task/87540/fd (opendir: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87541 mgr-fin ceph cwd unknown /proc/32130/task/87541/cwd (readlink: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87541 mgr-fin ceph rtd unknown /proc/32130/task/87541/root (readlink: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87541 mgr-fin ceph txt unknown /proc/32130/task/87541/exe (readlink: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87541 mgr-fin ceph NOFD /proc/32130/task/87541/fd (opendir: Permission denied) 2024-03-11T20:01:35.970 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87542 mgr-fin ceph cwd unknown /proc/32130/task/87542/cwd (readlink: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87542 mgr-fin ceph rtd unknown /proc/32130/task/87542/root (readlink: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87542 mgr-fin ceph txt unknown /proc/32130/task/87542/exe (readlink: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87542 mgr-fin ceph NOFD /proc/32130/task/87542/fd (opendir: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87543 cephadm ceph cwd unknown /proc/32130/task/87543/cwd (readlink: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87543 cephadm ceph rtd unknown /proc/32130/task/87543/root (readlink: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87543 cephadm ceph txt unknown /proc/32130/task/87543/exe (readlink: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87543 cephadm ceph NOFD /proc/32130/task/87543/fd (opendir: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87544 mgr-fin ceph cwd unknown /proc/32130/task/87544/cwd (readlink: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87544 mgr-fin ceph rtd unknown /proc/32130/task/87544/root (readlink: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87544 mgr-fin ceph txt unknown /proc/32130/task/87544/exe (readlink: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87544 mgr-fin ceph NOFD /proc/32130/task/87544/fd (opendir: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87545 crash ceph cwd unknown /proc/32130/task/87545/cwd (readlink: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87545 crash ceph rtd unknown /proc/32130/task/87545/root (readlink: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87545 crash ceph txt unknown /proc/32130/task/87545/exe (readlink: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87545 crash ceph NOFD /proc/32130/task/87545/fd (opendir: Permission denied) 2024-03-11T20:01:35.971 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87546 m-fin-cra ceph cwd unknown /proc/32130/task/87546/cwd (readlink: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87546 m-fin-cra ceph rtd unknown /proc/32130/task/87546/root (readlink: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87546 m-fin-cra ceph txt unknown /proc/32130/task/87546/exe (readlink: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87546 m-fin-cra ceph NOFD /proc/32130/task/87546/fd (opendir: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87547 mgr-fin ceph cwd unknown /proc/32130/task/87547/cwd (readlink: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87547 mgr-fin ceph rtd unknown /proc/32130/task/87547/root (readlink: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87547 mgr-fin ceph txt unknown /proc/32130/task/87547/exe (readlink: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87547 mgr-fin ceph NOFD /proc/32130/task/87547/fd (opendir: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87548 dashboard ceph cwd unknown /proc/32130/task/87548/cwd (readlink: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87548 dashboard ceph rtd unknown /proc/32130/task/87548/root (readlink: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87548 dashboard ceph txt unknown /proc/32130/task/87548/exe (readlink: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87548 dashboard ceph NOFD /proc/32130/task/87548/fd (opendir: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87549 mgr-fin ceph cwd unknown /proc/32130/task/87549/cwd (readlink: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87549 mgr-fin ceph rtd unknown /proc/32130/task/87549/root (readlink: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87549 mgr-fin ceph txt unknown /proc/32130/task/87549/exe (readlink: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87549 mgr-fin ceph NOFD /proc/32130/task/87549/fd (opendir: Permission denied) 2024-03-11T20:01:35.972 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87550 m-fin-das ceph cwd unknown /proc/32130/task/87550/cwd (readlink: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87550 m-fin-das ceph rtd unknown /proc/32130/task/87550/root (readlink: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87550 m-fin-das ceph txt unknown /proc/32130/task/87550/exe (readlink: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87550 m-fin-das ceph NOFD /proc/32130/task/87550/fd (opendir: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87551 mgr-fin ceph cwd unknown /proc/32130/task/87551/cwd (readlink: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87551 mgr-fin ceph rtd unknown /proc/32130/task/87551/root (readlink: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87551 mgr-fin ceph txt unknown /proc/32130/task/87551/exe (readlink: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87551 mgr-fin ceph NOFD /proc/32130/task/87551/fd (opendir: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87552 devicehea ceph cwd unknown /proc/32130/task/87552/cwd (readlink: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87552 devicehea ceph rtd unknown /proc/32130/task/87552/root (readlink: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87552 devicehea ceph txt unknown /proc/32130/task/87552/exe (readlink: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87552 devicehea ceph NOFD /proc/32130/task/87552/fd (opendir: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87553 mgr-fin ceph cwd unknown /proc/32130/task/87553/cwd (readlink: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87553 mgr-fin ceph rtd unknown /proc/32130/task/87553/root (readlink: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87553 mgr-fin ceph txt unknown /proc/32130/task/87553/exe (readlink: Permission denied) 2024-03-11T20:01:35.973 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87553 mgr-fin ceph NOFD /proc/32130/task/87553/fd (opendir: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87554 m-fin-dev ceph cwd unknown /proc/32130/task/87554/cwd (readlink: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87554 m-fin-dev ceph rtd unknown /proc/32130/task/87554/root (readlink: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87554 m-fin-dev ceph txt unknown /proc/32130/task/87554/exe (readlink: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87554 m-fin-dev ceph NOFD /proc/32130/task/87554/fd (opendir: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87555 mgr-fin ceph cwd unknown /proc/32130/task/87555/cwd (readlink: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87555 mgr-fin ceph rtd unknown /proc/32130/task/87555/root (readlink: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87555 mgr-fin ceph txt unknown /proc/32130/task/87555/exe (readlink: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87555 mgr-fin ceph NOFD /proc/32130/task/87555/fd (opendir: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87556 mgr-fin ceph cwd unknown /proc/32130/task/87556/cwd (readlink: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87556 mgr-fin ceph rtd unknown /proc/32130/task/87556/root (readlink: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87556 mgr-fin ceph txt unknown /proc/32130/task/87556/exe (readlink: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87556 mgr-fin ceph NOFD /proc/32130/task/87556/fd (opendir: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87557 mgr-fin ceph cwd unknown /proc/32130/task/87557/cwd (readlink: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87557 mgr-fin ceph rtd unknown /proc/32130/task/87557/root (readlink: Permission denied) 2024-03-11T20:01:35.974 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87557 mgr-fin ceph txt unknown /proc/32130/task/87557/exe (readlink: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87557 mgr-fin ceph NOFD /proc/32130/task/87557/fd (opendir: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87558 mgr-fin ceph cwd unknown /proc/32130/task/87558/cwd (readlink: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87558 mgr-fin ceph rtd unknown /proc/32130/task/87558/root (readlink: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87558 mgr-fin ceph txt unknown /proc/32130/task/87558/exe (readlink: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87558 mgr-fin ceph NOFD /proc/32130/task/87558/fd (opendir: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87560 mgr-fin ceph cwd unknown /proc/32130/task/87560/cwd (readlink: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87560 mgr-fin ceph rtd unknown /proc/32130/task/87560/root (readlink: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87560 mgr-fin ceph txt unknown /proc/32130/task/87560/exe (readlink: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87560 mgr-fin ceph NOFD /proc/32130/task/87560/fd (opendir: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87561 m-fin-ios ceph cwd unknown /proc/32130/task/87561/cwd (readlink: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87561 m-fin-ios ceph rtd unknown /proc/32130/task/87561/root (readlink: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87561 m-fin-ios ceph txt unknown /proc/32130/task/87561/exe (readlink: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87561 m-fin-ios ceph NOFD /proc/32130/task/87561/fd (opendir: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87563 m-fin-nfs ceph cwd unknown /proc/32130/task/87563/cwd (readlink: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87563 m-fin-nfs ceph rtd unknown /proc/32130/task/87563/root (readlink: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87563 m-fin-nfs ceph txt unknown /proc/32130/task/87563/exe (readlink: Permission denied) 2024-03-11T20:01:35.975 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87563 m-fin-nfs ceph NOFD /proc/32130/task/87563/fd (opendir: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87565 m-fin-orc ceph cwd unknown /proc/32130/task/87565/cwd (readlink: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87565 m-fin-orc ceph rtd unknown /proc/32130/task/87565/root (readlink: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87565 m-fin-orc ceph txt unknown /proc/32130/task/87565/exe (readlink: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87565 m-fin-orc ceph NOFD /proc/32130/task/87565/fd (opendir: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87566 pg_autosc ceph cwd unknown /proc/32130/task/87566/cwd (readlink: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87566 pg_autosc ceph rtd unknown /proc/32130/task/87566/root (readlink: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87566 pg_autosc ceph txt unknown /proc/32130/task/87566/exe (readlink: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87566 pg_autosc ceph NOFD /proc/32130/task/87566/fd (opendir: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87567 m-fin-pg_ ceph cwd unknown /proc/32130/task/87567/cwd (readlink: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87567 m-fin-pg_ ceph rtd unknown /proc/32130/task/87567/root (readlink: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87567 m-fin-pg_ ceph txt unknown /proc/32130/task/87567/exe (readlink: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87567 m-fin-pg_ ceph NOFD /proc/32130/task/87567/fd (opendir: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87568 progress ceph cwd unknown /proc/32130/task/87568/cwd (readlink: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87568 progress ceph rtd unknown /proc/32130/task/87568/root (readlink: Permission denied) 2024-03-11T20:01:35.976 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87568 progress ceph txt unknown /proc/32130/task/87568/exe (readlink: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87568 progress ceph NOFD /proc/32130/task/87568/fd (opendir: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87569 m-fin-pro ceph cwd unknown /proc/32130/task/87569/cwd (readlink: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87569 m-fin-pro ceph rtd unknown /proc/32130/task/87569/root (readlink: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87569 m-fin-pro ceph txt unknown /proc/32130/task/87569/exe (readlink: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87569 m-fin-pro ceph NOFD /proc/32130/task/87569/fd (opendir: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87573 io_contex ceph cwd unknown /proc/32130/task/87573/cwd (readlink: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87573 io_contex ceph rtd unknown /proc/32130/task/87573/root (readlink: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87573 io_contex ceph txt unknown /proc/32130/task/87573/exe (readlink: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87573 io_contex ceph NOFD /proc/32130/task/87573/fd (opendir: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87574 io_contex ceph cwd unknown /proc/32130/task/87574/cwd (readlink: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87574 io_contex ceph rtd unknown /proc/32130/task/87574/root (readlink: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87574 io_contex ceph txt unknown /proc/32130/task/87574/exe (readlink: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87574 io_contex ceph NOFD /proc/32130/task/87574/fd (opendir: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87575 ceph_time ceph cwd unknown /proc/32130/task/87575/cwd (readlink: Permission denied) 2024-03-11T20:01:35.977 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87575 ceph_time ceph rtd unknown /proc/32130/task/87575/root (readlink: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87575 ceph_time ceph txt unknown /proc/32130/task/87575/exe (readlink: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87575 ceph_time ceph NOFD /proc/32130/task/87575/fd (opendir: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87576 ms_dispat ceph cwd unknown /proc/32130/task/87576/cwd (readlink: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87576 ms_dispat ceph rtd unknown /proc/32130/task/87576/root (readlink: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87576 ms_dispat ceph txt unknown /proc/32130/task/87576/exe (readlink: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87576 ms_dispat ceph NOFD /proc/32130/task/87576/fd (opendir: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87577 ms_local ceph cwd unknown /proc/32130/task/87577/cwd (readlink: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87577 ms_local ceph rtd unknown /proc/32130/task/87577/root (readlink: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87577 ms_local ceph txt unknown /proc/32130/task/87577/exe (readlink: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87577 ms_local ceph NOFD /proc/32130/task/87577/fd (opendir: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87578 safe_time ceph cwd unknown /proc/32130/task/87578/cwd (readlink: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87578 safe_time ceph rtd unknown /proc/32130/task/87578/root (readlink: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87578 safe_time ceph txt unknown /proc/32130/task/87578/exe (readlink: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87578 safe_time ceph NOFD /proc/32130/task/87578/fd (opendir: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87579 safe_time ceph cwd unknown /proc/32130/task/87579/cwd (readlink: Permission denied) 2024-03-11T20:01:35.978 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87579 safe_time ceph rtd unknown /proc/32130/task/87579/root (readlink: Permission denied) 2024-03-11T20:01:35.979 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87579 safe_time ceph txt unknown /proc/32130/task/87579/exe (readlink: Permission denied) 2024-03-11T20:01:35.979 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87579 safe_time ceph NOFD /proc/32130/task/87579/fd (opendir: Permission denied) 2024-03-11T20:01:35.979 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87580 mgr-fin ceph cwd unknown /proc/32130/task/87580/cwd (readlink: Permission denied) 2024-03-11T20:01:35.979 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87580 mgr-fin ceph rtd unknown /proc/32130/task/87580/root (readlink: Permission denied) 2024-03-11T20:01:35.979 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87580 mgr-fin ceph txt unknown /proc/32130/task/87580/exe (readlink: Permission denied) 2024-03-11T20:01:35.979 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87580 mgr-fin ceph NOFD /proc/32130/task/87580/fd (opendir: Permission denied) 2024-03-11T20:01:35.979 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87582 m-fin-rbd ceph cwd unknown /proc/32130/task/87582/cwd (readlink: Permission denied) 2024-03-11T20:01:35.979 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87582 m-fin-rbd ceph rtd unknown /proc/32130/task/87582/root (readlink: Permission denied) 2024-03-11T20:01:35.979 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87582 m-fin-rbd ceph txt unknown /proc/32130/task/87582/exe (readlink: Permission denied) 2024-03-11T20:01:35.979 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87582 m-fin-rbd ceph NOFD /proc/32130/task/87582/fd (opendir: Permission denied) 2024-03-11T20:01:35.979 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87583 restful ceph cwd unknown /proc/32130/task/87583/cwd (readlink: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87583 restful ceph rtd unknown /proc/32130/task/87583/root (readlink: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87583 restful ceph txt unknown /proc/32130/task/87583/exe (readlink: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87583 restful ceph NOFD /proc/32130/task/87583/fd (opendir: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87584 m-fin-res ceph cwd unknown /proc/32130/task/87584/cwd (readlink: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87584 m-fin-res ceph rtd unknown /proc/32130/task/87584/root (readlink: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87584 m-fin-res ceph txt unknown /proc/32130/task/87584/exe (readlink: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87584 m-fin-res ceph NOFD /proc/32130/task/87584/fd (opendir: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87585 mgr-fin ceph cwd unknown /proc/32130/task/87585/cwd (readlink: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87585 mgr-fin ceph rtd unknown /proc/32130/task/87585/root (readlink: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87585 mgr-fin ceph txt unknown /proc/32130/task/87585/exe (readlink: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87585 mgr-fin ceph NOFD /proc/32130/task/87585/fd (opendir: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87586 mgr-fin ceph cwd unknown /proc/32130/task/87586/cwd (readlink: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87586 mgr-fin ceph rtd unknown /proc/32130/task/87586/root (readlink: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87586 mgr-fin ceph txt unknown /proc/32130/task/87586/exe (readlink: Permission denied) 2024-03-11T20:01:35.980 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87586 mgr-fin ceph NOFD /proc/32130/task/87586/fd (opendir: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87587 mgr-fin ceph cwd unknown /proc/32130/task/87587/cwd (readlink: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87587 mgr-fin ceph rtd unknown /proc/32130/task/87587/root (readlink: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87587 mgr-fin ceph txt unknown /proc/32130/task/87587/exe (readlink: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87587 mgr-fin ceph NOFD /proc/32130/task/87587/fd (opendir: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87588 mgr-fin ceph cwd unknown /proc/32130/task/87588/cwd (readlink: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87588 mgr-fin ceph rtd unknown /proc/32130/task/87588/root (readlink: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87588 mgr-fin ceph txt unknown /proc/32130/task/87588/exe (readlink: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87588 mgr-fin ceph NOFD /proc/32130/task/87588/fd (opendir: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87589 mgr-fin ceph cwd unknown /proc/32130/task/87589/cwd (readlink: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87589 mgr-fin ceph rtd unknown /proc/32130/task/87589/root (readlink: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87589 mgr-fin ceph txt unknown /proc/32130/task/87589/exe (readlink: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87589 mgr-fin ceph NOFD /proc/32130/task/87589/fd (opendir: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87590 mgr-fin ceph cwd unknown /proc/32130/task/87590/cwd (readlink: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87590 mgr-fin ceph rtd unknown /proc/32130/task/87590/root (readlink: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87590 mgr-fin ceph txt unknown /proc/32130/task/87590/exe (readlink: Permission denied) 2024-03-11T20:01:35.981 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87590 mgr-fin ceph NOFD /proc/32130/task/87590/fd (opendir: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87594 io_contex ceph cwd unknown /proc/32130/task/87594/cwd (readlink: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87594 io_contex ceph rtd unknown /proc/32130/task/87594/root (readlink: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87594 io_contex ceph txt unknown /proc/32130/task/87594/exe (readlink: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87594 io_contex ceph NOFD /proc/32130/task/87594/fd (opendir: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87595 io_contex ceph cwd unknown /proc/32130/task/87595/cwd (readlink: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87595 io_contex ceph rtd unknown /proc/32130/task/87595/root (readlink: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87595 io_contex ceph txt unknown /proc/32130/task/87595/exe (readlink: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87595 io_contex ceph NOFD /proc/32130/task/87595/fd (opendir: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87596 ceph_time ceph cwd unknown /proc/32130/task/87596/cwd (readlink: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87596 ceph_time ceph rtd unknown /proc/32130/task/87596/root (readlink: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87596 ceph_time ceph txt unknown /proc/32130/task/87596/exe (readlink: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87596 ceph_time ceph NOFD /proc/32130/task/87596/fd (opendir: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87597 ms_dispat ceph cwd unknown /proc/32130/task/87597/cwd (readlink: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87597 ms_dispat ceph rtd unknown /proc/32130/task/87597/root (readlink: Permission denied) 2024-03-11T20:01:35.982 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87597 ms_dispat ceph txt unknown /proc/32130/task/87597/exe (readlink: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87597 ms_dispat ceph NOFD /proc/32130/task/87597/fd (opendir: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87598 ms_local ceph cwd unknown /proc/32130/task/87598/cwd (readlink: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87598 ms_local ceph rtd unknown /proc/32130/task/87598/root (readlink: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87598 ms_local ceph txt unknown /proc/32130/task/87598/exe (readlink: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87598 ms_local ceph NOFD /proc/32130/task/87598/fd (opendir: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87599 safe_time ceph cwd unknown /proc/32130/task/87599/cwd (readlink: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87599 safe_time ceph rtd unknown /proc/32130/task/87599/root (readlink: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87599 safe_time ceph txt unknown /proc/32130/task/87599/exe (readlink: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87599 safe_time ceph NOFD /proc/32130/task/87599/fd (opendir: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87600 safe_time ceph cwd unknown /proc/32130/task/87600/cwd (readlink: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87600 safe_time ceph rtd unknown /proc/32130/task/87600/root (readlink: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87600 safe_time ceph txt unknown /proc/32130/task/87600/exe (readlink: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87600 safe_time ceph NOFD /proc/32130/task/87600/fd (opendir: Permission denied) 2024-03-11T20:01:35.983 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87604 m-fin-sna ceph cwd unknown /proc/32130/task/87604/cwd (readlink: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87604 m-fin-sna ceph rtd unknown /proc/32130/task/87604/root (readlink: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87604 m-fin-sna ceph txt unknown /proc/32130/task/87604/exe (readlink: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87604 m-fin-sna ceph NOFD /proc/32130/task/87604/fd (opendir: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87606 m-fin-sta ceph cwd unknown /proc/32130/task/87606/cwd (readlink: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87606 m-fin-sta ceph rtd unknown /proc/32130/task/87606/root (readlink: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87606 m-fin-sta ceph txt unknown /proc/32130/task/87606/exe (readlink: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87606 m-fin-sta ceph NOFD /proc/32130/task/87606/fd (opendir: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87607 telemetry ceph cwd unknown /proc/32130/task/87607/cwd (readlink: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87607 telemetry ceph rtd unknown /proc/32130/task/87607/root (readlink: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87607 telemetry ceph txt unknown /proc/32130/task/87607/exe (readlink: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87607 telemetry ceph NOFD /proc/32130/task/87607/fd (opendir: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87608 m-fin-tel ceph cwd unknown /proc/32130/task/87608/cwd (readlink: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87608 m-fin-tel ceph rtd unknown /proc/32130/task/87608/root (readlink: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87608 m-fin-tel ceph txt unknown /proc/32130/task/87608/exe (readlink: Permission denied) 2024-03-11T20:01:35.984 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87608 m-fin-tel ceph NOFD /proc/32130/task/87608/fd (opendir: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87609 mgr-fin ceph cwd unknown /proc/32130/task/87609/cwd (readlink: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87609 mgr-fin ceph rtd unknown /proc/32130/task/87609/root (readlink: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87609 mgr-fin ceph txt unknown /proc/32130/task/87609/exe (readlink: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87609 mgr-fin ceph NOFD /proc/32130/task/87609/fd (opendir: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87613 io_contex ceph cwd unknown /proc/32130/task/87613/cwd (readlink: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87613 io_contex ceph rtd unknown /proc/32130/task/87613/root (readlink: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87613 io_contex ceph txt unknown /proc/32130/task/87613/exe (readlink: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87613 io_contex ceph NOFD /proc/32130/task/87613/fd (opendir: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87614 io_contex ceph cwd unknown /proc/32130/task/87614/cwd (readlink: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87614 io_contex ceph rtd unknown /proc/32130/task/87614/root (readlink: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87614 io_contex ceph txt unknown /proc/32130/task/87614/exe (readlink: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87614 io_contex ceph NOFD /proc/32130/task/87614/fd (opendir: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87615 ceph_time ceph cwd unknown /proc/32130/task/87615/cwd (readlink: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87615 ceph_time ceph rtd unknown /proc/32130/task/87615/root (readlink: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87615 ceph_time ceph txt unknown /proc/32130/task/87615/exe (readlink: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87615 ceph_time ceph NOFD /proc/32130/task/87615/fd (opendir: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87616 ms_dispat ceph cwd unknown /proc/32130/task/87616/cwd (readlink: Permission denied) 2024-03-11T20:01:35.985 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87616 ms_dispat ceph rtd unknown /proc/32130/task/87616/root (readlink: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87616 ms_dispat ceph txt unknown /proc/32130/task/87616/exe (readlink: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87616 ms_dispat ceph NOFD /proc/32130/task/87616/fd (opendir: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87617 ms_local ceph cwd unknown /proc/32130/task/87617/cwd (readlink: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87617 ms_local ceph rtd unknown /proc/32130/task/87617/root (readlink: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87617 ms_local ceph txt unknown /proc/32130/task/87617/exe (readlink: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87617 ms_local ceph NOFD /proc/32130/task/87617/fd (opendir: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87618 safe_time ceph cwd unknown /proc/32130/task/87618/cwd (readlink: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87618 safe_time ceph rtd unknown /proc/32130/task/87618/root (readlink: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87618 safe_time ceph txt unknown /proc/32130/task/87618/exe (readlink: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87618 safe_time ceph NOFD /proc/32130/task/87618/fd (opendir: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87619 safe_time ceph cwd unknown /proc/32130/task/87619/cwd (readlink: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87619 safe_time ceph rtd unknown /proc/32130/task/87619/root (readlink: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87619 safe_time ceph txt unknown /proc/32130/task/87619/exe (readlink: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87619 safe_time ceph NOFD /proc/32130/task/87619/fd (opendir: Permission denied) 2024-03-11T20:01:35.986 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87620 mgr-fin ceph cwd unknown /proc/32130/task/87620/cwd (readlink: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87620 mgr-fin ceph rtd unknown /proc/32130/task/87620/root (readlink: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87620 mgr-fin ceph txt unknown /proc/32130/task/87620/exe (readlink: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87620 mgr-fin ceph NOFD /proc/32130/task/87620/fd (opendir: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87621 mgr-fin ceph cwd unknown /proc/32130/task/87621/cwd (readlink: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87621 mgr-fin ceph rtd unknown /proc/32130/task/87621/root (readlink: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87621 mgr-fin ceph txt unknown /proc/32130/task/87621/exe (readlink: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87621 mgr-fin ceph NOFD /proc/32130/task/87621/fd (opendir: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87622 mgr-fin ceph cwd unknown /proc/32130/task/87622/cwd (readlink: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87622 mgr-fin ceph rtd unknown /proc/32130/task/87622/root (readlink: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87622 mgr-fin ceph txt unknown /proc/32130/task/87622/exe (readlink: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87622 mgr-fin ceph NOFD /proc/32130/task/87622/fd (opendir: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87623 mgr-fin ceph cwd unknown /proc/32130/task/87623/cwd (readlink: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87623 mgr-fin ceph rtd unknown /proc/32130/task/87623/root (readlink: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87623 mgr-fin ceph txt unknown /proc/32130/task/87623/exe (readlink: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87623 mgr-fin ceph NOFD /proc/32130/task/87623/fd (opendir: Permission denied) 2024-03-11T20:01:35.987 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87624 mgr-fin ceph cwd unknown /proc/32130/task/87624/cwd (readlink: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87624 mgr-fin ceph rtd unknown /proc/32130/task/87624/root (readlink: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87624 mgr-fin ceph txt unknown /proc/32130/task/87624/exe (readlink: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87624 mgr-fin ceph NOFD /proc/32130/task/87624/fd (opendir: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87625 mgr-fin ceph cwd unknown /proc/32130/task/87625/cwd (readlink: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87625 mgr-fin ceph rtd unknown /proc/32130/task/87625/root (readlink: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87625 mgr-fin ceph txt unknown /proc/32130/task/87625/exe (readlink: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87625 mgr-fin ceph NOFD /proc/32130/task/87625/fd (opendir: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87626 mgr-fin ceph cwd unknown /proc/32130/task/87626/cwd (readlink: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87626 mgr-fin ceph rtd unknown /proc/32130/task/87626/root (readlink: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87626 mgr-fin ceph txt unknown /proc/32130/task/87626/exe (readlink: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87626 mgr-fin ceph NOFD /proc/32130/task/87626/fd (opendir: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87628 mgr-fin ceph cwd unknown /proc/32130/task/87628/cwd (readlink: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87628 mgr-fin ceph rtd unknown /proc/32130/task/87628/root (readlink: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87628 mgr-fin ceph txt unknown /proc/32130/task/87628/exe (readlink: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87628 mgr-fin ceph NOFD /proc/32130/task/87628/fd (opendir: Permission denied) 2024-03-11T20:01:35.988 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87629 mgr-fin ceph cwd unknown /proc/32130/task/87629/cwd (readlink: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87629 mgr-fin ceph rtd unknown /proc/32130/task/87629/root (readlink: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87629 mgr-fin ceph txt unknown /proc/32130/task/87629/exe (readlink: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87629 mgr-fin ceph NOFD /proc/32130/task/87629/fd (opendir: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87630 mgr-fin ceph cwd unknown /proc/32130/task/87630/cwd (readlink: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87630 mgr-fin ceph rtd unknown /proc/32130/task/87630/root (readlink: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87630 mgr-fin ceph txt unknown /proc/32130/task/87630/exe (readlink: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87630 mgr-fin ceph NOFD /proc/32130/task/87630/fd (opendir: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87631 mgr-fin ceph cwd unknown /proc/32130/task/87631/cwd (readlink: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87631 mgr-fin ceph rtd unknown /proc/32130/task/87631/root (readlink: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87631 mgr-fin ceph txt unknown /proc/32130/task/87631/exe (readlink: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87631 mgr-fin ceph NOFD /proc/32130/task/87631/fd (opendir: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87632 mgr-fin ceph cwd unknown /proc/32130/task/87632/cwd (readlink: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87632 mgr-fin ceph rtd unknown /proc/32130/task/87632/root (readlink: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87632 mgr-fin ceph txt unknown /proc/32130/task/87632/exe (readlink: Permission denied) 2024-03-11T20:01:35.989 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87632 mgr-fin ceph NOFD /proc/32130/task/87632/fd (opendir: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87634 m-fin-vol ceph cwd unknown /proc/32130/task/87634/cwd (readlink: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87634 m-fin-vol ceph rtd unknown /proc/32130/task/87634/root (readlink: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87634 m-fin-vol ceph txt unknown /proc/32130/task/87634/exe (readlink: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87634 m-fin-vol ceph NOFD /proc/32130/task/87634/fd (opendir: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87635 io_contex ceph cwd unknown /proc/32130/task/87635/cwd (readlink: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87635 io_contex ceph rtd unknown /proc/32130/task/87635/root (readlink: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87635 io_contex ceph txt unknown /proc/32130/task/87635/exe (readlink: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87635 io_contex ceph NOFD /proc/32130/task/87635/fd (opendir: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87636 io_contex ceph cwd unknown /proc/32130/task/87636/cwd (readlink: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87636 io_contex ceph rtd unknown /proc/32130/task/87636/root (readlink: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87636 io_contex ceph txt unknown /proc/32130/task/87636/exe (readlink: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87636 io_contex ceph NOFD /proc/32130/task/87636/fd (opendir: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87669 dashboard ceph cwd unknown /proc/32130/task/87669/cwd (readlink: Permission denied) 2024-03-11T20:01:35.990 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87669 dashboard ceph rtd unknown /proc/32130/task/87669/root (readlink: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87669 dashboard ceph txt unknown /proc/32130/task/87669/exe (readlink: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87669 dashboard ceph NOFD /proc/32130/task/87669/fd (opendir: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87670 dashboard ceph cwd unknown /proc/32130/task/87670/cwd (readlink: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87670 dashboard ceph rtd unknown /proc/32130/task/87670/root (readlink: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87670 dashboard ceph txt unknown /proc/32130/task/87670/exe (readlink: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87670 dashboard ceph NOFD /proc/32130/task/87670/fd (opendir: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87671 dashboard ceph cwd unknown /proc/32130/task/87671/cwd (readlink: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87671 dashboard ceph rtd unknown /proc/32130/task/87671/root (readlink: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87671 dashboard ceph txt unknown /proc/32130/task/87671/exe (readlink: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87671 dashboard ceph NOFD /proc/32130/task/87671/fd (opendir: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87672 dashboard ceph cwd unknown /proc/32130/task/87672/cwd (readlink: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87672 dashboard ceph rtd unknown /proc/32130/task/87672/root (readlink: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87672 dashboard ceph txt unknown /proc/32130/task/87672/exe (readlink: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87672 dashboard ceph NOFD /proc/32130/task/87672/fd (opendir: Permission denied) 2024-03-11T20:01:35.991 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87673 dashboard ceph cwd unknown /proc/32130/task/87673/cwd (readlink: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87673 dashboard ceph rtd unknown /proc/32130/task/87673/root (readlink: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87673 dashboard ceph txt unknown /proc/32130/task/87673/exe (readlink: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87673 dashboard ceph NOFD /proc/32130/task/87673/fd (opendir: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87675 dashboard ceph cwd unknown /proc/32130/task/87675/cwd (readlink: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87675 dashboard ceph rtd unknown /proc/32130/task/87675/root (readlink: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87675 dashboard ceph txt unknown /proc/32130/task/87675/exe (readlink: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87675 dashboard ceph NOFD /proc/32130/task/87675/fd (opendir: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87676 dashboard ceph cwd unknown /proc/32130/task/87676/cwd (readlink: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87676 dashboard ceph rtd unknown /proc/32130/task/87676/root (readlink: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87676 dashboard ceph txt unknown /proc/32130/task/87676/exe (readlink: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87676 dashboard ceph NOFD /proc/32130/task/87676/fd (opendir: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87677 dashboard ceph cwd unknown /proc/32130/task/87677/cwd (readlink: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87677 dashboard ceph rtd unknown /proc/32130/task/87677/root (readlink: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87677 dashboard ceph txt unknown /proc/32130/task/87677/exe (readlink: Permission denied) 2024-03-11T20:01:35.992 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87677 dashboard ceph NOFD /proc/32130/task/87677/fd (opendir: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87678 dashboard ceph cwd unknown /proc/32130/task/87678/cwd (readlink: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87678 dashboard ceph rtd unknown /proc/32130/task/87678/root (readlink: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87678 dashboard ceph txt unknown /proc/32130/task/87678/exe (readlink: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87678 dashboard ceph NOFD /proc/32130/task/87678/fd (opendir: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87679 dashboard ceph cwd unknown /proc/32130/task/87679/cwd (readlink: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87679 dashboard ceph rtd unknown /proc/32130/task/87679/root (readlink: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87679 dashboard ceph txt unknown /proc/32130/task/87679/exe (readlink: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87679 dashboard ceph NOFD /proc/32130/task/87679/fd (opendir: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87680 dashboard ceph cwd unknown /proc/32130/task/87680/cwd (readlink: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87680 dashboard ceph rtd unknown /proc/32130/task/87680/root (readlink: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87680 dashboard ceph txt unknown /proc/32130/task/87680/exe (readlink: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87680 dashboard ceph NOFD /proc/32130/task/87680/fd (opendir: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87694 mgr-fin ceph cwd unknown /proc/32130/task/87694/cwd (readlink: Permission denied) 2024-03-11T20:01:35.993 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87694 mgr-fin ceph rtd unknown /proc/32130/task/87694/root (readlink: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87694 mgr-fin ceph txt unknown /proc/32130/task/87694/exe (readlink: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87694 mgr-fin ceph NOFD /proc/32130/task/87694/fd (opendir: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87695 mgr-fin ceph cwd unknown /proc/32130/task/87695/cwd (readlink: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87695 mgr-fin ceph rtd unknown /proc/32130/task/87695/root (readlink: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87695 mgr-fin ceph txt unknown /proc/32130/task/87695/exe (readlink: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87695 mgr-fin ceph NOFD /proc/32130/task/87695/fd (opendir: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87696 mgr-fin ceph cwd unknown /proc/32130/task/87696/cwd (readlink: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87696 mgr-fin ceph rtd unknown /proc/32130/task/87696/root (readlink: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87696 mgr-fin ceph txt unknown /proc/32130/task/87696/exe (readlink: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87696 mgr-fin ceph NOFD /proc/32130/task/87696/fd (opendir: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87697 mgr-fin ceph cwd unknown /proc/32130/task/87697/cwd (readlink: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87697 mgr-fin ceph rtd unknown /proc/32130/task/87697/root (readlink: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87697 mgr-fin ceph txt unknown /proc/32130/task/87697/exe (readlink: Permission denied) 2024-03-11T20:01:35.994 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87697 mgr-fin ceph NOFD /proc/32130/task/87697/fd (opendir: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87698 mgr-fin ceph cwd unknown /proc/32130/task/87698/cwd (readlink: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87698 mgr-fin ceph rtd unknown /proc/32130/task/87698/root (readlink: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87698 mgr-fin ceph txt unknown /proc/32130/task/87698/exe (readlink: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87698 mgr-fin ceph NOFD /proc/32130/task/87698/fd (opendir: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87699 mgr-fin ceph cwd unknown /proc/32130/task/87699/cwd (readlink: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87699 mgr-fin ceph rtd unknown /proc/32130/task/87699/root (readlink: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87699 mgr-fin ceph txt unknown /proc/32130/task/87699/exe (readlink: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87699 mgr-fin ceph NOFD /proc/32130/task/87699/fd (opendir: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87700 mgr-fin ceph cwd unknown /proc/32130/task/87700/cwd (readlink: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87700 mgr-fin ceph rtd unknown /proc/32130/task/87700/root (readlink: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87700 mgr-fin ceph txt unknown /proc/32130/task/87700/exe (readlink: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87700 mgr-fin ceph NOFD /proc/32130/task/87700/fd (opendir: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87701 mgr-fin ceph cwd unknown /proc/32130/task/87701/cwd (readlink: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87701 mgr-fin ceph rtd unknown /proc/32130/task/87701/root (readlink: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87701 mgr-fin ceph txt unknown /proc/32130/task/87701/exe (readlink: Permission denied) 2024-03-11T20:01:35.995 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87701 mgr-fin ceph NOFD /proc/32130/task/87701/fd (opendir: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87702 mgr-fin ceph cwd unknown /proc/32130/task/87702/cwd (readlink: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87702 mgr-fin ceph rtd unknown /proc/32130/task/87702/root (readlink: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87702 mgr-fin ceph txt unknown /proc/32130/task/87702/exe (readlink: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87702 mgr-fin ceph NOFD /proc/32130/task/87702/fd (opendir: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87703 mgr-fin ceph cwd unknown /proc/32130/task/87703/cwd (readlink: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87703 mgr-fin ceph rtd unknown /proc/32130/task/87703/root (readlink: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87703 mgr-fin ceph txt unknown /proc/32130/task/87703/exe (readlink: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87703 mgr-fin ceph NOFD /proc/32130/task/87703/fd (opendir: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87704 mgr-fin ceph cwd unknown /proc/32130/task/87704/cwd (readlink: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87704 mgr-fin ceph rtd unknown /proc/32130/task/87704/root (readlink: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87704 mgr-fin ceph txt unknown /proc/32130/task/87704/exe (readlink: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87704 mgr-fin ceph NOFD /proc/32130/task/87704/fd (opendir: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87735 dashboard ceph cwd unknown /proc/32130/task/87735/cwd (readlink: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87735 dashboard ceph rtd unknown /proc/32130/task/87735/root (readlink: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87735 dashboard ceph txt unknown /proc/32130/task/87735/exe (readlink: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87735 dashboard ceph NOFD /proc/32130/task/87735/fd (opendir: Permission denied) 2024-03-11T20:01:35.996 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87752 mgr-fin ceph cwd unknown /proc/32130/task/87752/cwd (readlink: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87752 mgr-fin ceph rtd unknown /proc/32130/task/87752/root (readlink: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87752 mgr-fin ceph txt unknown /proc/32130/task/87752/exe (readlink: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87752 mgr-fin ceph NOFD /proc/32130/task/87752/fd (opendir: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87753 mgr-fin ceph cwd unknown /proc/32130/task/87753/cwd (readlink: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87753 mgr-fin ceph rtd unknown /proc/32130/task/87753/root (readlink: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87753 mgr-fin ceph txt unknown /proc/32130/task/87753/exe (readlink: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87753 mgr-fin ceph NOFD /proc/32130/task/87753/fd (opendir: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87754 mgr-fin ceph cwd unknown /proc/32130/task/87754/cwd (readlink: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87754 mgr-fin ceph rtd unknown /proc/32130/task/87754/root (readlink: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87754 mgr-fin ceph txt unknown /proc/32130/task/87754/exe (readlink: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87754 mgr-fin ceph NOFD /proc/32130/task/87754/fd (opendir: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87755 mgr-fin ceph cwd unknown /proc/32130/task/87755/cwd (readlink: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87755 mgr-fin ceph rtd unknown /proc/32130/task/87755/root (readlink: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87755 mgr-fin ceph txt unknown /proc/32130/task/87755/exe (readlink: Permission denied) 2024-03-11T20:01:35.997 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87755 mgr-fin ceph NOFD /proc/32130/task/87755/fd (opendir: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87757 mgr-fin ceph cwd unknown /proc/32130/task/87757/cwd (readlink: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87757 mgr-fin ceph rtd unknown /proc/32130/task/87757/root (readlink: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87757 mgr-fin ceph txt unknown /proc/32130/task/87757/exe (readlink: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87757 mgr-fin ceph NOFD /proc/32130/task/87757/fd (opendir: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87759 mgr-fin ceph cwd unknown /proc/32130/task/87759/cwd (readlink: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87759 mgr-fin ceph rtd unknown /proc/32130/task/87759/root (readlink: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87759 mgr-fin ceph txt unknown /proc/32130/task/87759/exe (readlink: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87759 mgr-fin ceph NOFD /proc/32130/task/87759/fd (opendir: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87760 mgr-fin ceph cwd unknown /proc/32130/task/87760/cwd (readlink: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87760 mgr-fin ceph rtd unknown /proc/32130/task/87760/root (readlink: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87760 mgr-fin ceph txt unknown /proc/32130/task/87760/exe (readlink: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87760 mgr-fin ceph NOFD /proc/32130/task/87760/fd (opendir: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87761 mgr-fin ceph cwd unknown /proc/32130/task/87761/cwd (readlink: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87761 mgr-fin ceph rtd unknown /proc/32130/task/87761/root (readlink: Permission denied) 2024-03-11T20:01:35.998 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87761 mgr-fin ceph txt unknown /proc/32130/task/87761/exe (readlink: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87761 mgr-fin ceph NOFD /proc/32130/task/87761/fd (opendir: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87762 mgr-fin ceph cwd unknown /proc/32130/task/87762/cwd (readlink: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87762 mgr-fin ceph rtd unknown /proc/32130/task/87762/root (readlink: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87762 mgr-fin ceph txt unknown /proc/32130/task/87762/exe (readlink: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87762 mgr-fin ceph NOFD /proc/32130/task/87762/fd (opendir: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87764 mgr-fin ceph cwd unknown /proc/32130/task/87764/cwd (readlink: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87764 mgr-fin ceph rtd unknown /proc/32130/task/87764/root (readlink: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87764 mgr-fin ceph txt unknown /proc/32130/task/87764/exe (readlink: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87764 mgr-fin ceph NOFD /proc/32130/task/87764/fd (opendir: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87765 mgr-fin ceph cwd unknown /proc/32130/task/87765/cwd (readlink: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87765 mgr-fin ceph rtd unknown /proc/32130/task/87765/root (readlink: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87765 mgr-fin ceph txt unknown /proc/32130/task/87765/exe (readlink: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87765 mgr-fin ceph NOFD /proc/32130/task/87765/fd (opendir: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87955 devicehea ceph cwd unknown /proc/32130/task/87955/cwd (readlink: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87955 devicehea ceph rtd unknown /proc/32130/task/87955/root (readlink: Permission denied) 2024-03-11T20:01:35.999 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87955 devicehea ceph txt unknown /proc/32130/task/87955/exe (readlink: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87955 devicehea ceph NOFD /proc/32130/task/87955/fd (opendir: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87960 io_contex ceph cwd unknown /proc/32130/task/87960/cwd (readlink: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87960 io_contex ceph rtd unknown /proc/32130/task/87960/root (readlink: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87960 io_contex ceph txt unknown /proc/32130/task/87960/exe (readlink: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87960 io_contex ceph NOFD /proc/32130/task/87960/fd (opendir: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87961 io_contex ceph cwd unknown /proc/32130/task/87961/cwd (readlink: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87961 io_contex ceph rtd unknown /proc/32130/task/87961/root (readlink: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87961 io_contex ceph txt unknown /proc/32130/task/87961/exe (readlink: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87961 io_contex ceph NOFD /proc/32130/task/87961/fd (opendir: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87962 ceph_time ceph cwd unknown /proc/32130/task/87962/cwd (readlink: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87962 ceph_time ceph rtd unknown /proc/32130/task/87962/root (readlink: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87962 ceph_time ceph txt unknown /proc/32130/task/87962/exe (readlink: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87962 ceph_time ceph NOFD /proc/32130/task/87962/fd (opendir: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87963 ms_dispat ceph cwd unknown /proc/32130/task/87963/cwd (readlink: Permission denied) 2024-03-11T20:01:36.000 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87963 ms_dispat ceph rtd unknown /proc/32130/task/87963/root (readlink: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87963 ms_dispat ceph txt unknown /proc/32130/task/87963/exe (readlink: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87963 ms_dispat ceph NOFD /proc/32130/task/87963/fd (opendir: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87964 ms_local ceph cwd unknown /proc/32130/task/87964/cwd (readlink: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87964 ms_local ceph rtd unknown /proc/32130/task/87964/root (readlink: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87964 ms_local ceph txt unknown /proc/32130/task/87964/exe (readlink: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87964 ms_local ceph NOFD /proc/32130/task/87964/fd (opendir: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87965 safe_time ceph cwd unknown /proc/32130/task/87965/cwd (readlink: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87965 safe_time ceph rtd unknown /proc/32130/task/87965/root (readlink: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87965 safe_time ceph txt unknown /proc/32130/task/87965/exe (readlink: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87965 safe_time ceph NOFD /proc/32130/task/87965/fd (opendir: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87966 safe_time ceph cwd unknown /proc/32130/task/87966/cwd (readlink: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87966 safe_time ceph rtd unknown /proc/32130/task/87966/root (readlink: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87966 safe_time ceph txt unknown /proc/32130/task/87966/exe (readlink: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 87966 safe_time ceph NOFD /proc/32130/task/87966/fd (opendir: Permission denied) 2024-03-11T20:01:36.001 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89503 ceph_time ceph cwd unknown /proc/32130/task/89503/cwd (readlink: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89503 ceph_time ceph rtd unknown /proc/32130/task/89503/root (readlink: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89503 ceph_time ceph txt unknown /proc/32130/task/89503/exe (readlink: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89503 ceph_time ceph NOFD /proc/32130/task/89503/fd (opendir: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89504 fn_anonym ceph cwd unknown /proc/32130/task/89504/cwd (readlink: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89504 fn_anonym ceph rtd unknown /proc/32130/task/89504/root (readlink: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89504 fn_anonym ceph txt unknown /proc/32130/task/89504/exe (readlink: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89504 fn_anonym ceph NOFD /proc/32130/task/89504/fd (opendir: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89505 safe_time ceph cwd unknown /proc/32130/task/89505/cwd (readlink: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89505 safe_time ceph rtd unknown /proc/32130/task/89505/root (readlink: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89505 safe_time ceph txt unknown /proc/32130/task/89505/exe (readlink: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89505 safe_time ceph NOFD /proc/32130/task/89505/fd (opendir: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89506 fn_anonym ceph cwd unknown /proc/32130/task/89506/cwd (readlink: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89506 fn_anonym ceph rtd unknown /proc/32130/task/89506/root (readlink: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89506 fn_anonym ceph txt unknown /proc/32130/task/89506/exe (readlink: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89506 fn_anonym ceph NOFD /proc/32130/task/89506/fd (opendir: Permission denied) 2024-03-11T20:01:36.002 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89507 flusher ceph cwd unknown /proc/32130/task/89507/cwd (readlink: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89507 flusher ceph rtd unknown /proc/32130/task/89507/root (readlink: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89507 flusher ceph txt unknown /proc/32130/task/89507/exe (readlink: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89507 flusher ceph NOFD /proc/32130/task/89507/fd (opendir: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89508 ms_dispat ceph cwd unknown /proc/32130/task/89508/cwd (readlink: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89508 ms_dispat ceph rtd unknown /proc/32130/task/89508/root (readlink: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89508 ms_dispat ceph txt unknown /proc/32130/task/89508/exe (readlink: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89508 ms_dispat ceph NOFD /proc/32130/task/89508/fd (opendir: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89509 ms_local ceph cwd unknown /proc/32130/task/89509/cwd (readlink: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89509 ms_local ceph rtd unknown /proc/32130/task/89509/root (readlink: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89509 ms_local ceph txt unknown /proc/32130/task/89509/exe (readlink: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89509 ms_local ceph NOFD /proc/32130/task/89509/fd (opendir: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89510 safe_time ceph cwd unknown /proc/32130/task/89510/cwd (readlink: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89510 safe_time ceph rtd unknown /proc/32130/task/89510/root (readlink: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89510 safe_time ceph txt unknown /proc/32130/task/89510/exe (readlink: Permission denied) 2024-03-11T20:01:36.003 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89510 safe_time ceph NOFD /proc/32130/task/89510/fd (opendir: Permission denied) 2024-03-11T20:01:36.004 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89511 m-fin-sna ceph cwd unknown /proc/32130/task/89511/cwd (readlink: Permission denied) 2024-03-11T20:01:36.004 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89511 m-fin-sna ceph rtd unknown /proc/32130/task/89511/root (readlink: Permission denied) 2024-03-11T20:01:36.004 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89511 m-fin-sna ceph txt unknown /proc/32130/task/89511/exe (readlink: Permission denied) 2024-03-11T20:01:36.004 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89511 m-fin-sna ceph NOFD /proc/32130/task/89511/fd (opendir: Permission denied) 2024-03-11T20:01:36.004 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89973 m-fin-sna ceph cwd unknown /proc/32130/task/89973/cwd (readlink: Permission denied) 2024-03-11T20:01:36.004 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89973 m-fin-sna ceph rtd unknown /proc/32130/task/89973/root (readlink: Permission denied) 2024-03-11T20:01:36.004 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89973 m-fin-sna ceph txt unknown /proc/32130/task/89973/exe (readlink: Permission denied) 2024-03-11T20:01:36.004 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mgr 32130 89973 m-fin-sna ceph NOFD /proc/32130/task/89973/fd (opendir: Permission denied) 2024-03-11T20:01:36.004 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 33269 root cwd unknown /proc/33269/cwd (readlink: Permission denied) 2024-03-11T20:01:36.004 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 33269 root rtd unknown /proc/33269/root (readlink: Permission denied) 2024-03-11T20:01:36.004 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 33269 root txt unknown /proc/33269/exe (readlink: Permission denied) 2024-03-11T20:01:36.004 INFO:tasks.workunit.client.0.smithi097.stdout:systemd 33269 root NOFD /proc/33269/fd (opendir: Permission denied) 2024-03-11T20:01:36.004 INFO:tasks.workunit.client.0.smithi097.stdout:(sd-pam) 33271 root cwd unknown /proc/33271/cwd (readlink: Permission denied) 2024-03-11T20:01:36.009 INFO:tasks.workunit.client.0.smithi097.stdout:(sd-pam) 33271 root rtd unknown /proc/33271/root (readlink: Permission denied) 2024-03-11T20:01:36.009 INFO:tasks.workunit.client.0.smithi097.stdout:(sd-pam) 33271 root txt unknown /proc/33271/exe (readlink: Permission denied) 2024-03-11T20:01:36.009 INFO:tasks.workunit.client.0.smithi097.stdout:(sd-pam) 33271 root NOFD /proc/33271/fd (opendir: Permission denied) 2024-03-11T20:01:36.009 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 44552 root cwd unknown /proc/44552/cwd (readlink: Permission denied) 2024-03-11T20:01:36.009 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 44552 root rtd unknown /proc/44552/root (readlink: Permission denied) 2024-03-11T20:01:36.009 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 44552 root txt unknown /proc/44552/exe (readlink: Permission denied) 2024-03-11T20:01:36.009 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 44552 root NOFD /proc/44552/fd (opendir: Permission denied) 2024-03-11T20:01:36.009 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 44829 root cwd unknown /proc/44829/cwd (readlink: Permission denied) 2024-03-11T20:01:36.009 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 44829 root rtd unknown /proc/44829/root (readlink: Permission denied) 2024-03-11T20:01:36.009 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 44829 root txt unknown /proc/44829/exe (readlink: Permission denied) 2024-03-11T20:01:36.009 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 44829 root NOFD /proc/44829/fd (opendir: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 46333 root cwd unknown /proc/46333/cwd (readlink: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 46333 root rtd unknown /proc/46333/root (readlink: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 46333 root txt unknown /proc/46333/exe (readlink: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 46333 root NOFD /proc/46333/fd (opendir: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 47090 root cwd unknown /proc/47090/cwd (readlink: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 47090 root rtd unknown /proc/47090/root (readlink: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 47090 root txt unknown /proc/47090/exe (readlink: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 47090 root NOFD /proc/47090/fd (opendir: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 47092 root cwd unknown /proc/47092/cwd (readlink: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 47092 root rtd unknown /proc/47092/root (readlink: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 47092 root txt unknown /proc/47092/exe (readlink: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 47092 root NOFD /proc/47092/fd (opendir: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 ceph cwd unknown /proc/47094/cwd (readlink: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 ceph rtd unknown /proc/47094/root (readlink: Permission denied) 2024-03-11T20:01:36.010 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 ceph txt unknown /proc/47094/exe (readlink: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 ceph NOFD /proc/47094/fd (opendir: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47095 log ceph cwd unknown /proc/47094/task/47095/cwd (readlink: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47095 log ceph rtd unknown /proc/47094/task/47095/root (readlink: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47095 log ceph txt unknown /proc/47094/task/47095/exe (readlink: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47095 log ceph NOFD /proc/47094/task/47095/fd (opendir: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47097 msgr-work ceph cwd unknown /proc/47094/task/47097/cwd (readlink: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47097 msgr-work ceph rtd unknown /proc/47094/task/47097/root (readlink: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47097 msgr-work ceph txt unknown /proc/47094/task/47097/exe (readlink: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47097 msgr-work ceph NOFD /proc/47094/task/47097/fd (opendir: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47098 msgr-work ceph cwd unknown /proc/47094/task/47098/cwd (readlink: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47098 msgr-work ceph rtd unknown /proc/47094/task/47098/root (readlink: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47098 msgr-work ceph txt unknown /proc/47094/task/47098/exe (readlink: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47098 msgr-work ceph NOFD /proc/47094/task/47098/fd (opendir: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47099 msgr-work ceph cwd unknown /proc/47094/task/47099/cwd (readlink: Permission denied) 2024-03-11T20:01:36.011 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47099 msgr-work ceph rtd unknown /proc/47094/task/47099/root (readlink: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47099 msgr-work ceph txt unknown /proc/47094/task/47099/exe (readlink: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47099 msgr-work ceph NOFD /proc/47094/task/47099/fd (opendir: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47103 service ceph cwd unknown /proc/47094/task/47103/cwd (readlink: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47103 service ceph rtd unknown /proc/47094/task/47103/root (readlink: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47103 service ceph txt unknown /proc/47094/task/47103/exe (readlink: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47103 service ceph NOFD /proc/47094/task/47103/fd (opendir: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47104 admin_soc ceph cwd unknown /proc/47094/task/47104/cwd (readlink: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47104 admin_soc ceph rtd unknown /proc/47094/task/47104/root (readlink: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47104 admin_soc ceph txt unknown /proc/47094/task/47104/exe (readlink: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47104 admin_soc ceph NOFD /proc/47094/task/47104/fd (opendir: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47247 signal_ha ceph cwd unknown /proc/47094/task/47247/cwd (readlink: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47247 signal_ha ceph rtd unknown /proc/47094/task/47247/root (readlink: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47247 signal_ha ceph txt unknown /proc/47094/task/47247/exe (readlink: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47247 signal_ha ceph NOFD /proc/47094/task/47247/fd (opendir: Permission denied) 2024-03-11T20:01:36.012 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47248 io_contex ceph cwd unknown /proc/47094/task/47248/cwd (readlink: Permission denied) 2024-03-11T20:01:36.013 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47248 io_contex ceph rtd unknown /proc/47094/task/47248/root (readlink: Permission denied) 2024-03-11T20:01:36.013 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47248 io_contex ceph txt unknown /proc/47094/task/47248/exe (readlink: Permission denied) 2024-03-11T20:01:36.013 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47248 io_contex ceph NOFD /proc/47094/task/47248/fd (opendir: Permission denied) 2024-03-11T20:01:36.013 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47249 io_contex ceph cwd unknown /proc/47094/task/47249/cwd (readlink: Permission denied) 2024-03-11T20:01:36.013 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47249 io_contex ceph rtd unknown /proc/47094/task/47249/root (readlink: Permission denied) 2024-03-11T20:01:36.013 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47249 io_contex ceph txt unknown /proc/47094/task/47249/exe (readlink: Permission denied) 2024-03-11T20:01:36.013 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47249 io_contex ceph NOFD /proc/47094/task/47249/fd (opendir: Permission denied) 2024-03-11T20:01:36.013 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47256 OpHistory ceph cwd unknown /proc/47094/task/47256/cwd (readlink: Permission denied) 2024-03-11T20:01:36.013 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47256 OpHistory ceph rtd unknown /proc/47094/task/47256/root (readlink: Permission denied) 2024-03-11T20:01:36.013 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47256 OpHistory ceph txt unknown /proc/47094/task/47256/exe (readlink: Permission denied) 2024-03-11T20:01:36.013 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47256 OpHistory ceph NOFD /proc/47094/task/47256/fd (opendir: Permission denied) 2024-03-11T20:01:36.013 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47257 ceph_time ceph cwd unknown /proc/47094/task/47257/cwd (readlink: Permission denied) 2024-03-11T20:01:36.013 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47257 ceph_time ceph rtd unknown /proc/47094/task/47257/root (readlink: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47257 ceph_time ceph txt unknown /proc/47094/task/47257/exe (readlink: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47257 ceph_time ceph NOFD /proc/47094/task/47257/fd (opendir: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47361 ceph-osd ceph cwd unknown /proc/47094/task/47361/cwd (readlink: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47361 ceph-osd ceph rtd unknown /proc/47094/task/47361/root (readlink: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47361 ceph-osd ceph txt unknown /proc/47094/task/47361/exe (readlink: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47361 ceph-osd ceph NOFD /proc/47094/task/47361/fd (opendir: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47383 ceph-osd ceph cwd unknown /proc/47094/task/47383/cwd (readlink: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47383 ceph-osd ceph rtd unknown /proc/47094/task/47383/root (readlink: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47383 ceph-osd ceph txt unknown /proc/47094/task/47383/exe (readlink: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47383 ceph-osd ceph NOFD /proc/47094/task/47383/fd (opendir: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47400 ceph-osd ceph cwd unknown /proc/47094/task/47400/cwd (readlink: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47400 ceph-osd ceph rtd unknown /proc/47094/task/47400/root (readlink: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47400 ceph-osd ceph txt unknown /proc/47094/task/47400/exe (readlink: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47400 ceph-osd ceph NOFD /proc/47094/task/47400/fd (opendir: Permission denied) 2024-03-11T20:01:36.014 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47422 ceph-osd ceph cwd unknown /proc/47094/task/47422/cwd (readlink: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47422 ceph-osd ceph rtd unknown /proc/47094/task/47422/root (readlink: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47422 ceph-osd ceph txt unknown /proc/47094/task/47422/exe (readlink: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47422 ceph-osd ceph NOFD /proc/47094/task/47422/fd (opendir: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47428 ceph-osd ceph cwd unknown /proc/47094/task/47428/cwd (readlink: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47428 ceph-osd ceph rtd unknown /proc/47094/task/47428/root (readlink: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47428 ceph-osd ceph txt unknown /proc/47094/task/47428/exe (readlink: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47428 ceph-osd ceph NOFD /proc/47094/task/47428/fd (opendir: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47438 ceph-osd ceph cwd unknown /proc/47094/task/47438/cwd (readlink: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47438 ceph-osd ceph rtd unknown /proc/47094/task/47438/root (readlink: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47438 ceph-osd ceph txt unknown /proc/47094/task/47438/exe (readlink: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47438 ceph-osd ceph NOFD /proc/47094/task/47438/fd (opendir: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47476 ceph-osd ceph cwd unknown /proc/47094/task/47476/cwd (readlink: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47476 ceph-osd ceph rtd unknown /proc/47094/task/47476/root (readlink: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47476 ceph-osd ceph txt unknown /proc/47094/task/47476/exe (readlink: Permission denied) 2024-03-11T20:01:36.015 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47476 ceph-osd ceph NOFD /proc/47094/task/47476/fd (opendir: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47481 ceph-osd ceph cwd unknown /proc/47094/task/47481/cwd (readlink: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47481 ceph-osd ceph rtd unknown /proc/47094/task/47481/root (readlink: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47481 ceph-osd ceph txt unknown /proc/47094/task/47481/exe (readlink: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47481 ceph-osd ceph NOFD /proc/47094/task/47481/fd (opendir: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47482 ceph-osd ceph cwd unknown /proc/47094/task/47482/cwd (readlink: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47482 ceph-osd ceph rtd unknown /proc/47094/task/47482/root (readlink: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47482 ceph-osd ceph txt unknown /proc/47094/task/47482/exe (readlink: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47482 ceph-osd ceph NOFD /proc/47094/task/47482/fd (opendir: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47483 safe_time ceph cwd unknown /proc/47094/task/47483/cwd (readlink: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47483 safe_time ceph rtd unknown /proc/47094/task/47483/root (readlink: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47483 safe_time ceph txt unknown /proc/47094/task/47483/exe (readlink: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47483 safe_time ceph NOFD /proc/47094/task/47483/fd (opendir: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47484 safe_time ceph cwd unknown /proc/47094/task/47484/cwd (readlink: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47484 safe_time ceph rtd unknown /proc/47094/task/47484/root (readlink: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47484 safe_time ceph txt unknown /proc/47094/task/47484/exe (readlink: Permission denied) 2024-03-11T20:01:36.016 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47484 safe_time ceph NOFD /proc/47094/task/47484/fd (opendir: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47485 safe_time ceph cwd unknown /proc/47094/task/47485/cwd (readlink: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47485 safe_time ceph rtd unknown /proc/47094/task/47485/root (readlink: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47485 safe_time ceph txt unknown /proc/47094/task/47485/exe (readlink: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47485 safe_time ceph NOFD /proc/47094/task/47485/fd (opendir: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47486 safe_time ceph cwd unknown /proc/47094/task/47486/cwd (readlink: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47486 safe_time ceph rtd unknown /proc/47094/task/47486/root (readlink: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47486 safe_time ceph txt unknown /proc/47094/task/47486/exe (readlink: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47486 safe_time ceph NOFD /proc/47094/task/47486/fd (opendir: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47487 fn_anonym ceph cwd unknown /proc/47094/task/47487/cwd (readlink: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47487 fn_anonym ceph rtd unknown /proc/47094/task/47487/root (readlink: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47487 fn_anonym ceph txt unknown /proc/47094/task/47487/exe (readlink: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47487 fn_anonym ceph NOFD /proc/47094/task/47487/fd (opendir: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47491 rocksdb:l ceph cwd unknown /proc/47094/task/47491/cwd (readlink: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47491 rocksdb:l ceph rtd unknown /proc/47094/task/47491/root (readlink: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47491 rocksdb:l ceph txt unknown /proc/47094/task/47491/exe (readlink: Permission denied) 2024-03-11T20:01:36.017 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47491 rocksdb:l ceph NOFD /proc/47094/task/47491/fd (opendir: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47492 rocksdb:l ceph cwd unknown /proc/47094/task/47492/cwd (readlink: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47492 rocksdb:l ceph rtd unknown /proc/47094/task/47492/root (readlink: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47492 rocksdb:l ceph txt unknown /proc/47094/task/47492/exe (readlink: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47492 rocksdb:l ceph NOFD /proc/47094/task/47492/fd (opendir: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47493 rocksdb:l ceph cwd unknown /proc/47094/task/47493/cwd (readlink: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47493 rocksdb:l ceph rtd unknown /proc/47094/task/47493/root (readlink: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47493 rocksdb:l ceph txt unknown /proc/47094/task/47493/exe (readlink: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47493 rocksdb:l ceph NOFD /proc/47094/task/47493/fd (opendir: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47494 rocksdb:h ceph cwd unknown /proc/47094/task/47494/cwd (readlink: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47494 rocksdb:h ceph rtd unknown /proc/47094/task/47494/root (readlink: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47494 rocksdb:h ceph txt unknown /proc/47094/task/47494/exe (readlink: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47494 rocksdb:h ceph NOFD /proc/47094/task/47494/fd (opendir: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47910 bstore_ai ceph cwd unknown /proc/47094/task/47910/cwd (readlink: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47910 bstore_ai ceph rtd unknown /proc/47094/task/47910/root (readlink: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47910 bstore_ai ceph txt unknown /proc/47094/task/47910/exe (readlink: Permission denied) 2024-03-11T20:01:36.018 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 47910 bstore_ai ceph NOFD /proc/47094/task/47910/fd (opendir: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48140 bstore_ai ceph cwd unknown /proc/47094/task/48140/cwd (readlink: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48140 bstore_ai ceph rtd unknown /proc/47094/task/48140/root (readlink: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48140 bstore_ai ceph txt unknown /proc/47094/task/48140/exe (readlink: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48140 bstore_ai ceph NOFD /proc/47094/task/48140/fd (opendir: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48325 ceph-osd ceph cwd unknown /proc/47094/task/48325/cwd (readlink: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48325 ceph-osd ceph rtd unknown /proc/47094/task/48325/root (readlink: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48325 ceph-osd ceph txt unknown /proc/47094/task/48325/exe (readlink: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48325 ceph-osd ceph NOFD /proc/47094/task/48325/fd (opendir: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48326 cfin ceph cwd unknown /proc/47094/task/48326/cwd (readlink: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48326 cfin ceph rtd unknown /proc/47094/task/48326/root (readlink: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48326 cfin ceph txt unknown /proc/47094/task/48326/exe (readlink: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48326 cfin ceph NOFD /proc/47094/task/48326/fd (opendir: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48327 bstore_kv ceph cwd unknown /proc/47094/task/48327/cwd (readlink: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48327 bstore_kv ceph rtd unknown /proc/47094/task/48327/root (readlink: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48327 bstore_kv ceph txt unknown /proc/47094/task/48327/exe (readlink: Permission denied) 2024-03-11T20:01:36.019 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48327 bstore_kv ceph NOFD /proc/47094/task/48327/fd (opendir: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48328 bstore_kv ceph cwd unknown /proc/47094/task/48328/cwd (readlink: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48328 bstore_kv ceph rtd unknown /proc/47094/task/48328/root (readlink: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48328 bstore_kv ceph txt unknown /proc/47094/task/48328/exe (readlink: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48328 bstore_kv ceph NOFD /proc/47094/task/48328/fd (opendir: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48329 bstore_me ceph cwd unknown /proc/47094/task/48329/cwd (readlink: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48329 bstore_me ceph rtd unknown /proc/47094/task/48329/root (readlink: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48329 bstore_me ceph txt unknown /proc/47094/task/48329/exe (readlink: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48329 bstore_me ceph NOFD /proc/47094/task/48329/fd (opendir: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48333 ms_dispat ceph cwd unknown /proc/47094/task/48333/cwd (readlink: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48333 ms_dispat ceph rtd unknown /proc/47094/task/48333/root (readlink: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48333 ms_dispat ceph txt unknown /proc/47094/task/48333/exe (readlink: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48333 ms_dispat ceph NOFD /proc/47094/task/48333/fd (opendir: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48334 ms_local ceph cwd unknown /proc/47094/task/48334/cwd (readlink: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48334 ms_local ceph rtd unknown /proc/47094/task/48334/root (readlink: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48334 ms_local ceph txt unknown /proc/47094/task/48334/exe (readlink: Permission denied) 2024-03-11T20:01:36.020 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48334 ms_local ceph NOFD /proc/47094/task/48334/fd (opendir: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48335 safe_time ceph cwd unknown /proc/47094/task/48335/cwd (readlink: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48335 safe_time ceph rtd unknown /proc/47094/task/48335/root (readlink: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48335 safe_time ceph txt unknown /proc/47094/task/48335/exe (readlink: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48335 safe_time ceph NOFD /proc/47094/task/48335/fd (opendir: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48336 safe_time ceph cwd unknown /proc/47094/task/48336/cwd (readlink: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48336 safe_time ceph rtd unknown /proc/47094/task/48336/root (readlink: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48336 safe_time ceph txt unknown /proc/47094/task/48336/exe (readlink: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48336 safe_time ceph NOFD /proc/47094/task/48336/fd (opendir: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48337 ms_dispat ceph cwd unknown /proc/47094/task/48337/cwd (readlink: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48337 ms_dispat ceph rtd unknown /proc/47094/task/48337/root (readlink: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48337 ms_dispat ceph txt unknown /proc/47094/task/48337/exe (readlink: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48337 ms_dispat ceph NOFD /proc/47094/task/48337/fd (opendir: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48338 ms_local ceph cwd unknown /proc/47094/task/48338/cwd (readlink: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48338 ms_local ceph rtd unknown /proc/47094/task/48338/root (readlink: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48338 ms_local ceph txt unknown /proc/47094/task/48338/exe (readlink: Permission denied) 2024-03-11T20:01:36.021 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48338 ms_local ceph NOFD /proc/47094/task/48338/fd (opendir: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48339 ms_dispat ceph cwd unknown /proc/47094/task/48339/cwd (readlink: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48339 ms_dispat ceph rtd unknown /proc/47094/task/48339/root (readlink: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48339 ms_dispat ceph txt unknown /proc/47094/task/48339/exe (readlink: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48339 ms_dispat ceph NOFD /proc/47094/task/48339/fd (opendir: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48340 ms_local ceph cwd unknown /proc/47094/task/48340/cwd (readlink: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48340 ms_local ceph rtd unknown /proc/47094/task/48340/root (readlink: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48340 ms_local ceph txt unknown /proc/47094/task/48340/exe (readlink: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48340 ms_local ceph NOFD /proc/47094/task/48340/fd (opendir: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48341 ms_dispat ceph cwd unknown /proc/47094/task/48341/cwd (readlink: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48341 ms_dispat ceph rtd unknown /proc/47094/task/48341/root (readlink: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48341 ms_dispat ceph txt unknown /proc/47094/task/48341/exe (readlink: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48341 ms_dispat ceph NOFD /proc/47094/task/48341/fd (opendir: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48342 ms_local ceph cwd unknown /proc/47094/task/48342/cwd (readlink: Permission denied) 2024-03-11T20:01:36.022 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48342 ms_local ceph rtd unknown /proc/47094/task/48342/root (readlink: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48342 ms_local ceph txt unknown /proc/47094/task/48342/exe (readlink: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48342 ms_local ceph NOFD /proc/47094/task/48342/fd (opendir: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48343 ms_dispat ceph cwd unknown /proc/47094/task/48343/cwd (readlink: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48343 ms_dispat ceph rtd unknown /proc/47094/task/48343/root (readlink: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48343 ms_dispat ceph txt unknown /proc/47094/task/48343/exe (readlink: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48343 ms_dispat ceph NOFD /proc/47094/task/48343/fd (opendir: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48344 ms_local ceph cwd unknown /proc/47094/task/48344/cwd (readlink: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48344 ms_local ceph rtd unknown /proc/47094/task/48344/root (readlink: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48344 ms_local ceph txt unknown /proc/47094/task/48344/exe (readlink: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48344 ms_local ceph NOFD /proc/47094/task/48344/fd (opendir: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48345 ms_dispat ceph cwd unknown /proc/47094/task/48345/cwd (readlink: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48345 ms_dispat ceph rtd unknown /proc/47094/task/48345/root (readlink: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48345 ms_dispat ceph txt unknown /proc/47094/task/48345/exe (readlink: Permission denied) 2024-03-11T20:01:36.023 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48345 ms_dispat ceph NOFD /proc/47094/task/48345/fd (opendir: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48346 ms_local ceph cwd unknown /proc/47094/task/48346/cwd (readlink: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48346 ms_local ceph rtd unknown /proc/47094/task/48346/root (readlink: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48346 ms_local ceph txt unknown /proc/47094/task/48346/exe (readlink: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48346 ms_local ceph NOFD /proc/47094/task/48346/fd (opendir: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48347 ms_dispat ceph cwd unknown /proc/47094/task/48347/cwd (readlink: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48347 ms_dispat ceph rtd unknown /proc/47094/task/48347/root (readlink: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48347 ms_dispat ceph txt unknown /proc/47094/task/48347/exe (readlink: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48347 ms_dispat ceph NOFD /proc/47094/task/48347/fd (opendir: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48348 ms_local ceph cwd unknown /proc/47094/task/48348/cwd (readlink: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48348 ms_local ceph rtd unknown /proc/47094/task/48348/root (readlink: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48348 ms_local ceph txt unknown /proc/47094/task/48348/exe (readlink: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48348 ms_local ceph NOFD /proc/47094/task/48348/fd (opendir: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48349 fn_anonym ceph cwd unknown /proc/47094/task/48349/cwd (readlink: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48349 fn_anonym ceph rtd unknown /proc/47094/task/48349/root (readlink: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48349 fn_anonym ceph txt unknown /proc/47094/task/48349/exe (readlink: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48349 fn_anonym ceph NOFD /proc/47094/task/48349/fd (opendir: Permission denied) 2024-03-11T20:01:36.024 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48350 finisher ceph cwd unknown /proc/47094/task/48350/cwd (readlink: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48350 finisher ceph rtd unknown /proc/47094/task/48350/root (readlink: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48350 finisher ceph txt unknown /proc/47094/task/48350/exe (readlink: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48350 finisher ceph NOFD /proc/47094/task/48350/fd (opendir: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48351 safe_time ceph cwd unknown /proc/47094/task/48351/cwd (readlink: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48351 safe_time ceph rtd unknown /proc/47094/task/48351/root (readlink: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48351 safe_time ceph txt unknown /proc/47094/task/48351/exe (readlink: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48351 safe_time ceph NOFD /proc/47094/task/48351/fd (opendir: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48352 safe_time ceph cwd unknown /proc/47094/task/48352/cwd (readlink: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48352 safe_time ceph rtd unknown /proc/47094/task/48352/root (readlink: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48352 safe_time ceph txt unknown /proc/47094/task/48352/exe (readlink: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48352 safe_time ceph NOFD /proc/47094/task/48352/fd (opendir: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48353 ceph-osd ceph cwd unknown /proc/47094/task/48353/cwd (readlink: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48353 ceph-osd ceph rtd unknown /proc/47094/task/48353/root (readlink: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48353 ceph-osd ceph txt unknown /proc/47094/task/48353/exe (readlink: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48353 ceph-osd ceph NOFD /proc/47094/task/48353/fd (opendir: Permission denied) 2024-03-11T20:01:36.025 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48354 osd_srv_a ceph cwd unknown /proc/47094/task/48354/cwd (readlink: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48354 osd_srv_a ceph rtd unknown /proc/47094/task/48354/root (readlink: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48354 osd_srv_a ceph txt unknown /proc/47094/task/48354/exe (readlink: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48354 osd_srv_a ceph NOFD /proc/47094/task/48354/fd (opendir: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48355 tp_osd_tp ceph cwd unknown /proc/47094/task/48355/cwd (readlink: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48355 tp_osd_tp ceph rtd unknown /proc/47094/task/48355/root (readlink: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48355 tp_osd_tp ceph txt unknown /proc/47094/task/48355/exe (readlink: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48355 tp_osd_tp ceph NOFD /proc/47094/task/48355/fd (opendir: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48356 tp_osd_tp ceph cwd unknown /proc/47094/task/48356/cwd (readlink: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48356 tp_osd_tp ceph rtd unknown /proc/47094/task/48356/root (readlink: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48356 tp_osd_tp ceph txt unknown /proc/47094/task/48356/exe (readlink: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48356 tp_osd_tp ceph NOFD /proc/47094/task/48356/fd (opendir: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48357 tp_osd_tp ceph cwd unknown /proc/47094/task/48357/cwd (readlink: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48357 tp_osd_tp ceph rtd unknown /proc/47094/task/48357/root (readlink: Permission denied) 2024-03-11T20:01:36.026 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48357 tp_osd_tp ceph txt unknown /proc/47094/task/48357/exe (readlink: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48357 tp_osd_tp ceph NOFD /proc/47094/task/48357/fd (opendir: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48358 tp_osd_tp ceph cwd unknown /proc/47094/task/48358/cwd (readlink: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48358 tp_osd_tp ceph rtd unknown /proc/47094/task/48358/root (readlink: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48358 tp_osd_tp ceph txt unknown /proc/47094/task/48358/exe (readlink: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48358 tp_osd_tp ceph NOFD /proc/47094/task/48358/fd (opendir: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48359 tp_osd_tp ceph cwd unknown /proc/47094/task/48359/cwd (readlink: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48359 tp_osd_tp ceph rtd unknown /proc/47094/task/48359/root (readlink: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48359 tp_osd_tp ceph txt unknown /proc/47094/task/48359/exe (readlink: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48359 tp_osd_tp ceph NOFD /proc/47094/task/48359/fd (opendir: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48360 tp_osd_tp ceph cwd unknown /proc/47094/task/48360/cwd (readlink: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48360 tp_osd_tp ceph rtd unknown /proc/47094/task/48360/root (readlink: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48360 tp_osd_tp ceph txt unknown /proc/47094/task/48360/exe (readlink: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48360 tp_osd_tp ceph NOFD /proc/47094/task/48360/fd (opendir: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48361 tp_osd_tp ceph cwd unknown /proc/47094/task/48361/cwd (readlink: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48361 tp_osd_tp ceph rtd unknown /proc/47094/task/48361/root (readlink: Permission denied) 2024-03-11T20:01:36.027 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48361 tp_osd_tp ceph txt unknown /proc/47094/task/48361/exe (readlink: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48361 tp_osd_tp ceph NOFD /proc/47094/task/48361/fd (opendir: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48362 tp_osd_tp ceph cwd unknown /proc/47094/task/48362/cwd (readlink: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48362 tp_osd_tp ceph rtd unknown /proc/47094/task/48362/root (readlink: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48362 tp_osd_tp ceph txt unknown /proc/47094/task/48362/exe (readlink: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48362 tp_osd_tp ceph NOFD /proc/47094/task/48362/fd (opendir: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48363 tp_osd_tp ceph cwd unknown /proc/47094/task/48363/cwd (readlink: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48363 tp_osd_tp ceph rtd unknown /proc/47094/task/48363/root (readlink: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48363 tp_osd_tp ceph txt unknown /proc/47094/task/48363/exe (readlink: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48363 tp_osd_tp ceph NOFD /proc/47094/task/48363/fd (opendir: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48364 tp_osd_tp ceph cwd unknown /proc/47094/task/48364/cwd (readlink: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48364 tp_osd_tp ceph rtd unknown /proc/47094/task/48364/root (readlink: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48364 tp_osd_tp ceph txt unknown /proc/47094/task/48364/exe (readlink: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48364 tp_osd_tp ceph NOFD /proc/47094/task/48364/fd (opendir: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48365 tp_osd_tp ceph cwd unknown /proc/47094/task/48365/cwd (readlink: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48365 tp_osd_tp ceph rtd unknown /proc/47094/task/48365/root (readlink: Permission denied) 2024-03-11T20:01:36.028 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48365 tp_osd_tp ceph txt unknown /proc/47094/task/48365/exe (readlink: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48365 tp_osd_tp ceph NOFD /proc/47094/task/48365/fd (opendir: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48366 tp_osd_tp ceph cwd unknown /proc/47094/task/48366/cwd (readlink: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48366 tp_osd_tp ceph rtd unknown /proc/47094/task/48366/root (readlink: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48366 tp_osd_tp ceph txt unknown /proc/47094/task/48366/exe (readlink: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48366 tp_osd_tp ceph NOFD /proc/47094/task/48366/fd (opendir: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48367 tp_osd_tp ceph cwd unknown /proc/47094/task/48367/cwd (readlink: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48367 tp_osd_tp ceph rtd unknown /proc/47094/task/48367/root (readlink: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48367 tp_osd_tp ceph txt unknown /proc/47094/task/48367/exe (readlink: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48367 tp_osd_tp ceph NOFD /proc/47094/task/48367/fd (opendir: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48368 tp_osd_tp ceph cwd unknown /proc/47094/task/48368/cwd (readlink: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48368 tp_osd_tp ceph rtd unknown /proc/47094/task/48368/root (readlink: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48368 tp_osd_tp ceph txt unknown /proc/47094/task/48368/exe (readlink: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48368 tp_osd_tp ceph NOFD /proc/47094/task/48368/fd (opendir: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48369 tp_osd_tp ceph cwd unknown /proc/47094/task/48369/cwd (readlink: Permission denied) 2024-03-11T20:01:36.029 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48369 tp_osd_tp ceph rtd unknown /proc/47094/task/48369/root (readlink: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48369 tp_osd_tp ceph txt unknown /proc/47094/task/48369/exe (readlink: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48369 tp_osd_tp ceph NOFD /proc/47094/task/48369/fd (opendir: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48370 tp_osd_tp ceph cwd unknown /proc/47094/task/48370/cwd (readlink: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48370 tp_osd_tp ceph rtd unknown /proc/47094/task/48370/root (readlink: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48370 tp_osd_tp ceph txt unknown /proc/47094/task/48370/exe (readlink: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48370 tp_osd_tp ceph NOFD /proc/47094/task/48370/fd (opendir: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48371 osd_srv_h ceph cwd unknown /proc/47094/task/48371/cwd (readlink: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48371 osd_srv_h ceph rtd unknown /proc/47094/task/48371/root (readlink: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48371 osd_srv_h ceph txt unknown /proc/47094/task/48371/exe (readlink: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 47094 48371 osd_srv_h ceph NOFD /proc/47094/task/48371/fd (opendir: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 48482 root cwd unknown /proc/48482/cwd (readlink: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 48482 root rtd unknown /proc/48482/root (readlink: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 48482 root txt unknown /proc/48482/exe (readlink: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 48482 root NOFD /proc/48482/fd (opendir: Permission denied) 2024-03-11T20:01:36.030 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 48511 root cwd unknown /proc/48511/cwd (readlink: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 48511 root rtd unknown /proc/48511/root (readlink: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 48511 root txt unknown /proc/48511/exe (readlink: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 48511 root NOFD /proc/48511/fd (opendir: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 51071 root cwd unknown /proc/51071/cwd (readlink: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 51071 root rtd unknown /proc/51071/root (readlink: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 51071 root txt unknown /proc/51071/exe (readlink: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 51071 root NOFD /proc/51071/fd (opendir: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 53628 root cwd unknown /proc/53628/cwd (readlink: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 53628 root rtd unknown /proc/53628/root (readlink: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 53628 root txt unknown /proc/53628/exe (readlink: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 53628 root NOFD /proc/53628/fd (opendir: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 53630 root cwd unknown /proc/53630/cwd (readlink: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 53630 root rtd unknown /proc/53630/root (readlink: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 53630 root txt unknown /proc/53630/exe (readlink: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 53630 root NOFD /proc/53630/fd (opendir: Permission denied) 2024-03-11T20:01:36.031 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 ceph cwd unknown /proc/53632/cwd (readlink: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 ceph rtd unknown /proc/53632/root (readlink: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 ceph txt unknown /proc/53632/exe (readlink: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 ceph NOFD /proc/53632/fd (opendir: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53633 log ceph cwd unknown /proc/53632/task/53633/cwd (readlink: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53633 log ceph rtd unknown /proc/53632/task/53633/root (readlink: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53633 log ceph txt unknown /proc/53632/task/53633/exe (readlink: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53633 log ceph NOFD /proc/53632/task/53633/fd (opendir: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53635 msgr-work ceph cwd unknown /proc/53632/task/53635/cwd (readlink: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53635 msgr-work ceph rtd unknown /proc/53632/task/53635/root (readlink: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53635 msgr-work ceph txt unknown /proc/53632/task/53635/exe (readlink: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53635 msgr-work ceph NOFD /proc/53632/task/53635/fd (opendir: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53636 msgr-work ceph cwd unknown /proc/53632/task/53636/cwd (readlink: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53636 msgr-work ceph rtd unknown /proc/53632/task/53636/root (readlink: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53636 msgr-work ceph txt unknown /proc/53632/task/53636/exe (readlink: Permission denied) 2024-03-11T20:01:36.032 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53636 msgr-work ceph NOFD /proc/53632/task/53636/fd (opendir: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53637 msgr-work ceph cwd unknown /proc/53632/task/53637/cwd (readlink: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53637 msgr-work ceph rtd unknown /proc/53632/task/53637/root (readlink: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53637 msgr-work ceph txt unknown /proc/53632/task/53637/exe (readlink: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53637 msgr-work ceph NOFD /proc/53632/task/53637/fd (opendir: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53641 service ceph cwd unknown /proc/53632/task/53641/cwd (readlink: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53641 service ceph rtd unknown /proc/53632/task/53641/root (readlink: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53641 service ceph txt unknown /proc/53632/task/53641/exe (readlink: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53641 service ceph NOFD /proc/53632/task/53641/fd (opendir: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53642 admin_soc ceph cwd unknown /proc/53632/task/53642/cwd (readlink: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53642 admin_soc ceph rtd unknown /proc/53632/task/53642/root (readlink: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53642 admin_soc ceph txt unknown /proc/53632/task/53642/exe (readlink: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53642 admin_soc ceph NOFD /proc/53632/task/53642/fd (opendir: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53785 signal_ha ceph cwd unknown /proc/53632/task/53785/cwd (readlink: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53785 signal_ha ceph rtd unknown /proc/53632/task/53785/root (readlink: Permission denied) 2024-03-11T20:01:36.033 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53785 signal_ha ceph txt unknown /proc/53632/task/53785/exe (readlink: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53785 signal_ha ceph NOFD /proc/53632/task/53785/fd (opendir: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53786 io_contex ceph cwd unknown /proc/53632/task/53786/cwd (readlink: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53786 io_contex ceph rtd unknown /proc/53632/task/53786/root (readlink: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53786 io_contex ceph txt unknown /proc/53632/task/53786/exe (readlink: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53786 io_contex ceph NOFD /proc/53632/task/53786/fd (opendir: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53787 io_contex ceph cwd unknown /proc/53632/task/53787/cwd (readlink: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53787 io_contex ceph rtd unknown /proc/53632/task/53787/root (readlink: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53787 io_contex ceph txt unknown /proc/53632/task/53787/exe (readlink: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53787 io_contex ceph NOFD /proc/53632/task/53787/fd (opendir: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53790 OpHistory ceph cwd unknown /proc/53632/task/53790/cwd (readlink: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53790 OpHistory ceph rtd unknown /proc/53632/task/53790/root (readlink: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53790 OpHistory ceph txt unknown /proc/53632/task/53790/exe (readlink: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53790 OpHistory ceph NOFD /proc/53632/task/53790/fd (opendir: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53791 ceph_time ceph cwd unknown /proc/53632/task/53791/cwd (readlink: Permission denied) 2024-03-11T20:01:36.034 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53791 ceph_time ceph rtd unknown /proc/53632/task/53791/root (readlink: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53791 ceph_time ceph txt unknown /proc/53632/task/53791/exe (readlink: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 53791 ceph_time ceph NOFD /proc/53632/task/53791/fd (opendir: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54004 ceph-osd ceph cwd unknown /proc/53632/task/54004/cwd (readlink: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54004 ceph-osd ceph rtd unknown /proc/53632/task/54004/root (readlink: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54004 ceph-osd ceph txt unknown /proc/53632/task/54004/exe (readlink: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54004 ceph-osd ceph NOFD /proc/53632/task/54004/fd (opendir: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54005 safe_time ceph cwd unknown /proc/53632/task/54005/cwd (readlink: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54005 safe_time ceph rtd unknown /proc/53632/task/54005/root (readlink: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54005 safe_time ceph txt unknown /proc/53632/task/54005/exe (readlink: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54005 safe_time ceph NOFD /proc/53632/task/54005/fd (opendir: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54006 safe_time ceph cwd unknown /proc/53632/task/54006/cwd (readlink: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54006 safe_time ceph rtd unknown /proc/53632/task/54006/root (readlink: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54006 safe_time ceph txt unknown /proc/53632/task/54006/exe (readlink: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54006 safe_time ceph NOFD /proc/53632/task/54006/fd (opendir: Permission denied) 2024-03-11T20:01:36.035 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54007 safe_time ceph cwd unknown /proc/53632/task/54007/cwd (readlink: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54007 safe_time ceph rtd unknown /proc/53632/task/54007/root (readlink: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54007 safe_time ceph txt unknown /proc/53632/task/54007/exe (readlink: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54007 safe_time ceph NOFD /proc/53632/task/54007/fd (opendir: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54008 safe_time ceph cwd unknown /proc/53632/task/54008/cwd (readlink: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54008 safe_time ceph rtd unknown /proc/53632/task/54008/root (readlink: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54008 safe_time ceph txt unknown /proc/53632/task/54008/exe (readlink: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54008 safe_time ceph NOFD /proc/53632/task/54008/fd (opendir: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54009 fn_anonym ceph cwd unknown /proc/53632/task/54009/cwd (readlink: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54009 fn_anonym ceph rtd unknown /proc/53632/task/54009/root (readlink: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54009 fn_anonym ceph txt unknown /proc/53632/task/54009/exe (readlink: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54009 fn_anonym ceph NOFD /proc/53632/task/54009/fd (opendir: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54014 rocksdb:l ceph cwd unknown /proc/53632/task/54014/cwd (readlink: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54014 rocksdb:l ceph rtd unknown /proc/53632/task/54014/root (readlink: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54014 rocksdb:l ceph txt unknown /proc/53632/task/54014/exe (readlink: Permission denied) 2024-03-11T20:01:36.036 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54014 rocksdb:l ceph NOFD /proc/53632/task/54014/fd (opendir: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54015 rocksdb:l ceph cwd unknown /proc/53632/task/54015/cwd (readlink: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54015 rocksdb:l ceph rtd unknown /proc/53632/task/54015/root (readlink: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54015 rocksdb:l ceph txt unknown /proc/53632/task/54015/exe (readlink: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54015 rocksdb:l ceph NOFD /proc/53632/task/54015/fd (opendir: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54016 rocksdb:l ceph cwd unknown /proc/53632/task/54016/cwd (readlink: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54016 rocksdb:l ceph rtd unknown /proc/53632/task/54016/root (readlink: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54016 rocksdb:l ceph txt unknown /proc/53632/task/54016/exe (readlink: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54016 rocksdb:l ceph NOFD /proc/53632/task/54016/fd (opendir: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54017 rocksdb:h ceph cwd unknown /proc/53632/task/54017/cwd (readlink: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54017 rocksdb:h ceph rtd unknown /proc/53632/task/54017/root (readlink: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54017 rocksdb:h ceph txt unknown /proc/53632/task/54017/exe (readlink: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54017 rocksdb:h ceph NOFD /proc/53632/task/54017/fd (opendir: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54419 bstore_ai ceph cwd unknown /proc/53632/task/54419/cwd (readlink: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54419 bstore_ai ceph rtd unknown /proc/53632/task/54419/root (readlink: Permission denied) 2024-03-11T20:01:36.037 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54419 bstore_ai ceph txt unknown /proc/53632/task/54419/exe (readlink: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54419 bstore_ai ceph NOFD /proc/53632/task/54419/fd (opendir: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54646 bstore_ai ceph cwd unknown /proc/53632/task/54646/cwd (readlink: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54646 bstore_ai ceph rtd unknown /proc/53632/task/54646/root (readlink: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54646 bstore_ai ceph txt unknown /proc/53632/task/54646/exe (readlink: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54646 bstore_ai ceph NOFD /proc/53632/task/54646/fd (opendir: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54830 ceph-osd ceph cwd unknown /proc/53632/task/54830/cwd (readlink: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54830 ceph-osd ceph rtd unknown /proc/53632/task/54830/root (readlink: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54830 ceph-osd ceph txt unknown /proc/53632/task/54830/exe (readlink: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54830 ceph-osd ceph NOFD /proc/53632/task/54830/fd (opendir: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54831 cfin ceph cwd unknown /proc/53632/task/54831/cwd (readlink: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54831 cfin ceph rtd unknown /proc/53632/task/54831/root (readlink: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54831 cfin ceph txt unknown /proc/53632/task/54831/exe (readlink: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54831 cfin ceph NOFD /proc/53632/task/54831/fd (opendir: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54832 bstore_kv ceph cwd unknown /proc/53632/task/54832/cwd (readlink: Permission denied) 2024-03-11T20:01:36.038 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54832 bstore_kv ceph rtd unknown /proc/53632/task/54832/root (readlink: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54832 bstore_kv ceph txt unknown /proc/53632/task/54832/exe (readlink: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54832 bstore_kv ceph NOFD /proc/53632/task/54832/fd (opendir: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54833 bstore_kv ceph cwd unknown /proc/53632/task/54833/cwd (readlink: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54833 bstore_kv ceph rtd unknown /proc/53632/task/54833/root (readlink: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54833 bstore_kv ceph txt unknown /proc/53632/task/54833/exe (readlink: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54833 bstore_kv ceph NOFD /proc/53632/task/54833/fd (opendir: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54834 bstore_me ceph cwd unknown /proc/53632/task/54834/cwd (readlink: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54834 bstore_me ceph rtd unknown /proc/53632/task/54834/root (readlink: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54834 bstore_me ceph txt unknown /proc/53632/task/54834/exe (readlink: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54834 bstore_me ceph NOFD /proc/53632/task/54834/fd (opendir: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54837 ms_dispat ceph cwd unknown /proc/53632/task/54837/cwd (readlink: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54837 ms_dispat ceph rtd unknown /proc/53632/task/54837/root (readlink: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54837 ms_dispat ceph txt unknown /proc/53632/task/54837/exe (readlink: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54837 ms_dispat ceph NOFD /proc/53632/task/54837/fd (opendir: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54838 ms_local ceph cwd unknown /proc/53632/task/54838/cwd (readlink: Permission denied) 2024-03-11T20:01:36.039 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54838 ms_local ceph rtd unknown /proc/53632/task/54838/root (readlink: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54838 ms_local ceph txt unknown /proc/53632/task/54838/exe (readlink: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54838 ms_local ceph NOFD /proc/53632/task/54838/fd (opendir: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54839 safe_time ceph cwd unknown /proc/53632/task/54839/cwd (readlink: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54839 safe_time ceph rtd unknown /proc/53632/task/54839/root (readlink: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54839 safe_time ceph txt unknown /proc/53632/task/54839/exe (readlink: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54839 safe_time ceph NOFD /proc/53632/task/54839/fd (opendir: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54840 safe_time ceph cwd unknown /proc/53632/task/54840/cwd (readlink: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54840 safe_time ceph rtd unknown /proc/53632/task/54840/root (readlink: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54840 safe_time ceph txt unknown /proc/53632/task/54840/exe (readlink: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54840 safe_time ceph NOFD /proc/53632/task/54840/fd (opendir: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54841 ms_dispat ceph cwd unknown /proc/53632/task/54841/cwd (readlink: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54841 ms_dispat ceph rtd unknown /proc/53632/task/54841/root (readlink: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54841 ms_dispat ceph txt unknown /proc/53632/task/54841/exe (readlink: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54841 ms_dispat ceph NOFD /proc/53632/task/54841/fd (opendir: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54842 ms_local ceph cwd unknown /proc/53632/task/54842/cwd (readlink: Permission denied) 2024-03-11T20:01:36.040 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54842 ms_local ceph rtd unknown /proc/53632/task/54842/root (readlink: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54842 ms_local ceph txt unknown /proc/53632/task/54842/exe (readlink: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54842 ms_local ceph NOFD /proc/53632/task/54842/fd (opendir: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54843 ms_dispat ceph cwd unknown /proc/53632/task/54843/cwd (readlink: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54843 ms_dispat ceph rtd unknown /proc/53632/task/54843/root (readlink: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54843 ms_dispat ceph txt unknown /proc/53632/task/54843/exe (readlink: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54843 ms_dispat ceph NOFD /proc/53632/task/54843/fd (opendir: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54845 ms_local ceph cwd unknown /proc/53632/task/54845/cwd (readlink: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54845 ms_local ceph rtd unknown /proc/53632/task/54845/root (readlink: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54845 ms_local ceph txt unknown /proc/53632/task/54845/exe (readlink: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54845 ms_local ceph NOFD /proc/53632/task/54845/fd (opendir: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54846 ms_dispat ceph cwd unknown /proc/53632/task/54846/cwd (readlink: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54846 ms_dispat ceph rtd unknown /proc/53632/task/54846/root (readlink: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54846 ms_dispat ceph txt unknown /proc/53632/task/54846/exe (readlink: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54846 ms_dispat ceph NOFD /proc/53632/task/54846/fd (opendir: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54847 ms_local ceph cwd unknown /proc/53632/task/54847/cwd (readlink: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54847 ms_local ceph rtd unknown /proc/53632/task/54847/root (readlink: Permission denied) 2024-03-11T20:01:36.041 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54847 ms_local ceph txt unknown /proc/53632/task/54847/exe (readlink: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54847 ms_local ceph NOFD /proc/53632/task/54847/fd (opendir: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54848 ms_dispat ceph cwd unknown /proc/53632/task/54848/cwd (readlink: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54848 ms_dispat ceph rtd unknown /proc/53632/task/54848/root (readlink: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54848 ms_dispat ceph txt unknown /proc/53632/task/54848/exe (readlink: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54848 ms_dispat ceph NOFD /proc/53632/task/54848/fd (opendir: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54849 ms_local ceph cwd unknown /proc/53632/task/54849/cwd (readlink: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54849 ms_local ceph rtd unknown /proc/53632/task/54849/root (readlink: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54849 ms_local ceph txt unknown /proc/53632/task/54849/exe (readlink: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54849 ms_local ceph NOFD /proc/53632/task/54849/fd (opendir: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54850 ms_dispat ceph cwd unknown /proc/53632/task/54850/cwd (readlink: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54850 ms_dispat ceph rtd unknown /proc/53632/task/54850/root (readlink: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54850 ms_dispat ceph txt unknown /proc/53632/task/54850/exe (readlink: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54850 ms_dispat ceph NOFD /proc/53632/task/54850/fd (opendir: Permission denied) 2024-03-11T20:01:36.042 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54851 ms_local ceph cwd unknown /proc/53632/task/54851/cwd (readlink: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54851 ms_local ceph rtd unknown /proc/53632/task/54851/root (readlink: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54851 ms_local ceph txt unknown /proc/53632/task/54851/exe (readlink: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54851 ms_local ceph NOFD /proc/53632/task/54851/fd (opendir: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54852 ms_dispat ceph cwd unknown /proc/53632/task/54852/cwd (readlink: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54852 ms_dispat ceph rtd unknown /proc/53632/task/54852/root (readlink: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54852 ms_dispat ceph txt unknown /proc/53632/task/54852/exe (readlink: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54852 ms_dispat ceph NOFD /proc/53632/task/54852/fd (opendir: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54853 ms_local ceph cwd unknown /proc/53632/task/54853/cwd (readlink: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54853 ms_local ceph rtd unknown /proc/53632/task/54853/root (readlink: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54853 ms_local ceph txt unknown /proc/53632/task/54853/exe (readlink: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54853 ms_local ceph NOFD /proc/53632/task/54853/fd (opendir: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54854 fn_anonym ceph cwd unknown /proc/53632/task/54854/cwd (readlink: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54854 fn_anonym ceph rtd unknown /proc/53632/task/54854/root (readlink: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54854 fn_anonym ceph txt unknown /proc/53632/task/54854/exe (readlink: Permission denied) 2024-03-11T20:01:36.043 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54854 fn_anonym ceph NOFD /proc/53632/task/54854/fd (opendir: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54855 finisher ceph cwd unknown /proc/53632/task/54855/cwd (readlink: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54855 finisher ceph rtd unknown /proc/53632/task/54855/root (readlink: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54855 finisher ceph txt unknown /proc/53632/task/54855/exe (readlink: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54855 finisher ceph NOFD /proc/53632/task/54855/fd (opendir: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54856 safe_time ceph cwd unknown /proc/53632/task/54856/cwd (readlink: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54856 safe_time ceph rtd unknown /proc/53632/task/54856/root (readlink: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54856 safe_time ceph txt unknown /proc/53632/task/54856/exe (readlink: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54856 safe_time ceph NOFD /proc/53632/task/54856/fd (opendir: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54857 safe_time ceph cwd unknown /proc/53632/task/54857/cwd (readlink: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54857 safe_time ceph rtd unknown /proc/53632/task/54857/root (readlink: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54857 safe_time ceph txt unknown /proc/53632/task/54857/exe (readlink: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54857 safe_time ceph NOFD /proc/53632/task/54857/fd (opendir: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54858 ceph-osd ceph cwd unknown /proc/53632/task/54858/cwd (readlink: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54858 ceph-osd ceph rtd unknown /proc/53632/task/54858/root (readlink: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54858 ceph-osd ceph txt unknown /proc/53632/task/54858/exe (readlink: Permission denied) 2024-03-11T20:01:36.044 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54858 ceph-osd ceph NOFD /proc/53632/task/54858/fd (opendir: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54859 osd_srv_a ceph cwd unknown /proc/53632/task/54859/cwd (readlink: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54859 osd_srv_a ceph rtd unknown /proc/53632/task/54859/root (readlink: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54859 osd_srv_a ceph txt unknown /proc/53632/task/54859/exe (readlink: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54859 osd_srv_a ceph NOFD /proc/53632/task/54859/fd (opendir: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54860 tp_osd_tp ceph cwd unknown /proc/53632/task/54860/cwd (readlink: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54860 tp_osd_tp ceph rtd unknown /proc/53632/task/54860/root (readlink: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54860 tp_osd_tp ceph txt unknown /proc/53632/task/54860/exe (readlink: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54860 tp_osd_tp ceph NOFD /proc/53632/task/54860/fd (opendir: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54861 tp_osd_tp ceph cwd unknown /proc/53632/task/54861/cwd (readlink: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54861 tp_osd_tp ceph rtd unknown /proc/53632/task/54861/root (readlink: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54861 tp_osd_tp ceph txt unknown /proc/53632/task/54861/exe (readlink: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54861 tp_osd_tp ceph NOFD /proc/53632/task/54861/fd (opendir: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54862 tp_osd_tp ceph cwd unknown /proc/53632/task/54862/cwd (readlink: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54862 tp_osd_tp ceph rtd unknown /proc/53632/task/54862/root (readlink: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54862 tp_osd_tp ceph txt unknown /proc/53632/task/54862/exe (readlink: Permission denied) 2024-03-11T20:01:36.045 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54862 tp_osd_tp ceph NOFD /proc/53632/task/54862/fd (opendir: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54863 tp_osd_tp ceph cwd unknown /proc/53632/task/54863/cwd (readlink: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54863 tp_osd_tp ceph rtd unknown /proc/53632/task/54863/root (readlink: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54863 tp_osd_tp ceph txt unknown /proc/53632/task/54863/exe (readlink: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54863 tp_osd_tp ceph NOFD /proc/53632/task/54863/fd (opendir: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54864 tp_osd_tp ceph cwd unknown /proc/53632/task/54864/cwd (readlink: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54864 tp_osd_tp ceph rtd unknown /proc/53632/task/54864/root (readlink: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54864 tp_osd_tp ceph txt unknown /proc/53632/task/54864/exe (readlink: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54864 tp_osd_tp ceph NOFD /proc/53632/task/54864/fd (opendir: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54865 tp_osd_tp ceph cwd unknown /proc/53632/task/54865/cwd (readlink: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54865 tp_osd_tp ceph rtd unknown /proc/53632/task/54865/root (readlink: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54865 tp_osd_tp ceph txt unknown /proc/53632/task/54865/exe (readlink: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54865 tp_osd_tp ceph NOFD /proc/53632/task/54865/fd (opendir: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54866 tp_osd_tp ceph cwd unknown /proc/53632/task/54866/cwd (readlink: Permission denied) 2024-03-11T20:01:36.046 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54866 tp_osd_tp ceph rtd unknown /proc/53632/task/54866/root (readlink: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54866 tp_osd_tp ceph txt unknown /proc/53632/task/54866/exe (readlink: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54866 tp_osd_tp ceph NOFD /proc/53632/task/54866/fd (opendir: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54867 tp_osd_tp ceph cwd unknown /proc/53632/task/54867/cwd (readlink: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54867 tp_osd_tp ceph rtd unknown /proc/53632/task/54867/root (readlink: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54867 tp_osd_tp ceph txt unknown /proc/53632/task/54867/exe (readlink: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54867 tp_osd_tp ceph NOFD /proc/53632/task/54867/fd (opendir: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54868 tp_osd_tp ceph cwd unknown /proc/53632/task/54868/cwd (readlink: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54868 tp_osd_tp ceph rtd unknown /proc/53632/task/54868/root (readlink: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54868 tp_osd_tp ceph txt unknown /proc/53632/task/54868/exe (readlink: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54868 tp_osd_tp ceph NOFD /proc/53632/task/54868/fd (opendir: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54869 tp_osd_tp ceph cwd unknown /proc/53632/task/54869/cwd (readlink: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54869 tp_osd_tp ceph rtd unknown /proc/53632/task/54869/root (readlink: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54869 tp_osd_tp ceph txt unknown /proc/53632/task/54869/exe (readlink: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54869 tp_osd_tp ceph NOFD /proc/53632/task/54869/fd (opendir: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54870 tp_osd_tp ceph cwd unknown /proc/53632/task/54870/cwd (readlink: Permission denied) 2024-03-11T20:01:36.047 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54870 tp_osd_tp ceph rtd unknown /proc/53632/task/54870/root (readlink: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54870 tp_osd_tp ceph txt unknown /proc/53632/task/54870/exe (readlink: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54870 tp_osd_tp ceph NOFD /proc/53632/task/54870/fd (opendir: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54871 tp_osd_tp ceph cwd unknown /proc/53632/task/54871/cwd (readlink: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54871 tp_osd_tp ceph rtd unknown /proc/53632/task/54871/root (readlink: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54871 tp_osd_tp ceph txt unknown /proc/53632/task/54871/exe (readlink: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54871 tp_osd_tp ceph NOFD /proc/53632/task/54871/fd (opendir: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54872 tp_osd_tp ceph cwd unknown /proc/53632/task/54872/cwd (readlink: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54872 tp_osd_tp ceph rtd unknown /proc/53632/task/54872/root (readlink: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54872 tp_osd_tp ceph txt unknown /proc/53632/task/54872/exe (readlink: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54872 tp_osd_tp ceph NOFD /proc/53632/task/54872/fd (opendir: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54873 tp_osd_tp ceph cwd unknown /proc/53632/task/54873/cwd (readlink: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54873 tp_osd_tp ceph rtd unknown /proc/53632/task/54873/root (readlink: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54873 tp_osd_tp ceph txt unknown /proc/53632/task/54873/exe (readlink: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54873 tp_osd_tp ceph NOFD /proc/53632/task/54873/fd (opendir: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54874 tp_osd_tp ceph cwd unknown /proc/53632/task/54874/cwd (readlink: Permission denied) 2024-03-11T20:01:36.048 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54874 tp_osd_tp ceph rtd unknown /proc/53632/task/54874/root (readlink: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54874 tp_osd_tp ceph txt unknown /proc/53632/task/54874/exe (readlink: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54874 tp_osd_tp ceph NOFD /proc/53632/task/54874/fd (opendir: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54875 tp_osd_tp ceph cwd unknown /proc/53632/task/54875/cwd (readlink: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54875 tp_osd_tp ceph rtd unknown /proc/53632/task/54875/root (readlink: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54875 tp_osd_tp ceph txt unknown /proc/53632/task/54875/exe (readlink: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54875 tp_osd_tp ceph NOFD /proc/53632/task/54875/fd (opendir: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54876 osd_srv_h ceph cwd unknown /proc/53632/task/54876/cwd (readlink: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54876 osd_srv_h ceph rtd unknown /proc/53632/task/54876/root (readlink: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54876 osd_srv_h ceph txt unknown /proc/53632/task/54876/exe (readlink: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 53632 54876 osd_srv_h ceph NOFD /proc/53632/task/54876/fd (opendir: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 54943 root cwd unknown /proc/54943/cwd (readlink: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 54943 root rtd unknown /proc/54943/root (readlink: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 54943 root txt unknown /proc/54943/exe (readlink: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 54943 root NOFD /proc/54943/fd (opendir: Permission denied) 2024-03-11T20:01:36.049 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 55044 root cwd unknown /proc/55044/cwd (readlink: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 55044 root rtd unknown /proc/55044/root (readlink: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 55044 root txt unknown /proc/55044/exe (readlink: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 55044 root NOFD /proc/55044/fd (opendir: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 55064 root cwd unknown /proc/55064/cwd (readlink: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 55064 root rtd unknown /proc/55064/root (readlink: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 55064 root txt unknown /proc/55064/exe (readlink: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 55064 root NOFD /proc/55064/fd (opendir: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 60183 root cwd unknown /proc/60183/cwd (readlink: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 60183 root rtd unknown /proc/60183/root (readlink: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 60183 root txt unknown /proc/60183/exe (readlink: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 60183 root NOFD /proc/60183/fd (opendir: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 60185 root cwd unknown /proc/60185/cwd (readlink: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 60185 root rtd unknown /proc/60185/root (readlink: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 60185 root txt unknown /proc/60185/exe (readlink: Permission denied) 2024-03-11T20:01:36.050 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 60185 root NOFD /proc/60185/fd (opendir: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 ceph cwd unknown /proc/60188/cwd (readlink: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 ceph rtd unknown /proc/60188/root (readlink: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 ceph txt unknown /proc/60188/exe (readlink: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 ceph NOFD /proc/60188/fd (opendir: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60189 log ceph cwd unknown /proc/60188/task/60189/cwd (readlink: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60189 log ceph rtd unknown /proc/60188/task/60189/root (readlink: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60189 log ceph txt unknown /proc/60188/task/60189/exe (readlink: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60189 log ceph NOFD /proc/60188/task/60189/fd (opendir: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60191 msgr-work ceph cwd unknown /proc/60188/task/60191/cwd (readlink: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60191 msgr-work ceph rtd unknown /proc/60188/task/60191/root (readlink: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60191 msgr-work ceph txt unknown /proc/60188/task/60191/exe (readlink: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60191 msgr-work ceph NOFD /proc/60188/task/60191/fd (opendir: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60192 msgr-work ceph cwd unknown /proc/60188/task/60192/cwd (readlink: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60192 msgr-work ceph rtd unknown /proc/60188/task/60192/root (readlink: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60192 msgr-work ceph txt unknown /proc/60188/task/60192/exe (readlink: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60192 msgr-work ceph NOFD /proc/60188/task/60192/fd (opendir: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60193 msgr-work ceph cwd unknown /proc/60188/task/60193/cwd (readlink: Permission denied) 2024-03-11T20:01:36.051 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60193 msgr-work ceph rtd unknown /proc/60188/task/60193/root (readlink: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60193 msgr-work ceph txt unknown /proc/60188/task/60193/exe (readlink: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60193 msgr-work ceph NOFD /proc/60188/task/60193/fd (opendir: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60197 service ceph cwd unknown /proc/60188/task/60197/cwd (readlink: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60197 service ceph rtd unknown /proc/60188/task/60197/root (readlink: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60197 service ceph txt unknown /proc/60188/task/60197/exe (readlink: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60197 service ceph NOFD /proc/60188/task/60197/fd (opendir: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60198 admin_soc ceph cwd unknown /proc/60188/task/60198/cwd (readlink: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60198 admin_soc ceph rtd unknown /proc/60188/task/60198/root (readlink: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60198 admin_soc ceph txt unknown /proc/60188/task/60198/exe (readlink: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60198 admin_soc ceph NOFD /proc/60188/task/60198/fd (opendir: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60341 signal_ha ceph cwd unknown /proc/60188/task/60341/cwd (readlink: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60341 signal_ha ceph rtd unknown /proc/60188/task/60341/root (readlink: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60341 signal_ha ceph txt unknown /proc/60188/task/60341/exe (readlink: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60341 signal_ha ceph NOFD /proc/60188/task/60341/fd (opendir: Permission denied) 2024-03-11T20:01:36.052 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60342 io_contex ceph cwd unknown /proc/60188/task/60342/cwd (readlink: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60342 io_contex ceph rtd unknown /proc/60188/task/60342/root (readlink: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60342 io_contex ceph txt unknown /proc/60188/task/60342/exe (readlink: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60342 io_contex ceph NOFD /proc/60188/task/60342/fd (opendir: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60343 io_contex ceph cwd unknown /proc/60188/task/60343/cwd (readlink: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60343 io_contex ceph rtd unknown /proc/60188/task/60343/root (readlink: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60343 io_contex ceph txt unknown /proc/60188/task/60343/exe (readlink: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60343 io_contex ceph NOFD /proc/60188/task/60343/fd (opendir: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60347 OpHistory ceph cwd unknown /proc/60188/task/60347/cwd (readlink: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60347 OpHistory ceph rtd unknown /proc/60188/task/60347/root (readlink: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60347 OpHistory ceph txt unknown /proc/60188/task/60347/exe (readlink: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60347 OpHistory ceph NOFD /proc/60188/task/60347/fd (opendir: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60348 ceph_time ceph cwd unknown /proc/60188/task/60348/cwd (readlink: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60348 ceph_time ceph rtd unknown /proc/60188/task/60348/root (readlink: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60348 ceph_time ceph txt unknown /proc/60188/task/60348/exe (readlink: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60348 ceph_time ceph NOFD /proc/60188/task/60348/fd (opendir: Permission denied) 2024-03-11T20:01:36.053 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60569 ceph-osd ceph cwd unknown /proc/60188/task/60569/cwd (readlink: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60569 ceph-osd ceph rtd unknown /proc/60188/task/60569/root (readlink: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60569 ceph-osd ceph txt unknown /proc/60188/task/60569/exe (readlink: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60569 ceph-osd ceph NOFD /proc/60188/task/60569/fd (opendir: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60570 safe_time ceph cwd unknown /proc/60188/task/60570/cwd (readlink: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60570 safe_time ceph rtd unknown /proc/60188/task/60570/root (readlink: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60570 safe_time ceph txt unknown /proc/60188/task/60570/exe (readlink: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60570 safe_time ceph NOFD /proc/60188/task/60570/fd (opendir: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60571 safe_time ceph cwd unknown /proc/60188/task/60571/cwd (readlink: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60571 safe_time ceph rtd unknown /proc/60188/task/60571/root (readlink: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60571 safe_time ceph txt unknown /proc/60188/task/60571/exe (readlink: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60571 safe_time ceph NOFD /proc/60188/task/60571/fd (opendir: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60572 safe_time ceph cwd unknown /proc/60188/task/60572/cwd (readlink: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60572 safe_time ceph rtd unknown /proc/60188/task/60572/root (readlink: Permission denied) 2024-03-11T20:01:36.054 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60572 safe_time ceph txt unknown /proc/60188/task/60572/exe (readlink: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60572 safe_time ceph NOFD /proc/60188/task/60572/fd (opendir: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60573 safe_time ceph cwd unknown /proc/60188/task/60573/cwd (readlink: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60573 safe_time ceph rtd unknown /proc/60188/task/60573/root (readlink: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60573 safe_time ceph txt unknown /proc/60188/task/60573/exe (readlink: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60573 safe_time ceph NOFD /proc/60188/task/60573/fd (opendir: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60574 fn_anonym ceph cwd unknown /proc/60188/task/60574/cwd (readlink: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60574 fn_anonym ceph rtd unknown /proc/60188/task/60574/root (readlink: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60574 fn_anonym ceph txt unknown /proc/60188/task/60574/exe (readlink: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60574 fn_anonym ceph NOFD /proc/60188/task/60574/fd (opendir: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60581 rocksdb:l ceph cwd unknown /proc/60188/task/60581/cwd (readlink: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60581 rocksdb:l ceph rtd unknown /proc/60188/task/60581/root (readlink: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60581 rocksdb:l ceph txt unknown /proc/60188/task/60581/exe (readlink: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60581 rocksdb:l ceph NOFD /proc/60188/task/60581/fd (opendir: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60582 rocksdb:l ceph cwd unknown /proc/60188/task/60582/cwd (readlink: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60582 rocksdb:l ceph rtd unknown /proc/60188/task/60582/root (readlink: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60582 rocksdb:l ceph txt unknown /proc/60188/task/60582/exe (readlink: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60582 rocksdb:l ceph NOFD /proc/60188/task/60582/fd (opendir: Permission denied) 2024-03-11T20:01:36.055 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60583 rocksdb:l ceph cwd unknown /proc/60188/task/60583/cwd (readlink: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60583 rocksdb:l ceph rtd unknown /proc/60188/task/60583/root (readlink: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60583 rocksdb:l ceph txt unknown /proc/60188/task/60583/exe (readlink: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60583 rocksdb:l ceph NOFD /proc/60188/task/60583/fd (opendir: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60584 rocksdb:h ceph cwd unknown /proc/60188/task/60584/cwd (readlink: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60584 rocksdb:h ceph rtd unknown /proc/60188/task/60584/root (readlink: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60584 rocksdb:h ceph txt unknown /proc/60188/task/60584/exe (readlink: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60584 rocksdb:h ceph NOFD /proc/60188/task/60584/fd (opendir: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60993 bstore_ai ceph cwd unknown /proc/60188/task/60993/cwd (readlink: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60993 bstore_ai ceph rtd unknown /proc/60188/task/60993/root (readlink: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60993 bstore_ai ceph txt unknown /proc/60188/task/60993/exe (readlink: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 60993 bstore_ai ceph NOFD /proc/60188/task/60993/fd (opendir: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61221 bstore_ai ceph cwd unknown /proc/60188/task/61221/cwd (readlink: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61221 bstore_ai ceph rtd unknown /proc/60188/task/61221/root (readlink: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61221 bstore_ai ceph txt unknown /proc/60188/task/61221/exe (readlink: Permission denied) 2024-03-11T20:01:36.056 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61221 bstore_ai ceph NOFD /proc/60188/task/61221/fd (opendir: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61407 ceph-osd ceph cwd unknown /proc/60188/task/61407/cwd (readlink: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61407 ceph-osd ceph rtd unknown /proc/60188/task/61407/root (readlink: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61407 ceph-osd ceph txt unknown /proc/60188/task/61407/exe (readlink: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61407 ceph-osd ceph NOFD /proc/60188/task/61407/fd (opendir: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61408 cfin ceph cwd unknown /proc/60188/task/61408/cwd (readlink: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61408 cfin ceph rtd unknown /proc/60188/task/61408/root (readlink: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61408 cfin ceph txt unknown /proc/60188/task/61408/exe (readlink: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61408 cfin ceph NOFD /proc/60188/task/61408/fd (opendir: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61409 bstore_kv ceph cwd unknown /proc/60188/task/61409/cwd (readlink: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61409 bstore_kv ceph rtd unknown /proc/60188/task/61409/root (readlink: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61409 bstore_kv ceph txt unknown /proc/60188/task/61409/exe (readlink: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61409 bstore_kv ceph NOFD /proc/60188/task/61409/fd (opendir: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61410 bstore_kv ceph cwd unknown /proc/60188/task/61410/cwd (readlink: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61410 bstore_kv ceph rtd unknown /proc/60188/task/61410/root (readlink: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61410 bstore_kv ceph txt unknown /proc/60188/task/61410/exe (readlink: Permission denied) 2024-03-11T20:01:36.057 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61410 bstore_kv ceph NOFD /proc/60188/task/61410/fd (opendir: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61411 bstore_me ceph cwd unknown /proc/60188/task/61411/cwd (readlink: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61411 bstore_me ceph rtd unknown /proc/60188/task/61411/root (readlink: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61411 bstore_me ceph txt unknown /proc/60188/task/61411/exe (readlink: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61411 bstore_me ceph NOFD /proc/60188/task/61411/fd (opendir: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61416 ms_dispat ceph cwd unknown /proc/60188/task/61416/cwd (readlink: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61416 ms_dispat ceph rtd unknown /proc/60188/task/61416/root (readlink: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61416 ms_dispat ceph txt unknown /proc/60188/task/61416/exe (readlink: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61416 ms_dispat ceph NOFD /proc/60188/task/61416/fd (opendir: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61417 ms_local ceph cwd unknown /proc/60188/task/61417/cwd (readlink: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61417 ms_local ceph rtd unknown /proc/60188/task/61417/root (readlink: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61417 ms_local ceph txt unknown /proc/60188/task/61417/exe (readlink: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61417 ms_local ceph NOFD /proc/60188/task/61417/fd (opendir: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61418 safe_time ceph cwd unknown /proc/60188/task/61418/cwd (readlink: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61418 safe_time ceph rtd unknown /proc/60188/task/61418/root (readlink: Permission denied) 2024-03-11T20:01:36.058 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61418 safe_time ceph txt unknown /proc/60188/task/61418/exe (readlink: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61418 safe_time ceph NOFD /proc/60188/task/61418/fd (opendir: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61419 safe_time ceph cwd unknown /proc/60188/task/61419/cwd (readlink: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61419 safe_time ceph rtd unknown /proc/60188/task/61419/root (readlink: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61419 safe_time ceph txt unknown /proc/60188/task/61419/exe (readlink: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61419 safe_time ceph NOFD /proc/60188/task/61419/fd (opendir: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61420 ms_dispat ceph cwd unknown /proc/60188/task/61420/cwd (readlink: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61420 ms_dispat ceph rtd unknown /proc/60188/task/61420/root (readlink: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61420 ms_dispat ceph txt unknown /proc/60188/task/61420/exe (readlink: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61420 ms_dispat ceph NOFD /proc/60188/task/61420/fd (opendir: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61421 ms_local ceph cwd unknown /proc/60188/task/61421/cwd (readlink: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61421 ms_local ceph rtd unknown /proc/60188/task/61421/root (readlink: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61421 ms_local ceph txt unknown /proc/60188/task/61421/exe (readlink: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61421 ms_local ceph NOFD /proc/60188/task/61421/fd (opendir: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61422 ms_dispat ceph cwd unknown /proc/60188/task/61422/cwd (readlink: Permission denied) 2024-03-11T20:01:36.059 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61422 ms_dispat ceph rtd unknown /proc/60188/task/61422/root (readlink: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61422 ms_dispat ceph txt unknown /proc/60188/task/61422/exe (readlink: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61422 ms_dispat ceph NOFD /proc/60188/task/61422/fd (opendir: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61423 ms_local ceph cwd unknown /proc/60188/task/61423/cwd (readlink: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61423 ms_local ceph rtd unknown /proc/60188/task/61423/root (readlink: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61423 ms_local ceph txt unknown /proc/60188/task/61423/exe (readlink: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61423 ms_local ceph NOFD /proc/60188/task/61423/fd (opendir: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61424 ms_dispat ceph cwd unknown /proc/60188/task/61424/cwd (readlink: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61424 ms_dispat ceph rtd unknown /proc/60188/task/61424/root (readlink: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61424 ms_dispat ceph txt unknown /proc/60188/task/61424/exe (readlink: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61424 ms_dispat ceph NOFD /proc/60188/task/61424/fd (opendir: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61425 ms_local ceph cwd unknown /proc/60188/task/61425/cwd (readlink: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61425 ms_local ceph rtd unknown /proc/60188/task/61425/root (readlink: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61425 ms_local ceph txt unknown /proc/60188/task/61425/exe (readlink: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61425 ms_local ceph NOFD /proc/60188/task/61425/fd (opendir: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61426 ms_dispat ceph cwd unknown /proc/60188/task/61426/cwd (readlink: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61426 ms_dispat ceph rtd unknown /proc/60188/task/61426/root (readlink: Permission denied) 2024-03-11T20:01:36.060 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61426 ms_dispat ceph txt unknown /proc/60188/task/61426/exe (readlink: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61426 ms_dispat ceph NOFD /proc/60188/task/61426/fd (opendir: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61427 ms_local ceph cwd unknown /proc/60188/task/61427/cwd (readlink: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61427 ms_local ceph rtd unknown /proc/60188/task/61427/root (readlink: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61427 ms_local ceph txt unknown /proc/60188/task/61427/exe (readlink: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61427 ms_local ceph NOFD /proc/60188/task/61427/fd (opendir: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61428 ms_dispat ceph cwd unknown /proc/60188/task/61428/cwd (readlink: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61428 ms_dispat ceph rtd unknown /proc/60188/task/61428/root (readlink: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61428 ms_dispat ceph txt unknown /proc/60188/task/61428/exe (readlink: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61428 ms_dispat ceph NOFD /proc/60188/task/61428/fd (opendir: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61429 ms_local ceph cwd unknown /proc/60188/task/61429/cwd (readlink: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61429 ms_local ceph rtd unknown /proc/60188/task/61429/root (readlink: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61429 ms_local ceph txt unknown /proc/60188/task/61429/exe (readlink: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61429 ms_local ceph NOFD /proc/60188/task/61429/fd (opendir: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61430 ms_dispat ceph cwd unknown /proc/60188/task/61430/cwd (readlink: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61430 ms_dispat ceph rtd unknown /proc/60188/task/61430/root (readlink: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61430 ms_dispat ceph txt unknown /proc/60188/task/61430/exe (readlink: Permission denied) 2024-03-11T20:01:36.061 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61430 ms_dispat ceph NOFD /proc/60188/task/61430/fd (opendir: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61431 ms_local ceph cwd unknown /proc/60188/task/61431/cwd (readlink: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61431 ms_local ceph rtd unknown /proc/60188/task/61431/root (readlink: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61431 ms_local ceph txt unknown /proc/60188/task/61431/exe (readlink: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61431 ms_local ceph NOFD /proc/60188/task/61431/fd (opendir: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61432 fn_anonym ceph cwd unknown /proc/60188/task/61432/cwd (readlink: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61432 fn_anonym ceph rtd unknown /proc/60188/task/61432/root (readlink: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61432 fn_anonym ceph txt unknown /proc/60188/task/61432/exe (readlink: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61432 fn_anonym ceph NOFD /proc/60188/task/61432/fd (opendir: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61433 finisher ceph cwd unknown /proc/60188/task/61433/cwd (readlink: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61433 finisher ceph rtd unknown /proc/60188/task/61433/root (readlink: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61433 finisher ceph txt unknown /proc/60188/task/61433/exe (readlink: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61433 finisher ceph NOFD /proc/60188/task/61433/fd (opendir: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61434 safe_time ceph cwd unknown /proc/60188/task/61434/cwd (readlink: Permission denied) 2024-03-11T20:01:36.062 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61434 safe_time ceph rtd unknown /proc/60188/task/61434/root (readlink: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61434 safe_time ceph txt unknown /proc/60188/task/61434/exe (readlink: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61434 safe_time ceph NOFD /proc/60188/task/61434/fd (opendir: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61435 safe_time ceph cwd unknown /proc/60188/task/61435/cwd (readlink: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61435 safe_time ceph rtd unknown /proc/60188/task/61435/root (readlink: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61435 safe_time ceph txt unknown /proc/60188/task/61435/exe (readlink: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61435 safe_time ceph NOFD /proc/60188/task/61435/fd (opendir: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61436 ceph-osd ceph cwd unknown /proc/60188/task/61436/cwd (readlink: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61436 ceph-osd ceph rtd unknown /proc/60188/task/61436/root (readlink: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61436 ceph-osd ceph txt unknown /proc/60188/task/61436/exe (readlink: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61436 ceph-osd ceph NOFD /proc/60188/task/61436/fd (opendir: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61437 osd_srv_a ceph cwd unknown /proc/60188/task/61437/cwd (readlink: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61437 osd_srv_a ceph rtd unknown /proc/60188/task/61437/root (readlink: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61437 osd_srv_a ceph txt unknown /proc/60188/task/61437/exe (readlink: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61437 osd_srv_a ceph NOFD /proc/60188/task/61437/fd (opendir: Permission denied) 2024-03-11T20:01:36.063 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61438 tp_osd_tp ceph cwd unknown /proc/60188/task/61438/cwd (readlink: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61438 tp_osd_tp ceph rtd unknown /proc/60188/task/61438/root (readlink: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61438 tp_osd_tp ceph txt unknown /proc/60188/task/61438/exe (readlink: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61438 tp_osd_tp ceph NOFD /proc/60188/task/61438/fd (opendir: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61439 tp_osd_tp ceph cwd unknown /proc/60188/task/61439/cwd (readlink: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61439 tp_osd_tp ceph rtd unknown /proc/60188/task/61439/root (readlink: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61439 tp_osd_tp ceph txt unknown /proc/60188/task/61439/exe (readlink: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61439 tp_osd_tp ceph NOFD /proc/60188/task/61439/fd (opendir: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61440 tp_osd_tp ceph cwd unknown /proc/60188/task/61440/cwd (readlink: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61440 tp_osd_tp ceph rtd unknown /proc/60188/task/61440/root (readlink: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61440 tp_osd_tp ceph txt unknown /proc/60188/task/61440/exe (readlink: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61440 tp_osd_tp ceph NOFD /proc/60188/task/61440/fd (opendir: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61441 tp_osd_tp ceph cwd unknown /proc/60188/task/61441/cwd (readlink: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61441 tp_osd_tp ceph rtd unknown /proc/60188/task/61441/root (readlink: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61441 tp_osd_tp ceph txt unknown /proc/60188/task/61441/exe (readlink: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61441 tp_osd_tp ceph NOFD /proc/60188/task/61441/fd (opendir: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61442 tp_osd_tp ceph cwd unknown /proc/60188/task/61442/cwd (readlink: Permission denied) 2024-03-11T20:01:36.064 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61442 tp_osd_tp ceph rtd unknown /proc/60188/task/61442/root (readlink: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61442 tp_osd_tp ceph txt unknown /proc/60188/task/61442/exe (readlink: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61442 tp_osd_tp ceph NOFD /proc/60188/task/61442/fd (opendir: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61443 tp_osd_tp ceph cwd unknown /proc/60188/task/61443/cwd (readlink: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61443 tp_osd_tp ceph rtd unknown /proc/60188/task/61443/root (readlink: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61443 tp_osd_tp ceph txt unknown /proc/60188/task/61443/exe (readlink: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61443 tp_osd_tp ceph NOFD /proc/60188/task/61443/fd (opendir: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61444 tp_osd_tp ceph cwd unknown /proc/60188/task/61444/cwd (readlink: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61444 tp_osd_tp ceph rtd unknown /proc/60188/task/61444/root (readlink: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61444 tp_osd_tp ceph txt unknown /proc/60188/task/61444/exe (readlink: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61444 tp_osd_tp ceph NOFD /proc/60188/task/61444/fd (opendir: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61445 tp_osd_tp ceph cwd unknown /proc/60188/task/61445/cwd (readlink: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61445 tp_osd_tp ceph rtd unknown /proc/60188/task/61445/root (readlink: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61445 tp_osd_tp ceph txt unknown /proc/60188/task/61445/exe (readlink: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61445 tp_osd_tp ceph NOFD /proc/60188/task/61445/fd (opendir: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61446 tp_osd_tp ceph cwd unknown /proc/60188/task/61446/cwd (readlink: Permission denied) 2024-03-11T20:01:36.065 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61446 tp_osd_tp ceph rtd unknown /proc/60188/task/61446/root (readlink: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61446 tp_osd_tp ceph txt unknown /proc/60188/task/61446/exe (readlink: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61446 tp_osd_tp ceph NOFD /proc/60188/task/61446/fd (opendir: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61447 tp_osd_tp ceph cwd unknown /proc/60188/task/61447/cwd (readlink: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61447 tp_osd_tp ceph rtd unknown /proc/60188/task/61447/root (readlink: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61447 tp_osd_tp ceph txt unknown /proc/60188/task/61447/exe (readlink: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61447 tp_osd_tp ceph NOFD /proc/60188/task/61447/fd (opendir: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61448 tp_osd_tp ceph cwd unknown /proc/60188/task/61448/cwd (readlink: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61448 tp_osd_tp ceph rtd unknown /proc/60188/task/61448/root (readlink: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61448 tp_osd_tp ceph txt unknown /proc/60188/task/61448/exe (readlink: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61448 tp_osd_tp ceph NOFD /proc/60188/task/61448/fd (opendir: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61449 tp_osd_tp ceph cwd unknown /proc/60188/task/61449/cwd (readlink: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61449 tp_osd_tp ceph rtd unknown /proc/60188/task/61449/root (readlink: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61449 tp_osd_tp ceph txt unknown /proc/60188/task/61449/exe (readlink: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61449 tp_osd_tp ceph NOFD /proc/60188/task/61449/fd (opendir: Permission denied) 2024-03-11T20:01:36.066 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61450 tp_osd_tp ceph cwd unknown /proc/60188/task/61450/cwd (readlink: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61450 tp_osd_tp ceph rtd unknown /proc/60188/task/61450/root (readlink: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61450 tp_osd_tp ceph txt unknown /proc/60188/task/61450/exe (readlink: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61450 tp_osd_tp ceph NOFD /proc/60188/task/61450/fd (opendir: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61451 tp_osd_tp ceph cwd unknown /proc/60188/task/61451/cwd (readlink: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61451 tp_osd_tp ceph rtd unknown /proc/60188/task/61451/root (readlink: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61451 tp_osd_tp ceph txt unknown /proc/60188/task/61451/exe (readlink: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61451 tp_osd_tp ceph NOFD /proc/60188/task/61451/fd (opendir: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61452 tp_osd_tp ceph cwd unknown /proc/60188/task/61452/cwd (readlink: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61452 tp_osd_tp ceph rtd unknown /proc/60188/task/61452/root (readlink: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61452 tp_osd_tp ceph txt unknown /proc/60188/task/61452/exe (readlink: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61452 tp_osd_tp ceph NOFD /proc/60188/task/61452/fd (opendir: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61453 tp_osd_tp ceph cwd unknown /proc/60188/task/61453/cwd (readlink: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61453 tp_osd_tp ceph rtd unknown /proc/60188/task/61453/root (readlink: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61453 tp_osd_tp ceph txt unknown /proc/60188/task/61453/exe (readlink: Permission denied) 2024-03-11T20:01:36.067 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61453 tp_osd_tp ceph NOFD /proc/60188/task/61453/fd (opendir: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61454 osd_srv_h ceph cwd unknown /proc/60188/task/61454/cwd (readlink: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61454 osd_srv_h ceph rtd unknown /proc/60188/task/61454/root (readlink: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61454 osd_srv_h ceph txt unknown /proc/60188/task/61454/exe (readlink: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 60188 61454 osd_srv_h ceph NOFD /proc/60188/task/61454/fd (opendir: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 60578 root cwd unknown /proc/60578/cwd (readlink: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 60578 root rtd unknown /proc/60578/root (readlink: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 60578 root txt unknown /proc/60578/exe (readlink: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 60578 root NOFD /proc/60578/fd (opendir: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 61687 root cwd unknown /proc/61687/cwd (readlink: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 61687 root rtd unknown /proc/61687/root (readlink: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 61687 root txt unknown /proc/61687/exe (readlink: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 61687 root NOFD /proc/61687/fd (opendir: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 61718 root cwd unknown /proc/61718/cwd (readlink: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 61718 root rtd unknown /proc/61718/root (readlink: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 61718 root txt unknown /proc/61718/exe (readlink: Permission denied) 2024-03-11T20:01:36.068 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 61718 root NOFD /proc/61718/fd (opendir: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 64359 root cwd unknown /proc/64359/cwd (readlink: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 64359 root rtd unknown /proc/64359/root (readlink: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 64359 root txt unknown /proc/64359/exe (readlink: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 64359 root NOFD /proc/64359/fd (opendir: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 66779 root cwd unknown /proc/66779/cwd (readlink: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 66779 root rtd unknown /proc/66779/root (readlink: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 66779 root txt unknown /proc/66779/exe (readlink: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 66779 root NOFD /proc/66779/fd (opendir: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 66781 root cwd unknown /proc/66781/cwd (readlink: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 66781 root rtd unknown /proc/66781/root (readlink: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 66781 root txt unknown /proc/66781/exe (readlink: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 66781 root NOFD /proc/66781/fd (opendir: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 ceph cwd unknown /proc/66783/cwd (readlink: Permission denied) 2024-03-11T20:01:36.069 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 ceph rtd unknown /proc/66783/root (readlink: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 ceph txt unknown /proc/66783/exe (readlink: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 ceph NOFD /proc/66783/fd (opendir: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66784 log ceph cwd unknown /proc/66783/task/66784/cwd (readlink: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66784 log ceph rtd unknown /proc/66783/task/66784/root (readlink: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66784 log ceph txt unknown /proc/66783/task/66784/exe (readlink: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66784 log ceph NOFD /proc/66783/task/66784/fd (opendir: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66786 msgr-work ceph cwd unknown /proc/66783/task/66786/cwd (readlink: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66786 msgr-work ceph rtd unknown /proc/66783/task/66786/root (readlink: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66786 msgr-work ceph txt unknown /proc/66783/task/66786/exe (readlink: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66786 msgr-work ceph NOFD /proc/66783/task/66786/fd (opendir: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66787 msgr-work ceph cwd unknown /proc/66783/task/66787/cwd (readlink: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66787 msgr-work ceph rtd unknown /proc/66783/task/66787/root (readlink: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66787 msgr-work ceph txt unknown /proc/66783/task/66787/exe (readlink: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66787 msgr-work ceph NOFD /proc/66783/task/66787/fd (opendir: Permission denied) 2024-03-11T20:01:36.070 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66788 msgr-work ceph cwd unknown /proc/66783/task/66788/cwd (readlink: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66788 msgr-work ceph rtd unknown /proc/66783/task/66788/root (readlink: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66788 msgr-work ceph txt unknown /proc/66783/task/66788/exe (readlink: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66788 msgr-work ceph NOFD /proc/66783/task/66788/fd (opendir: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66792 service ceph cwd unknown /proc/66783/task/66792/cwd (readlink: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66792 service ceph rtd unknown /proc/66783/task/66792/root (readlink: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66792 service ceph txt unknown /proc/66783/task/66792/exe (readlink: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66792 service ceph NOFD /proc/66783/task/66792/fd (opendir: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66793 admin_soc ceph cwd unknown /proc/66783/task/66793/cwd (readlink: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66793 admin_soc ceph rtd unknown /proc/66783/task/66793/root (readlink: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66793 admin_soc ceph txt unknown /proc/66783/task/66793/exe (readlink: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66793 admin_soc ceph NOFD /proc/66783/task/66793/fd (opendir: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66938 signal_ha ceph cwd unknown /proc/66783/task/66938/cwd (readlink: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66938 signal_ha ceph rtd unknown /proc/66783/task/66938/root (readlink: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66938 signal_ha ceph txt unknown /proc/66783/task/66938/exe (readlink: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66938 signal_ha ceph NOFD /proc/66783/task/66938/fd (opendir: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66939 io_contex ceph cwd unknown /proc/66783/task/66939/cwd (readlink: Permission denied) 2024-03-11T20:01:36.071 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66939 io_contex ceph rtd unknown /proc/66783/task/66939/root (readlink: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66939 io_contex ceph txt unknown /proc/66783/task/66939/exe (readlink: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66939 io_contex ceph NOFD /proc/66783/task/66939/fd (opendir: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66940 io_contex ceph cwd unknown /proc/66783/task/66940/cwd (readlink: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66940 io_contex ceph rtd unknown /proc/66783/task/66940/root (readlink: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66940 io_contex ceph txt unknown /proc/66783/task/66940/exe (readlink: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66940 io_contex ceph NOFD /proc/66783/task/66940/fd (opendir: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66942 OpHistory ceph cwd unknown /proc/66783/task/66942/cwd (readlink: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66942 OpHistory ceph rtd unknown /proc/66783/task/66942/root (readlink: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66942 OpHistory ceph txt unknown /proc/66783/task/66942/exe (readlink: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66942 OpHistory ceph NOFD /proc/66783/task/66942/fd (opendir: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66943 ceph_time ceph cwd unknown /proc/66783/task/66943/cwd (readlink: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66943 ceph_time ceph rtd unknown /proc/66783/task/66943/root (readlink: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66943 ceph_time ceph txt unknown /proc/66783/task/66943/exe (readlink: Permission denied) 2024-03-11T20:01:36.072 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 66943 ceph_time ceph NOFD /proc/66783/task/66943/fd (opendir: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67056 ceph-osd ceph cwd unknown /proc/66783/task/67056/cwd (readlink: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67056 ceph-osd ceph rtd unknown /proc/66783/task/67056/root (readlink: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67056 ceph-osd ceph txt unknown /proc/66783/task/67056/exe (readlink: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67056 ceph-osd ceph NOFD /proc/66783/task/67056/fd (opendir: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67085 ceph-osd ceph cwd unknown /proc/66783/task/67085/cwd (readlink: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67085 ceph-osd ceph rtd unknown /proc/66783/task/67085/root (readlink: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67085 ceph-osd ceph txt unknown /proc/66783/task/67085/exe (readlink: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67085 ceph-osd ceph NOFD /proc/66783/task/67085/fd (opendir: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67103 ceph-osd ceph cwd unknown /proc/66783/task/67103/cwd (readlink: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67103 ceph-osd ceph rtd unknown /proc/66783/task/67103/root (readlink: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67103 ceph-osd ceph txt unknown /proc/66783/task/67103/exe (readlink: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67103 ceph-osd ceph NOFD /proc/66783/task/67103/fd (opendir: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67133 ceph-osd ceph cwd unknown /proc/66783/task/67133/cwd (readlink: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67133 ceph-osd ceph rtd unknown /proc/66783/task/67133/root (readlink: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67133 ceph-osd ceph txt unknown /proc/66783/task/67133/exe (readlink: Permission denied) 2024-03-11T20:01:36.073 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67133 ceph-osd ceph NOFD /proc/66783/task/67133/fd (opendir: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67143 ceph-osd ceph cwd unknown /proc/66783/task/67143/cwd (readlink: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67143 ceph-osd ceph rtd unknown /proc/66783/task/67143/root (readlink: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67143 ceph-osd ceph txt unknown /proc/66783/task/67143/exe (readlink: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67143 ceph-osd ceph NOFD /proc/66783/task/67143/fd (opendir: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67162 ceph-osd ceph cwd unknown /proc/66783/task/67162/cwd (readlink: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67162 ceph-osd ceph rtd unknown /proc/66783/task/67162/root (readlink: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67162 ceph-osd ceph txt unknown /proc/66783/task/67162/exe (readlink: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67162 ceph-osd ceph NOFD /proc/66783/task/67162/fd (opendir: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67175 ceph-osd ceph cwd unknown /proc/66783/task/67175/cwd (readlink: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67175 ceph-osd ceph rtd unknown /proc/66783/task/67175/root (readlink: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67175 ceph-osd ceph txt unknown /proc/66783/task/67175/exe (readlink: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67175 ceph-osd ceph NOFD /proc/66783/task/67175/fd (opendir: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67182 ceph-osd ceph cwd unknown /proc/66783/task/67182/cwd (readlink: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67182 ceph-osd ceph rtd unknown /proc/66783/task/67182/root (readlink: Permission denied) 2024-03-11T20:01:36.074 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67182 ceph-osd ceph txt unknown /proc/66783/task/67182/exe (readlink: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67182 ceph-osd ceph NOFD /proc/66783/task/67182/fd (opendir: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67183 ceph-osd ceph cwd unknown /proc/66783/task/67183/cwd (readlink: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67183 ceph-osd ceph rtd unknown /proc/66783/task/67183/root (readlink: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67183 ceph-osd ceph txt unknown /proc/66783/task/67183/exe (readlink: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67183 ceph-osd ceph NOFD /proc/66783/task/67183/fd (opendir: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67184 safe_time ceph cwd unknown /proc/66783/task/67184/cwd (readlink: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67184 safe_time ceph rtd unknown /proc/66783/task/67184/root (readlink: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67184 safe_time ceph txt unknown /proc/66783/task/67184/exe (readlink: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67184 safe_time ceph NOFD /proc/66783/task/67184/fd (opendir: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67185 safe_time ceph cwd unknown /proc/66783/task/67185/cwd (readlink: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67185 safe_time ceph rtd unknown /proc/66783/task/67185/root (readlink: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67185 safe_time ceph txt unknown /proc/66783/task/67185/exe (readlink: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67185 safe_time ceph NOFD /proc/66783/task/67185/fd (opendir: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67187 safe_time ceph cwd unknown /proc/66783/task/67187/cwd (readlink: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67187 safe_time ceph rtd unknown /proc/66783/task/67187/root (readlink: Permission denied) 2024-03-11T20:01:36.075 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67187 safe_time ceph txt unknown /proc/66783/task/67187/exe (readlink: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67187 safe_time ceph NOFD /proc/66783/task/67187/fd (opendir: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67188 safe_time ceph cwd unknown /proc/66783/task/67188/cwd (readlink: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67188 safe_time ceph rtd unknown /proc/66783/task/67188/root (readlink: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67188 safe_time ceph txt unknown /proc/66783/task/67188/exe (readlink: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67188 safe_time ceph NOFD /proc/66783/task/67188/fd (opendir: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67189 fn_anonym ceph cwd unknown /proc/66783/task/67189/cwd (readlink: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67189 fn_anonym ceph rtd unknown /proc/66783/task/67189/root (readlink: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67189 fn_anonym ceph txt unknown /proc/66783/task/67189/exe (readlink: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67189 fn_anonym ceph NOFD /proc/66783/task/67189/fd (opendir: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67193 rocksdb:l ceph cwd unknown /proc/66783/task/67193/cwd (readlink: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67193 rocksdb:l ceph rtd unknown /proc/66783/task/67193/root (readlink: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67193 rocksdb:l ceph txt unknown /proc/66783/task/67193/exe (readlink: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67193 rocksdb:l ceph NOFD /proc/66783/task/67193/fd (opendir: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67194 rocksdb:l ceph cwd unknown /proc/66783/task/67194/cwd (readlink: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67194 rocksdb:l ceph rtd unknown /proc/66783/task/67194/root (readlink: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67194 rocksdb:l ceph txt unknown /proc/66783/task/67194/exe (readlink: Permission denied) 2024-03-11T20:01:36.076 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67194 rocksdb:l ceph NOFD /proc/66783/task/67194/fd (opendir: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67195 rocksdb:l ceph cwd unknown /proc/66783/task/67195/cwd (readlink: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67195 rocksdb:l ceph rtd unknown /proc/66783/task/67195/root (readlink: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67195 rocksdb:l ceph txt unknown /proc/66783/task/67195/exe (readlink: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67195 rocksdb:l ceph NOFD /proc/66783/task/67195/fd (opendir: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67196 rocksdb:h ceph cwd unknown /proc/66783/task/67196/cwd (readlink: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67196 rocksdb:h ceph rtd unknown /proc/66783/task/67196/root (readlink: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67196 rocksdb:h ceph txt unknown /proc/66783/task/67196/exe (readlink: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67196 rocksdb:h ceph NOFD /proc/66783/task/67196/fd (opendir: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67627 bstore_ai ceph cwd unknown /proc/66783/task/67627/cwd (readlink: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67627 bstore_ai ceph rtd unknown /proc/66783/task/67627/root (readlink: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67627 bstore_ai ceph txt unknown /proc/66783/task/67627/exe (readlink: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67627 bstore_ai ceph NOFD /proc/66783/task/67627/fd (opendir: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67841 bstore_ai ceph cwd unknown /proc/66783/task/67841/cwd (readlink: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67841 bstore_ai ceph rtd unknown /proc/66783/task/67841/root (readlink: Permission denied) 2024-03-11T20:01:36.077 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67841 bstore_ai ceph txt unknown /proc/66783/task/67841/exe (readlink: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 67841 bstore_ai ceph NOFD /proc/66783/task/67841/fd (opendir: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68034 ceph-osd ceph cwd unknown /proc/66783/task/68034/cwd (readlink: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68034 ceph-osd ceph rtd unknown /proc/66783/task/68034/root (readlink: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68034 ceph-osd ceph txt unknown /proc/66783/task/68034/exe (readlink: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68034 ceph-osd ceph NOFD /proc/66783/task/68034/fd (opendir: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68035 cfin ceph cwd unknown /proc/66783/task/68035/cwd (readlink: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68035 cfin ceph rtd unknown /proc/66783/task/68035/root (readlink: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68035 cfin ceph txt unknown /proc/66783/task/68035/exe (readlink: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68035 cfin ceph NOFD /proc/66783/task/68035/fd (opendir: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68036 bstore_kv ceph cwd unknown /proc/66783/task/68036/cwd (readlink: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68036 bstore_kv ceph rtd unknown /proc/66783/task/68036/root (readlink: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68036 bstore_kv ceph txt unknown /proc/66783/task/68036/exe (readlink: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68036 bstore_kv ceph NOFD /proc/66783/task/68036/fd (opendir: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68037 bstore_kv ceph cwd unknown /proc/66783/task/68037/cwd (readlink: Permission denied) 2024-03-11T20:01:36.078 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68037 bstore_kv ceph rtd unknown /proc/66783/task/68037/root (readlink: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68037 bstore_kv ceph txt unknown /proc/66783/task/68037/exe (readlink: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68037 bstore_kv ceph NOFD /proc/66783/task/68037/fd (opendir: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68038 bstore_me ceph cwd unknown /proc/66783/task/68038/cwd (readlink: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68038 bstore_me ceph rtd unknown /proc/66783/task/68038/root (readlink: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68038 bstore_me ceph txt unknown /proc/66783/task/68038/exe (readlink: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68038 bstore_me ceph NOFD /proc/66783/task/68038/fd (opendir: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68045 ms_dispat ceph cwd unknown /proc/66783/task/68045/cwd (readlink: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68045 ms_dispat ceph rtd unknown /proc/66783/task/68045/root (readlink: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68045 ms_dispat ceph txt unknown /proc/66783/task/68045/exe (readlink: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68045 ms_dispat ceph NOFD /proc/66783/task/68045/fd (opendir: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68046 ms_local ceph cwd unknown /proc/66783/task/68046/cwd (readlink: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68046 ms_local ceph rtd unknown /proc/66783/task/68046/root (readlink: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68046 ms_local ceph txt unknown /proc/66783/task/68046/exe (readlink: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68046 ms_local ceph NOFD /proc/66783/task/68046/fd (opendir: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68047 safe_time ceph cwd unknown /proc/66783/task/68047/cwd (readlink: Permission denied) 2024-03-11T20:01:36.079 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68047 safe_time ceph rtd unknown /proc/66783/task/68047/root (readlink: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68047 safe_time ceph txt unknown /proc/66783/task/68047/exe (readlink: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68047 safe_time ceph NOFD /proc/66783/task/68047/fd (opendir: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68048 safe_time ceph cwd unknown /proc/66783/task/68048/cwd (readlink: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68048 safe_time ceph rtd unknown /proc/66783/task/68048/root (readlink: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68048 safe_time ceph txt unknown /proc/66783/task/68048/exe (readlink: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68048 safe_time ceph NOFD /proc/66783/task/68048/fd (opendir: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68049 ms_dispat ceph cwd unknown /proc/66783/task/68049/cwd (readlink: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68049 ms_dispat ceph rtd unknown /proc/66783/task/68049/root (readlink: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68049 ms_dispat ceph txt unknown /proc/66783/task/68049/exe (readlink: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68049 ms_dispat ceph NOFD /proc/66783/task/68049/fd (opendir: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68050 ms_local ceph cwd unknown /proc/66783/task/68050/cwd (readlink: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68050 ms_local ceph rtd unknown /proc/66783/task/68050/root (readlink: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68050 ms_local ceph txt unknown /proc/66783/task/68050/exe (readlink: Permission denied) 2024-03-11T20:01:36.080 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68050 ms_local ceph NOFD /proc/66783/task/68050/fd (opendir: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68051 ms_dispat ceph cwd unknown /proc/66783/task/68051/cwd (readlink: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68051 ms_dispat ceph rtd unknown /proc/66783/task/68051/root (readlink: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68051 ms_dispat ceph txt unknown /proc/66783/task/68051/exe (readlink: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68051 ms_dispat ceph NOFD /proc/66783/task/68051/fd (opendir: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68052 ms_local ceph cwd unknown /proc/66783/task/68052/cwd (readlink: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68052 ms_local ceph rtd unknown /proc/66783/task/68052/root (readlink: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68052 ms_local ceph txt unknown /proc/66783/task/68052/exe (readlink: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68052 ms_local ceph NOFD /proc/66783/task/68052/fd (opendir: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68053 ms_dispat ceph cwd unknown /proc/66783/task/68053/cwd (readlink: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68053 ms_dispat ceph rtd unknown /proc/66783/task/68053/root (readlink: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68053 ms_dispat ceph txt unknown /proc/66783/task/68053/exe (readlink: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68053 ms_dispat ceph NOFD /proc/66783/task/68053/fd (opendir: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68054 ms_local ceph cwd unknown /proc/66783/task/68054/cwd (readlink: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68054 ms_local ceph rtd unknown /proc/66783/task/68054/root (readlink: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68054 ms_local ceph txt unknown /proc/66783/task/68054/exe (readlink: Permission denied) 2024-03-11T20:01:36.081 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68054 ms_local ceph NOFD /proc/66783/task/68054/fd (opendir: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68055 ms_dispat ceph cwd unknown /proc/66783/task/68055/cwd (readlink: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68055 ms_dispat ceph rtd unknown /proc/66783/task/68055/root (readlink: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68055 ms_dispat ceph txt unknown /proc/66783/task/68055/exe (readlink: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68055 ms_dispat ceph NOFD /proc/66783/task/68055/fd (opendir: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68056 ms_local ceph cwd unknown /proc/66783/task/68056/cwd (readlink: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68056 ms_local ceph rtd unknown /proc/66783/task/68056/root (readlink: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68056 ms_local ceph txt unknown /proc/66783/task/68056/exe (readlink: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68056 ms_local ceph NOFD /proc/66783/task/68056/fd (opendir: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68057 ms_dispat ceph cwd unknown /proc/66783/task/68057/cwd (readlink: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68057 ms_dispat ceph rtd unknown /proc/66783/task/68057/root (readlink: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68057 ms_dispat ceph txt unknown /proc/66783/task/68057/exe (readlink: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68057 ms_dispat ceph NOFD /proc/66783/task/68057/fd (opendir: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68058 ms_local ceph cwd unknown /proc/66783/task/68058/cwd (readlink: Permission denied) 2024-03-11T20:01:36.082 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68058 ms_local ceph rtd unknown /proc/66783/task/68058/root (readlink: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68058 ms_local ceph txt unknown /proc/66783/task/68058/exe (readlink: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68058 ms_local ceph NOFD /proc/66783/task/68058/fd (opendir: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68059 ms_dispat ceph cwd unknown /proc/66783/task/68059/cwd (readlink: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68059 ms_dispat ceph rtd unknown /proc/66783/task/68059/root (readlink: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68059 ms_dispat ceph txt unknown /proc/66783/task/68059/exe (readlink: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68059 ms_dispat ceph NOFD /proc/66783/task/68059/fd (opendir: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68060 ms_local ceph cwd unknown /proc/66783/task/68060/cwd (readlink: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68060 ms_local ceph rtd unknown /proc/66783/task/68060/root (readlink: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68060 ms_local ceph txt unknown /proc/66783/task/68060/exe (readlink: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68060 ms_local ceph NOFD /proc/66783/task/68060/fd (opendir: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68061 fn_anonym ceph cwd unknown /proc/66783/task/68061/cwd (readlink: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68061 fn_anonym ceph rtd unknown /proc/66783/task/68061/root (readlink: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68061 fn_anonym ceph txt unknown /proc/66783/task/68061/exe (readlink: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68061 fn_anonym ceph NOFD /proc/66783/task/68061/fd (opendir: Permission denied) 2024-03-11T20:01:36.083 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68062 finisher ceph cwd unknown /proc/66783/task/68062/cwd (readlink: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68062 finisher ceph rtd unknown /proc/66783/task/68062/root (readlink: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68062 finisher ceph txt unknown /proc/66783/task/68062/exe (readlink: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68062 finisher ceph NOFD /proc/66783/task/68062/fd (opendir: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68063 safe_time ceph cwd unknown /proc/66783/task/68063/cwd (readlink: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68063 safe_time ceph rtd unknown /proc/66783/task/68063/root (readlink: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68063 safe_time ceph txt unknown /proc/66783/task/68063/exe (readlink: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68063 safe_time ceph NOFD /proc/66783/task/68063/fd (opendir: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68064 safe_time ceph cwd unknown /proc/66783/task/68064/cwd (readlink: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68064 safe_time ceph rtd unknown /proc/66783/task/68064/root (readlink: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68064 safe_time ceph txt unknown /proc/66783/task/68064/exe (readlink: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68064 safe_time ceph NOFD /proc/66783/task/68064/fd (opendir: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68065 ceph-osd ceph cwd unknown /proc/66783/task/68065/cwd (readlink: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68065 ceph-osd ceph rtd unknown /proc/66783/task/68065/root (readlink: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68065 ceph-osd ceph txt unknown /proc/66783/task/68065/exe (readlink: Permission denied) 2024-03-11T20:01:36.084 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68065 ceph-osd ceph NOFD /proc/66783/task/68065/fd (opendir: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68066 osd_srv_a ceph cwd unknown /proc/66783/task/68066/cwd (readlink: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68066 osd_srv_a ceph rtd unknown /proc/66783/task/68066/root (readlink: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68066 osd_srv_a ceph txt unknown /proc/66783/task/68066/exe (readlink: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68066 osd_srv_a ceph NOFD /proc/66783/task/68066/fd (opendir: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68067 tp_osd_tp ceph cwd unknown /proc/66783/task/68067/cwd (readlink: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68067 tp_osd_tp ceph rtd unknown /proc/66783/task/68067/root (readlink: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68067 tp_osd_tp ceph txt unknown /proc/66783/task/68067/exe (readlink: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68067 tp_osd_tp ceph NOFD /proc/66783/task/68067/fd (opendir: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68068 tp_osd_tp ceph cwd unknown /proc/66783/task/68068/cwd (readlink: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68068 tp_osd_tp ceph rtd unknown /proc/66783/task/68068/root (readlink: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68068 tp_osd_tp ceph txt unknown /proc/66783/task/68068/exe (readlink: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68068 tp_osd_tp ceph NOFD /proc/66783/task/68068/fd (opendir: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68069 tp_osd_tp ceph cwd unknown /proc/66783/task/68069/cwd (readlink: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68069 tp_osd_tp ceph rtd unknown /proc/66783/task/68069/root (readlink: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68069 tp_osd_tp ceph txt unknown /proc/66783/task/68069/exe (readlink: Permission denied) 2024-03-11T20:01:36.085 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68069 tp_osd_tp ceph NOFD /proc/66783/task/68069/fd (opendir: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68070 tp_osd_tp ceph cwd unknown /proc/66783/task/68070/cwd (readlink: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68070 tp_osd_tp ceph rtd unknown /proc/66783/task/68070/root (readlink: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68070 tp_osd_tp ceph txt unknown /proc/66783/task/68070/exe (readlink: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68070 tp_osd_tp ceph NOFD /proc/66783/task/68070/fd (opendir: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68071 tp_osd_tp ceph cwd unknown /proc/66783/task/68071/cwd (readlink: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68071 tp_osd_tp ceph rtd unknown /proc/66783/task/68071/root (readlink: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68071 tp_osd_tp ceph txt unknown /proc/66783/task/68071/exe (readlink: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68071 tp_osd_tp ceph NOFD /proc/66783/task/68071/fd (opendir: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68072 tp_osd_tp ceph cwd unknown /proc/66783/task/68072/cwd (readlink: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68072 tp_osd_tp ceph rtd unknown /proc/66783/task/68072/root (readlink: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68072 tp_osd_tp ceph txt unknown /proc/66783/task/68072/exe (readlink: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68072 tp_osd_tp ceph NOFD /proc/66783/task/68072/fd (opendir: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68073 tp_osd_tp ceph cwd unknown /proc/66783/task/68073/cwd (readlink: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68073 tp_osd_tp ceph rtd unknown /proc/66783/task/68073/root (readlink: Permission denied) 2024-03-11T20:01:36.086 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68073 tp_osd_tp ceph txt unknown /proc/66783/task/68073/exe (readlink: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68073 tp_osd_tp ceph NOFD /proc/66783/task/68073/fd (opendir: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68074 tp_osd_tp ceph cwd unknown /proc/66783/task/68074/cwd (readlink: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68074 tp_osd_tp ceph rtd unknown /proc/66783/task/68074/root (readlink: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68074 tp_osd_tp ceph txt unknown /proc/66783/task/68074/exe (readlink: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68074 tp_osd_tp ceph NOFD /proc/66783/task/68074/fd (opendir: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68075 tp_osd_tp ceph cwd unknown /proc/66783/task/68075/cwd (readlink: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68075 tp_osd_tp ceph rtd unknown /proc/66783/task/68075/root (readlink: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68075 tp_osd_tp ceph txt unknown /proc/66783/task/68075/exe (readlink: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68075 tp_osd_tp ceph NOFD /proc/66783/task/68075/fd (opendir: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68076 tp_osd_tp ceph cwd unknown /proc/66783/task/68076/cwd (readlink: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68076 tp_osd_tp ceph rtd unknown /proc/66783/task/68076/root (readlink: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68076 tp_osd_tp ceph txt unknown /proc/66783/task/68076/exe (readlink: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68076 tp_osd_tp ceph NOFD /proc/66783/task/68076/fd (opendir: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68077 tp_osd_tp ceph cwd unknown /proc/66783/task/68077/cwd (readlink: Permission denied) 2024-03-11T20:01:36.087 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68077 tp_osd_tp ceph rtd unknown /proc/66783/task/68077/root (readlink: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68077 tp_osd_tp ceph txt unknown /proc/66783/task/68077/exe (readlink: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68077 tp_osd_tp ceph NOFD /proc/66783/task/68077/fd (opendir: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68078 tp_osd_tp ceph cwd unknown /proc/66783/task/68078/cwd (readlink: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68078 tp_osd_tp ceph rtd unknown /proc/66783/task/68078/root (readlink: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68078 tp_osd_tp ceph txt unknown /proc/66783/task/68078/exe (readlink: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68078 tp_osd_tp ceph NOFD /proc/66783/task/68078/fd (opendir: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68079 tp_osd_tp ceph cwd unknown /proc/66783/task/68079/cwd (readlink: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68079 tp_osd_tp ceph rtd unknown /proc/66783/task/68079/root (readlink: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68079 tp_osd_tp ceph txt unknown /proc/66783/task/68079/exe (readlink: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68079 tp_osd_tp ceph NOFD /proc/66783/task/68079/fd (opendir: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68080 tp_osd_tp ceph cwd unknown /proc/66783/task/68080/cwd (readlink: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68080 tp_osd_tp ceph rtd unknown /proc/66783/task/68080/root (readlink: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68080 tp_osd_tp ceph txt unknown /proc/66783/task/68080/exe (readlink: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68080 tp_osd_tp ceph NOFD /proc/66783/task/68080/fd (opendir: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68081 tp_osd_tp ceph cwd unknown /proc/66783/task/68081/cwd (readlink: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68081 tp_osd_tp ceph rtd unknown /proc/66783/task/68081/root (readlink: Permission denied) 2024-03-11T20:01:36.088 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68081 tp_osd_tp ceph txt unknown /proc/66783/task/68081/exe (readlink: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68081 tp_osd_tp ceph NOFD /proc/66783/task/68081/fd (opendir: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68082 tp_osd_tp ceph cwd unknown /proc/66783/task/68082/cwd (readlink: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68082 tp_osd_tp ceph rtd unknown /proc/66783/task/68082/root (readlink: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68082 tp_osd_tp ceph txt unknown /proc/66783/task/68082/exe (readlink: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68082 tp_osd_tp ceph NOFD /proc/66783/task/68082/fd (opendir: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68083 osd_srv_h ceph cwd unknown /proc/66783/task/68083/cwd (readlink: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68083 osd_srv_h ceph rtd unknown /proc/66783/task/68083/root (readlink: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68083 osd_srv_h ceph txt unknown /proc/66783/task/68083/exe (readlink: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-osd 66783 68083 osd_srv_h ceph NOFD /proc/66783/task/68083/fd (opendir: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 68317 root cwd unknown /proc/68317/cwd (readlink: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 68317 root rtd unknown /proc/68317/root (readlink: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 68317 root txt unknown /proc/68317/exe (readlink: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 68317 root NOFD /proc/68317/fd (opendir: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 68344 root cwd unknown /proc/68344/cwd (readlink: Permission denied) 2024-03-11T20:01:36.089 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 68344 root rtd unknown /proc/68344/root (readlink: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 68344 root txt unknown /proc/68344/exe (readlink: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 68344 root NOFD /proc/68344/fd (opendir: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 71493 root cwd unknown /proc/71493/cwd (readlink: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 71493 root rtd unknown /proc/71493/root (readlink: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 71493 root txt unknown /proc/71493/exe (readlink: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 71493 root NOFD /proc/71493/fd (opendir: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 71656 root cwd unknown /proc/71656/cwd (readlink: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 71656 root rtd unknown /proc/71656/root (readlink: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 71656 root txt unknown /proc/71656/exe (readlink: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 71656 root NOFD /proc/71656/fd (opendir: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71786 root cwd unknown /proc/71786/cwd (readlink: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71786 root rtd unknown /proc/71786/root (readlink: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71786 root txt unknown /proc/71786/exe (readlink: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71786 root NOFD /proc/71786/fd (opendir: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71809 root cwd unknown /proc/71809/cwd (readlink: Permission denied) 2024-03-11T20:01:36.090 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71809 root rtd unknown /proc/71809/root (readlink: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71809 root txt unknown /proc/71809/exe (readlink: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71809 root NOFD /proc/71809/fd (opendir: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71830 root cwd unknown /proc/71830/cwd (readlink: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71830 root rtd unknown /proc/71830/root (readlink: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71830 root txt unknown /proc/71830/exe (readlink: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71830 root NOFD /proc/71830/fd (opendir: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71835 root cwd unknown /proc/71835/cwd (readlink: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71835 root rtd unknown /proc/71835/root (readlink: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71835 root txt unknown /proc/71835/exe (readlink: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71835 root NOFD /proc/71835/fd (opendir: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71843 root cwd unknown /proc/71843/cwd (readlink: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71843 root rtd unknown /proc/71843/root (readlink: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71843 root txt unknown /proc/71843/exe (readlink: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71843 root NOFD /proc/71843/fd (opendir: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71853 root cwd unknown /proc/71853/cwd (readlink: Permission denied) 2024-03-11T20:01:36.091 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71853 root rtd unknown /proc/71853/root (readlink: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71853 root txt unknown /proc/71853/exe (readlink: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 71853 root NOFD /proc/71853/fd (opendir: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71862 root cwd unknown /proc/71862/cwd (readlink: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71862 root rtd unknown /proc/71862/root (readlink: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71862 root txt unknown /proc/71862/exe (readlink: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71862 root NOFD /proc/71862/fd (opendir: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71875 root cwd unknown /proc/71875/cwd (readlink: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71875 root rtd unknown /proc/71875/root (readlink: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71875 root txt unknown /proc/71875/exe (readlink: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:journalct 71875 root NOFD /proc/71875/fd (opendir: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 72416 root cwd unknown /proc/72416/cwd (readlink: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 72416 root rtd unknown /proc/72416/root (readlink: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 72416 root txt unknown /proc/72416/exe (readlink: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 72416 root NOFD /proc/72416/fd (opendir: Permission denied) 2024-03-11T20:01:36.092 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 72418 root cwd unknown /proc/72418/cwd (readlink: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 72418 root rtd unknown /proc/72418/root (readlink: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 72418 root txt unknown /proc/72418/exe (readlink: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 72418 root NOFD /proc/72418/fd (opendir: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 ceph cwd unknown /proc/72420/cwd (readlink: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 ceph rtd unknown /proc/72420/root (readlink: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 ceph txt unknown /proc/72420/exe (readlink: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 ceph NOFD /proc/72420/fd (opendir: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72421 log ceph cwd unknown /proc/72420/task/72421/cwd (readlink: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72421 log ceph rtd unknown /proc/72420/task/72421/root (readlink: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72421 log ceph txt unknown /proc/72420/task/72421/exe (readlink: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72421 log ceph NOFD /proc/72420/task/72421/fd (opendir: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72423 msgr-work ceph cwd unknown /proc/72420/task/72423/cwd (readlink: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72423 msgr-work ceph rtd unknown /proc/72420/task/72423/root (readlink: Permission denied) 2024-03-11T20:01:36.093 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72423 msgr-work ceph txt unknown /proc/72420/task/72423/exe (readlink: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72423 msgr-work ceph NOFD /proc/72420/task/72423/fd (opendir: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72424 msgr-work ceph cwd unknown /proc/72420/task/72424/cwd (readlink: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72424 msgr-work ceph rtd unknown /proc/72420/task/72424/root (readlink: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72424 msgr-work ceph txt unknown /proc/72420/task/72424/exe (readlink: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72424 msgr-work ceph NOFD /proc/72420/task/72424/fd (opendir: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72425 msgr-work ceph cwd unknown /proc/72420/task/72425/cwd (readlink: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72425 msgr-work ceph rtd unknown /proc/72420/task/72425/root (readlink: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72425 msgr-work ceph txt unknown /proc/72420/task/72425/exe (readlink: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72425 msgr-work ceph NOFD /proc/72420/task/72425/fd (opendir: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72429 service ceph cwd unknown /proc/72420/task/72429/cwd (readlink: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72429 service ceph rtd unknown /proc/72420/task/72429/root (readlink: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72429 service ceph txt unknown /proc/72420/task/72429/exe (readlink: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72429 service ceph NOFD /proc/72420/task/72429/fd (opendir: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72430 admin_soc ceph cwd unknown /proc/72420/task/72430/cwd (readlink: Permission denied) 2024-03-11T20:01:36.094 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72430 admin_soc ceph rtd unknown /proc/72420/task/72430/root (readlink: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72430 admin_soc ceph txt unknown /proc/72420/task/72430/exe (readlink: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72430 admin_soc ceph NOFD /proc/72420/task/72430/fd (opendir: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72431 signal_ha ceph cwd unknown /proc/72420/task/72431/cwd (readlink: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72431 signal_ha ceph rtd unknown /proc/72420/task/72431/root (readlink: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72431 signal_ha ceph txt unknown /proc/72420/task/72431/exe (readlink: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72431 signal_ha ceph NOFD /proc/72420/task/72431/fd (opendir: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72432 io_contex ceph cwd unknown /proc/72420/task/72432/cwd (readlink: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72432 io_contex ceph rtd unknown /proc/72420/task/72432/root (readlink: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72432 io_contex ceph txt unknown /proc/72420/task/72432/exe (readlink: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72432 io_contex ceph NOFD /proc/72420/task/72432/fd (opendir: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72433 io_contex ceph cwd unknown /proc/72420/task/72433/cwd (readlink: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72433 io_contex ceph rtd unknown /proc/72420/task/72433/root (readlink: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72433 io_contex ceph txt unknown /proc/72420/task/72433/exe (readlink: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72433 io_contex ceph NOFD /proc/72420/task/72433/fd (opendir: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72434 ms_dispat ceph cwd unknown /proc/72420/task/72434/cwd (readlink: Permission denied) 2024-03-11T20:01:36.095 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72434 ms_dispat ceph rtd unknown /proc/72420/task/72434/root (readlink: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72434 ms_dispat ceph txt unknown /proc/72420/task/72434/exe (readlink: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72434 ms_dispat ceph NOFD /proc/72420/task/72434/fd (opendir: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72435 ms_local ceph cwd unknown /proc/72420/task/72435/cwd (readlink: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72435 ms_local ceph rtd unknown /proc/72420/task/72435/root (readlink: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72435 ms_local ceph txt unknown /proc/72420/task/72435/exe (readlink: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72435 ms_local ceph NOFD /proc/72420/task/72435/fd (opendir: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72436 safe_time ceph cwd unknown /proc/72420/task/72436/cwd (readlink: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72436 safe_time ceph rtd unknown /proc/72420/task/72436/root (readlink: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72436 safe_time ceph txt unknown /proc/72420/task/72436/exe (readlink: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72436 safe_time ceph NOFD /proc/72420/task/72436/fd (opendir: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72437 safe_time ceph cwd unknown /proc/72420/task/72437/cwd (readlink: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72437 safe_time ceph rtd unknown /proc/72420/task/72437/root (readlink: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72437 safe_time ceph txt unknown /proc/72420/task/72437/exe (readlink: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72437 safe_time ceph NOFD /proc/72420/task/72437/fd (opendir: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72438 ceph-mds ceph cwd unknown /proc/72420/task/72438/cwd (readlink: Permission denied) 2024-03-11T20:01:36.096 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72438 ceph-mds ceph rtd unknown /proc/72420/task/72438/root (readlink: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72438 ceph-mds ceph txt unknown /proc/72420/task/72438/exe (readlink: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72438 ceph-mds ceph NOFD /proc/72420/task/72438/fd (opendir: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72491 safe_time ceph cwd unknown /proc/72420/task/72491/cwd (readlink: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72491 safe_time ceph rtd unknown /proc/72420/task/72491/root (readlink: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72491 safe_time ceph txt unknown /proc/72420/task/72491/exe (readlink: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72420 72491 safe_time ceph NOFD /proc/72420/task/72491/fd (opendir: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 72782 root cwd unknown /proc/72782/cwd (readlink: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 72782 root rtd unknown /proc/72782/root (readlink: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 72782 root txt unknown /proc/72782/exe (readlink: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 72782 root NOFD /proc/72782/fd (opendir: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 72785 root cwd unknown /proc/72785/cwd (readlink: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 72785 root rtd unknown /proc/72785/root (readlink: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 72785 root txt unknown /proc/72785/exe (readlink: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 72785 root NOFD /proc/72785/fd (opendir: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 ceph cwd unknown /proc/72810/cwd (readlink: Permission denied) 2024-03-11T20:01:36.097 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 ceph rtd unknown /proc/72810/root (readlink: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 ceph txt unknown /proc/72810/exe (readlink: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 ceph NOFD /proc/72810/fd (opendir: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72811 log ceph cwd unknown /proc/72810/task/72811/cwd (readlink: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72811 log ceph rtd unknown /proc/72810/task/72811/root (readlink: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72811 log ceph txt unknown /proc/72810/task/72811/exe (readlink: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72811 log ceph NOFD /proc/72810/task/72811/fd (opendir: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72813 msgr-work ceph cwd unknown /proc/72810/task/72813/cwd (readlink: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72813 msgr-work ceph rtd unknown /proc/72810/task/72813/root (readlink: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72813 msgr-work ceph txt unknown /proc/72810/task/72813/exe (readlink: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72813 msgr-work ceph NOFD /proc/72810/task/72813/fd (opendir: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72814 msgr-work ceph cwd unknown /proc/72810/task/72814/cwd (readlink: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72814 msgr-work ceph rtd unknown /proc/72810/task/72814/root (readlink: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72814 msgr-work ceph txt unknown /proc/72810/task/72814/exe (readlink: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72814 msgr-work ceph NOFD /proc/72810/task/72814/fd (opendir: Permission denied) 2024-03-11T20:01:36.098 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72815 msgr-work ceph cwd unknown /proc/72810/task/72815/cwd (readlink: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72815 msgr-work ceph rtd unknown /proc/72810/task/72815/root (readlink: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72815 msgr-work ceph txt unknown /proc/72810/task/72815/exe (readlink: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72815 msgr-work ceph NOFD /proc/72810/task/72815/fd (opendir: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72819 service ceph cwd unknown /proc/72810/task/72819/cwd (readlink: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72819 service ceph rtd unknown /proc/72810/task/72819/root (readlink: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72819 service ceph txt unknown /proc/72810/task/72819/exe (readlink: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72819 service ceph NOFD /proc/72810/task/72819/fd (opendir: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72820 admin_soc ceph cwd unknown /proc/72810/task/72820/cwd (readlink: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72820 admin_soc ceph rtd unknown /proc/72810/task/72820/root (readlink: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72820 admin_soc ceph txt unknown /proc/72810/task/72820/exe (readlink: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72820 admin_soc ceph NOFD /proc/72810/task/72820/fd (opendir: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72821 signal_ha ceph cwd unknown /proc/72810/task/72821/cwd (readlink: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72821 signal_ha ceph rtd unknown /proc/72810/task/72821/root (readlink: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72821 signal_ha ceph txt unknown /proc/72810/task/72821/exe (readlink: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72821 signal_ha ceph NOFD /proc/72810/task/72821/fd (opendir: Permission denied) 2024-03-11T20:01:36.099 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72822 io_contex ceph cwd unknown /proc/72810/task/72822/cwd (readlink: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72822 io_contex ceph rtd unknown /proc/72810/task/72822/root (readlink: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72822 io_contex ceph txt unknown /proc/72810/task/72822/exe (readlink: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72822 io_contex ceph NOFD /proc/72810/task/72822/fd (opendir: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72823 io_contex ceph cwd unknown /proc/72810/task/72823/cwd (readlink: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72823 io_contex ceph rtd unknown /proc/72810/task/72823/root (readlink: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72823 io_contex ceph txt unknown /proc/72810/task/72823/exe (readlink: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72823 io_contex ceph NOFD /proc/72810/task/72823/fd (opendir: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72824 ms_dispat ceph cwd unknown /proc/72810/task/72824/cwd (readlink: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72824 ms_dispat ceph rtd unknown /proc/72810/task/72824/root (readlink: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72824 ms_dispat ceph txt unknown /proc/72810/task/72824/exe (readlink: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72824 ms_dispat ceph NOFD /proc/72810/task/72824/fd (opendir: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72825 ms_local ceph cwd unknown /proc/72810/task/72825/cwd (readlink: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72825 ms_local ceph rtd unknown /proc/72810/task/72825/root (readlink: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72825 ms_local ceph txt unknown /proc/72810/task/72825/exe (readlink: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72825 ms_local ceph NOFD /proc/72810/task/72825/fd (opendir: Permission denied) 2024-03-11T20:01:36.100 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72826 safe_time ceph cwd unknown /proc/72810/task/72826/cwd (readlink: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72826 safe_time ceph rtd unknown /proc/72810/task/72826/root (readlink: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72826 safe_time ceph txt unknown /proc/72810/task/72826/exe (readlink: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72826 safe_time ceph NOFD /proc/72810/task/72826/fd (opendir: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72827 safe_time ceph cwd unknown /proc/72810/task/72827/cwd (readlink: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72827 safe_time ceph rtd unknown /proc/72810/task/72827/root (readlink: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72827 safe_time ceph txt unknown /proc/72810/task/72827/exe (readlink: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72827 safe_time ceph NOFD /proc/72810/task/72827/fd (opendir: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72828 ceph-mds ceph cwd unknown /proc/72810/task/72828/cwd (readlink: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72828 ceph-mds ceph rtd unknown /proc/72810/task/72828/root (readlink: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72828 ceph-mds ceph txt unknown /proc/72810/task/72828/exe (readlink: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72828 ceph-mds ceph NOFD /proc/72810/task/72828/fd (opendir: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72835 safe_time ceph cwd unknown /proc/72810/task/72835/cwd (readlink: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72835 safe_time ceph rtd unknown /proc/72810/task/72835/root (readlink: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72835 safe_time ceph txt unknown /proc/72810/task/72835/exe (readlink: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 72835 safe_time ceph NOFD /proc/72810/task/72835/fd (opendir: Permission denied) 2024-03-11T20:01:36.101 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76928 ceph_time ceph cwd unknown /proc/72810/task/76928/cwd (readlink: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76928 ceph_time ceph rtd unknown /proc/72810/task/76928/root (readlink: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76928 ceph_time ceph txt unknown /proc/72810/task/76928/exe (readlink: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76928 ceph_time ceph NOFD /proc/72810/task/76928/fd (opendir: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76929 OpHistory ceph cwd unknown /proc/72810/task/76929/cwd (readlink: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76929 OpHistory ceph rtd unknown /proc/72810/task/76929/root (readlink: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76929 OpHistory ceph txt unknown /proc/72810/task/76929/exe (readlink: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76929 OpHistory ceph NOFD /proc/72810/task/76929/fd (opendir: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76930 ms_dispat ceph cwd unknown /proc/72810/task/76930/cwd (readlink: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76930 ms_dispat ceph rtd unknown /proc/72810/task/76930/root (readlink: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76930 ms_dispat ceph txt unknown /proc/72810/task/76930/exe (readlink: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76930 ms_dispat ceph NOFD /proc/72810/task/76930/fd (opendir: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76931 ms_dispat ceph cwd unknown /proc/72810/task/76931/cwd (readlink: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76931 ms_dispat ceph rtd unknown /proc/72810/task/76931/root (readlink: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76931 ms_dispat ceph txt unknown /proc/72810/task/76931/exe (readlink: Permission denied) 2024-03-11T20:01:36.102 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76931 ms_dispat ceph NOFD /proc/72810/task/76931/fd (opendir: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76932 mds_rank_ ceph cwd unknown /proc/72810/task/76932/cwd (readlink: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76932 mds_rank_ ceph rtd unknown /proc/72810/task/76932/root (readlink: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76932 mds_rank_ ceph txt unknown /proc/72810/task/76932/exe (readlink: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76932 mds_rank_ ceph NOFD /proc/72810/task/76932/fd (opendir: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76933 PQ_Finish ceph cwd unknown /proc/72810/task/76933/cwd (readlink: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76933 PQ_Finish ceph rtd unknown /proc/72810/task/76933/root (readlink: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76933 PQ_Finish ceph txt unknown /proc/72810/task/76933/exe (readlink: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76933 PQ_Finish ceph NOFD /proc/72810/task/76933/fd (opendir: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76934 safe_time ceph cwd unknown /proc/72810/task/76934/cwd (readlink: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76934 safe_time ceph rtd unknown /proc/72810/task/76934/root (readlink: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76934 safe_time ceph txt unknown /proc/72810/task/76934/exe (readlink: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76934 safe_time ceph NOFD /proc/72810/task/76934/fd (opendir: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76935 MR_Finish ceph cwd unknown /proc/72810/task/76935/cwd (readlink: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76935 MR_Finish ceph rtd unknown /proc/72810/task/76935/root (readlink: Permission denied) 2024-03-11T20:01:36.103 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76935 MR_Finish ceph txt unknown /proc/72810/task/76935/exe (readlink: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76935 MR_Finish ceph NOFD /proc/72810/task/76935/fd (opendir: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76937 md_submit ceph cwd unknown /proc/72810/task/76937/cwd (readlink: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76937 md_submit ceph rtd unknown /proc/72810/task/76937/root (readlink: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76937 md_submit ceph txt unknown /proc/72810/task/76937/exe (readlink: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 72810 76937 md_submit ceph NOFD /proc/72810/task/76937/fd (opendir: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 73041 root cwd unknown /proc/73041/cwd (readlink: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 73041 root rtd unknown /proc/73041/root (readlink: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 73041 root txt unknown /proc/73041/exe (readlink: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 73041 root NOFD /proc/73041/fd (opendir: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 73399 root cwd unknown /proc/73399/cwd (readlink: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 73399 root rtd unknown /proc/73399/root (readlink: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 73399 root txt unknown /proc/73399/exe (readlink: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 73399 root NOFD /proc/73399/fd (opendir: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 73401 root cwd unknown /proc/73401/cwd (readlink: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 73401 root rtd unknown /proc/73401/root (readlink: Permission denied) 2024-03-11T20:01:36.104 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 73401 root txt unknown /proc/73401/exe (readlink: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 73401 root NOFD /proc/73401/fd (opendir: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 ceph cwd unknown /proc/73403/cwd (readlink: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 ceph rtd unknown /proc/73403/root (readlink: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 ceph txt unknown /proc/73403/exe (readlink: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 ceph NOFD /proc/73403/fd (opendir: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73404 log ceph cwd unknown /proc/73403/task/73404/cwd (readlink: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73404 log ceph rtd unknown /proc/73403/task/73404/root (readlink: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73404 log ceph txt unknown /proc/73403/task/73404/exe (readlink: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73404 log ceph NOFD /proc/73403/task/73404/fd (opendir: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73406 msgr-work ceph cwd unknown /proc/73403/task/73406/cwd (readlink: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73406 msgr-work ceph rtd unknown /proc/73403/task/73406/root (readlink: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73406 msgr-work ceph txt unknown /proc/73403/task/73406/exe (readlink: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73406 msgr-work ceph NOFD /proc/73403/task/73406/fd (opendir: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73407 msgr-work ceph cwd unknown /proc/73403/task/73407/cwd (readlink: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73407 msgr-work ceph rtd unknown /proc/73403/task/73407/root (readlink: Permission denied) 2024-03-11T20:01:36.105 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73407 msgr-work ceph txt unknown /proc/73403/task/73407/exe (readlink: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73407 msgr-work ceph NOFD /proc/73403/task/73407/fd (opendir: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73408 msgr-work ceph cwd unknown /proc/73403/task/73408/cwd (readlink: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73408 msgr-work ceph rtd unknown /proc/73403/task/73408/root (readlink: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73408 msgr-work ceph txt unknown /proc/73403/task/73408/exe (readlink: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73408 msgr-work ceph NOFD /proc/73403/task/73408/fd (opendir: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73412 service ceph cwd unknown /proc/73403/task/73412/cwd (readlink: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73412 service ceph rtd unknown /proc/73403/task/73412/root (readlink: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73412 service ceph txt unknown /proc/73403/task/73412/exe (readlink: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73412 service ceph NOFD /proc/73403/task/73412/fd (opendir: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73413 admin_soc ceph cwd unknown /proc/73403/task/73413/cwd (readlink: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73413 admin_soc ceph rtd unknown /proc/73403/task/73413/root (readlink: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73413 admin_soc ceph txt unknown /proc/73403/task/73413/exe (readlink: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73413 admin_soc ceph NOFD /proc/73403/task/73413/fd (opendir: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73414 signal_ha ceph cwd unknown /proc/73403/task/73414/cwd (readlink: Permission denied) 2024-03-11T20:01:36.106 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73414 signal_ha ceph rtd unknown /proc/73403/task/73414/root (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73414 signal_ha ceph txt unknown /proc/73403/task/73414/exe (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73414 signal_ha ceph NOFD /proc/73403/task/73414/fd (opendir: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73415 io_contex ceph cwd unknown /proc/73403/task/73415/cwd (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73415 io_contex ceph rtd unknown /proc/73403/task/73415/root (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73415 io_contex ceph txt unknown /proc/73403/task/73415/exe (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73415 io_contex ceph NOFD /proc/73403/task/73415/fd (opendir: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73416 io_contex ceph cwd unknown /proc/73403/task/73416/cwd (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73416 io_contex ceph rtd unknown /proc/73403/task/73416/root (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73416 io_contex ceph txt unknown /proc/73403/task/73416/exe (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73416 io_contex ceph NOFD /proc/73403/task/73416/fd (opendir: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73417 ms_dispat ceph cwd unknown /proc/73403/task/73417/cwd (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73417 ms_dispat ceph rtd unknown /proc/73403/task/73417/root (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73417 ms_dispat ceph txt unknown /proc/73403/task/73417/exe (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73417 ms_dispat ceph NOFD /proc/73403/task/73417/fd (opendir: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73418 ms_local ceph cwd unknown /proc/73403/task/73418/cwd (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73418 ms_local ceph rtd unknown /proc/73403/task/73418/root (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73418 ms_local ceph txt unknown /proc/73403/task/73418/exe (readlink: Permission denied) 2024-03-11T20:01:36.107 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73418 ms_local ceph NOFD /proc/73403/task/73418/fd (opendir: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73419 safe_time ceph cwd unknown /proc/73403/task/73419/cwd (readlink: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73419 safe_time ceph rtd unknown /proc/73403/task/73419/root (readlink: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73419 safe_time ceph txt unknown /proc/73403/task/73419/exe (readlink: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73419 safe_time ceph NOFD /proc/73403/task/73419/fd (opendir: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73420 safe_time ceph cwd unknown /proc/73403/task/73420/cwd (readlink: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73420 safe_time ceph rtd unknown /proc/73403/task/73420/root (readlink: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73420 safe_time ceph txt unknown /proc/73403/task/73420/exe (readlink: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73420 safe_time ceph NOFD /proc/73403/task/73420/fd (opendir: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73421 ceph-mds ceph cwd unknown /proc/73403/task/73421/cwd (readlink: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73421 ceph-mds ceph rtd unknown /proc/73403/task/73421/root (readlink: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73421 ceph-mds ceph txt unknown /proc/73403/task/73421/exe (readlink: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73421 ceph-mds ceph NOFD /proc/73403/task/73421/fd (opendir: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73492 safe_time ceph cwd unknown /proc/73403/task/73492/cwd (readlink: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73492 safe_time ceph rtd unknown /proc/73403/task/73492/root (readlink: Permission denied) 2024-03-11T20:01:36.108 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73492 safe_time ceph txt unknown /proc/73403/task/73492/exe (readlink: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73403 73492 safe_time ceph NOFD /proc/73403/task/73492/fd (opendir: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 73688 root cwd unknown /proc/73688/cwd (readlink: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 73688 root rtd unknown /proc/73688/root (readlink: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 73688 root txt unknown /proc/73688/exe (readlink: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 73688 root NOFD /proc/73688/fd (opendir: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 73765 root cwd unknown /proc/73765/cwd (readlink: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 73765 root rtd unknown /proc/73765/root (readlink: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 73765 root txt unknown /proc/73765/exe (readlink: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:conmon 73765 root NOFD /proc/73765/fd (opendir: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 73775 root cwd unknown /proc/73775/cwd (readlink: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 73775 root rtd unknown /proc/73775/root (readlink: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 73775 root txt unknown /proc/73775/exe (readlink: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:podman-in 73775 root NOFD /proc/73775/fd (opendir: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 ceph cwd unknown /proc/73787/cwd (readlink: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 ceph rtd unknown /proc/73787/root (readlink: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 ceph txt unknown /proc/73787/exe (readlink: Permission denied) 2024-03-11T20:01:36.109 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 ceph NOFD /proc/73787/fd (opendir: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73788 log ceph cwd unknown /proc/73787/task/73788/cwd (readlink: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73788 log ceph rtd unknown /proc/73787/task/73788/root (readlink: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73788 log ceph txt unknown /proc/73787/task/73788/exe (readlink: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73788 log ceph NOFD /proc/73787/task/73788/fd (opendir: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73790 msgr-work ceph cwd unknown /proc/73787/task/73790/cwd (readlink: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73790 msgr-work ceph rtd unknown /proc/73787/task/73790/root (readlink: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73790 msgr-work ceph txt unknown /proc/73787/task/73790/exe (readlink: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73790 msgr-work ceph NOFD /proc/73787/task/73790/fd (opendir: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73791 msgr-work ceph cwd unknown /proc/73787/task/73791/cwd (readlink: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73791 msgr-work ceph rtd unknown /proc/73787/task/73791/root (readlink: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73791 msgr-work ceph txt unknown /proc/73787/task/73791/exe (readlink: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73791 msgr-work ceph NOFD /proc/73787/task/73791/fd (opendir: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73792 msgr-work ceph cwd unknown /proc/73787/task/73792/cwd (readlink: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73792 msgr-work ceph rtd unknown /proc/73787/task/73792/root (readlink: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73792 msgr-work ceph txt unknown /proc/73787/task/73792/exe (readlink: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73792 msgr-work ceph NOFD /proc/73787/task/73792/fd (opendir: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73796 service ceph cwd unknown /proc/73787/task/73796/cwd (readlink: Permission denied) 2024-03-11T20:01:36.110 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73796 service ceph rtd unknown /proc/73787/task/73796/root (readlink: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73796 service ceph txt unknown /proc/73787/task/73796/exe (readlink: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73796 service ceph NOFD /proc/73787/task/73796/fd (opendir: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73797 admin_soc ceph cwd unknown /proc/73787/task/73797/cwd (readlink: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73797 admin_soc ceph rtd unknown /proc/73787/task/73797/root (readlink: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73797 admin_soc ceph txt unknown /proc/73787/task/73797/exe (readlink: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73797 admin_soc ceph NOFD /proc/73787/task/73797/fd (opendir: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73798 signal_ha ceph cwd unknown /proc/73787/task/73798/cwd (readlink: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73798 signal_ha ceph rtd unknown /proc/73787/task/73798/root (readlink: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73798 signal_ha ceph txt unknown /proc/73787/task/73798/exe (readlink: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73798 signal_ha ceph NOFD /proc/73787/task/73798/fd (opendir: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73799 io_contex ceph cwd unknown /proc/73787/task/73799/cwd (readlink: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73799 io_contex ceph rtd unknown /proc/73787/task/73799/root (readlink: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73799 io_contex ceph txt unknown /proc/73787/task/73799/exe (readlink: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73799 io_contex ceph NOFD /proc/73787/task/73799/fd (opendir: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73800 io_contex ceph cwd unknown /proc/73787/task/73800/cwd (readlink: Permission denied) 2024-03-11T20:01:36.111 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73800 io_contex ceph rtd unknown /proc/73787/task/73800/root (readlink: Permission denied) 2024-03-11T20:01:36.112 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73800 io_contex ceph txt unknown /proc/73787/task/73800/exe (readlink: Permission denied) 2024-03-11T20:01:36.112 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73800 io_contex ceph NOFD /proc/73787/task/73800/fd (opendir: Permission denied) 2024-03-11T20:01:36.112 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73801 ms_dispat ceph cwd unknown /proc/73787/task/73801/cwd (readlink: Permission denied) 2024-03-11T20:01:36.112 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73801 ms_dispat ceph rtd unknown /proc/73787/task/73801/root (readlink: Permission denied) 2024-03-11T20:01:36.112 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73801 ms_dispat ceph txt unknown /proc/73787/task/73801/exe (readlink: Permission denied) 2024-03-11T20:01:36.112 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73801 ms_dispat ceph NOFD /proc/73787/task/73801/fd (opendir: Permission denied) 2024-03-11T20:01:36.112 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73802 ms_local ceph cwd unknown /proc/73787/task/73802/cwd (readlink: Permission denied) 2024-03-11T20:01:36.112 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73802 ms_local ceph rtd unknown /proc/73787/task/73802/root (readlink: Permission denied) 2024-03-11T20:01:36.112 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73802 ms_local ceph txt unknown /proc/73787/task/73802/exe (readlink: Permission denied) 2024-03-11T20:01:36.112 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73802 ms_local ceph NOFD /proc/73787/task/73802/fd (opendir: Permission denied) 2024-03-11T20:01:36.112 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73803 safe_time ceph cwd unknown /proc/73787/task/73803/cwd (readlink: Permission denied) 2024-03-11T20:01:36.112 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73803 safe_time ceph rtd unknown /proc/73787/task/73803/root (readlink: Permission denied) 2024-03-11T20:01:36.112 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73803 safe_time ceph txt unknown /proc/73787/task/73803/exe (readlink: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73803 safe_time ceph NOFD /proc/73787/task/73803/fd (opendir: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73804 safe_time ceph cwd unknown /proc/73787/task/73804/cwd (readlink: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73804 safe_time ceph rtd unknown /proc/73787/task/73804/root (readlink: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73804 safe_time ceph txt unknown /proc/73787/task/73804/exe (readlink: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73804 safe_time ceph NOFD /proc/73787/task/73804/fd (opendir: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73805 ceph-mds ceph cwd unknown /proc/73787/task/73805/cwd (readlink: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73805 ceph-mds ceph rtd unknown /proc/73787/task/73805/root (readlink: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73805 ceph-mds ceph txt unknown /proc/73787/task/73805/exe (readlink: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73805 ceph-mds ceph NOFD /proc/73787/task/73805/fd (opendir: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73806 safe_time ceph cwd unknown /proc/73787/task/73806/cwd (readlink: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73806 safe_time ceph rtd unknown /proc/73787/task/73806/root (readlink: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73806 safe_time ceph txt unknown /proc/73787/task/73806/exe (readlink: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 73806 safe_time ceph NOFD /proc/73787/task/73806/fd (opendir: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76770 ceph_time ceph cwd unknown /proc/73787/task/76770/cwd (readlink: Permission denied) 2024-03-11T20:01:36.113 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76770 ceph_time ceph rtd unknown /proc/73787/task/76770/root (readlink: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76770 ceph_time ceph txt unknown /proc/73787/task/76770/exe (readlink: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76770 ceph_time ceph NOFD /proc/73787/task/76770/fd (opendir: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76771 OpHistory ceph cwd unknown /proc/73787/task/76771/cwd (readlink: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76771 OpHistory ceph rtd unknown /proc/73787/task/76771/root (readlink: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76771 OpHistory ceph txt unknown /proc/73787/task/76771/exe (readlink: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76771 OpHistory ceph NOFD /proc/73787/task/76771/fd (opendir: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76772 ms_dispat ceph cwd unknown /proc/73787/task/76772/cwd (readlink: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76772 ms_dispat ceph rtd unknown /proc/73787/task/76772/root (readlink: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76772 ms_dispat ceph txt unknown /proc/73787/task/76772/exe (readlink: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76772 ms_dispat ceph NOFD /proc/73787/task/76772/fd (opendir: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76773 ms_dispat ceph cwd unknown /proc/73787/task/76773/cwd (readlink: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76773 ms_dispat ceph rtd unknown /proc/73787/task/76773/root (readlink: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76773 ms_dispat ceph txt unknown /proc/73787/task/76773/exe (readlink: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76773 ms_dispat ceph NOFD /proc/73787/task/76773/fd (opendir: Permission denied) 2024-03-11T20:01:36.114 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76774 mds_rank_ ceph cwd unknown /proc/73787/task/76774/cwd (readlink: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76774 mds_rank_ ceph rtd unknown /proc/73787/task/76774/root (readlink: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76774 mds_rank_ ceph txt unknown /proc/73787/task/76774/exe (readlink: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76774 mds_rank_ ceph NOFD /proc/73787/task/76774/fd (opendir: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76775 PQ_Finish ceph cwd unknown /proc/73787/task/76775/cwd (readlink: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76775 PQ_Finish ceph rtd unknown /proc/73787/task/76775/root (readlink: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76775 PQ_Finish ceph txt unknown /proc/73787/task/76775/exe (readlink: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76775 PQ_Finish ceph NOFD /proc/73787/task/76775/fd (opendir: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76776 safe_time ceph cwd unknown /proc/73787/task/76776/cwd (readlink: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76776 safe_time ceph rtd unknown /proc/73787/task/76776/root (readlink: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76776 safe_time ceph txt unknown /proc/73787/task/76776/exe (readlink: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76776 safe_time ceph NOFD /proc/73787/task/76776/fd (opendir: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76777 MR_Finish ceph cwd unknown /proc/73787/task/76777/cwd (readlink: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76777 MR_Finish ceph rtd unknown /proc/73787/task/76777/root (readlink: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76777 MR_Finish ceph txt unknown /proc/73787/task/76777/exe (readlink: Permission denied) 2024-03-11T20:01:36.115 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76777 MR_Finish ceph NOFD /proc/73787/task/76777/fd (opendir: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76778 md_submit ceph cwd unknown /proc/73787/task/76778/cwd (readlink: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76778 md_submit ceph rtd unknown /proc/73787/task/76778/root (readlink: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76778 md_submit ceph txt unknown /proc/73787/task/76778/exe (readlink: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76778 md_submit ceph NOFD /proc/73787/task/76778/fd (opendir: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76819 ms_dispat ceph cwd unknown /proc/73787/task/76819/cwd (readlink: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76819 ms_dispat ceph rtd unknown /proc/73787/task/76819/root (readlink: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76819 ms_dispat ceph txt unknown /proc/73787/task/76819/exe (readlink: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:ceph-mds 73787 76819 ms_dispat ceph NOFD /proc/73787/task/76819/fd (opendir: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 73852 root cwd unknown /proc/73852/cwd (readlink: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 73852 root rtd unknown /proc/73852/root (readlink: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 73852 root txt unknown /proc/73852/exe (readlink: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/u 73852 root NOFD /proc/73852/fd (opendir: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 75507 root cwd unknown /proc/75507/cwd (readlink: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 75507 root rtd unknown /proc/75507/root (readlink: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 75507 root txt unknown /proc/75507/exe (readlink: Permission denied) 2024-03-11T20:01:36.116 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/5 75507 root NOFD /proc/75507/fd (opendir: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 75715 root cwd unknown /proc/75715/cwd (readlink: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 75715 root rtd unknown /proc/75715/root (readlink: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 75715 root txt unknown /proc/75715/exe (readlink: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 75715 root NOFD /proc/75715/fd (opendir: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 75716 root cwd unknown /proc/75716/cwd (readlink: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 75716 root rtd unknown /proc/75716/root (readlink: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 75716 root txt unknown /proc/75716/exe (readlink: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 75716 root NOFD /proc/75716/fd (opendir: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 82619 root cwd unknown /proc/82619/cwd (readlink: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 82619 root rtd unknown /proc/82619/root (readlink: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 82619 root txt unknown /proc/82619/exe (readlink: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/2 82619 root NOFD /proc/82619/fd (opendir: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 84674 root cwd unknown /proc/84674/cwd (readlink: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 84674 root rtd unknown /proc/84674/root (readlink: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 84674 root txt unknown /proc/84674/exe (readlink: Permission denied) 2024-03-11T20:01:36.117 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/6 84674 root NOFD /proc/84674/fd (opendir: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 86607 root cwd unknown /proc/86607/cwd (readlink: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 86607 root rtd unknown /proc/86607/root (readlink: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 86607 root txt unknown /proc/86607/exe (readlink: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/3 86607 root NOFD /proc/86607/fd (opendir: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87097 root cwd unknown /proc/87097/cwd (readlink: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87097 root rtd unknown /proc/87097/root (readlink: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87097 root txt unknown /proc/87097/exe (readlink: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87097 root NOFD /proc/87097/fd (opendir: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87112 root cwd unknown /proc/87112/cwd (readlink: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87112 root rtd unknown /proc/87112/root (readlink: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87112 root txt unknown /proc/87112/exe (readlink: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87112 root NOFD /proc/87112/fd (opendir: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87113 root cwd unknown /proc/87113/cwd (readlink: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87113 root rtd unknown /proc/87113/root (readlink: Permission denied) 2024-03-11T20:01:36.118 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87113 root txt unknown /proc/87113/exe (readlink: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/R 87113 root NOFD /proc/87113/fd (opendir: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 87601 root cwd unknown /proc/87601/cwd (readlink: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 87601 root rtd unknown /proc/87601/root (readlink: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 87601 root txt unknown /proc/87601/exe (readlink: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 87601 root NOFD /proc/87601/fd (opendir: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 87661 root cwd unknown /proc/87661/cwd (readlink: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 87661 root rtd unknown /proc/87661/root (readlink: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 87661 root txt unknown /proc/87661/exe (readlink: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:sshd 87661 root NOFD /proc/87661/fd (opendir: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 88116 root cwd unknown /proc/88116/cwd (readlink: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 88116 root rtd unknown /proc/88116/root (readlink: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 88116 root txt unknown /proc/88116/exe (readlink: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:sudo 88116 root NOFD /proc/88116/fd (opendir: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:daemon-he 88160 root cwd unknown /proc/88160/cwd (readlink: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:daemon-he 88160 root rtd unknown /proc/88160/root (readlink: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:daemon-he 88160 root txt unknown /proc/88160/exe (readlink: Permission denied) 2024-03-11T20:01:36.119 INFO:tasks.workunit.client.0.smithi097.stdout:daemon-he 88160 root NOFD /proc/88160/fd (opendir: Permission denied) 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:bash 88170 root cwd unknown /proc/88170/cwd (readlink: Permission denied) 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:bash 88170 root rtd unknown /proc/88170/root (readlink: Permission denied) 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:bash 88170 root txt unknown /proc/88170/exe (readlink: Permission denied) 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:bash 88170 root NOFD /proc/88170/fd (opendir: Permission denied) 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 89499 root cwd unknown /proc/89499/cwd (readlink: Permission denied) 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 89499 root rtd unknown /proc/89499/root (readlink: Permission denied) 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 89499 root txt unknown /proc/89499/exe (readlink: Permission denied) 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/1 89499 root NOFD /proc/89499/fd (opendir: Permission denied) 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu cwd DIR 0,118 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu rtd DIR 8,1 4096 2 / 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu txt REG 8,1 40944 10098 /usr/bin/timeout 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-03-11T20:01:36.120 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu 0r FIFO 0,13 0t0 228465 pipe 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu 1w FIFO 0,13 0t0 228466 pipe 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:timeout 89851 ubuntu 2w FIFO 0,13 0t0 228467 pipe 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu cwd DIR 0,118 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu rtd DIR 8,1 4096 2 / 2024-03-11T20:01:36.121 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu txt REG 8,1 1388920 9667 /usr/bin/bash 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 195080 9664 /usr/lib64/libtinfo.so.6.2 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-11T20:01:36.122 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu 0r FIFO 0,13 0t0 228465 pipe 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu 1w FIFO 0,13 0t0 228466 pipe 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu 2w FIFO 0,13 0t0 228467 pipe 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:fsync-tes 89869 ubuntu 255r REG 8,1 207 792405 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsync-tester.sh 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 89972 root cwd unknown /proc/89972/cwd (readlink: Permission denied) 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 89972 root rtd unknown /proc/89972/root (readlink: Permission denied) 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 89972 root txt unknown /proc/89972/exe (readlink: Permission denied) 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/4 89972 root NOFD /proc/89972/fd (opendir: Permission denied) 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:anacron 89986 root cwd unknown /proc/89986/cwd (readlink: Permission denied) 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:anacron 89986 root rtd unknown /proc/89986/root (readlink: Permission denied) 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:anacron 89986 root txt unknown /proc/89986/exe (readlink: Permission denied) 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:anacron 89986 root NOFD /proc/89986/fd (opendir: Permission denied) 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 90036 root cwd unknown /proc/90036/cwd (readlink: Permission denied) 2024-03-11T20:01:36.123 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 90036 root rtd unknown /proc/90036/root (readlink: Permission denied) 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 90036 root txt unknown /proc/90036/exe (readlink: Permission denied) 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/7 90036 root NOFD /proc/90036/fd (opendir: Permission denied) 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 90084 root cwd unknown /proc/90084/cwd (readlink: Permission denied) 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 90084 root rtd unknown /proc/90084/root (readlink: Permission denied) 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 90084 root txt unknown /proc/90084/exe (readlink: Permission denied) 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:kworker/0 90084 root NOFD /proc/90084/fd (opendir: Permission denied) 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:sleep 90259 root cwd unknown /proc/90259/cwd (readlink: Permission denied) 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:sleep 90259 root rtd unknown /proc/90259/root (readlink: Permission denied) 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:sleep 90259 root txt unknown /proc/90259/exe (readlink: Permission denied) 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:sleep 90259 root NOFD /proc/90259/fd (opendir: Permission denied) 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu cwd DIR 0,118 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu rtd DIR 8,1 4096 2 / 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu txt REG 8,1 179232 12343 /usr/bin/lsof 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-11T20:01:36.124 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 4487176 2250 /usr/lib64/libcrypto.so.3.0.7 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 196112 11431 /usr/lib64/libtirpc.so.3.0.0 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu 0r FIFO 0,13 0t0 228465 pipe 2024-03-11T20:01:36.125 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu 1w FIFO 0,13 0t0 228466 pipe 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu 2w FIFO 0,13 0t0 228467 pipe 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu 3r DIR 0,20 0 1 /proc 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu 4r DIR 0,20 7 230411 /proc/90261/fd 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu 5w FIFO 0,13 0t0 230416 pipe 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90261 ubuntu 6r FIFO 0,13 0t0 230417 pipe 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu cwd DIR 0,118 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu rtd DIR 8,1 4096 2 / 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu txt REG 8,1 179232 12343 /usr/bin/lsof 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 4487176 2250 /usr/lib64/libcrypto.so.3.0.7 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-11T20:01:36.126 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-03-11T20:01:36.127 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-03-11T20:01:36.127 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-03-11T20:01:36.127 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-03-11T20:01:36.127 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-03-11T20:01:36.127 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 196112 11431 /usr/lib64/libtirpc.so.3.0.0 2024-03-11T20:01:36.127 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-11T20:01:36.127 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-11T20:01:36.127 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-11T20:01:36.127 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu 4r FIFO 0,13 0t0 230416 pipe 2024-03-11T20:01:36.127 INFO:tasks.workunit.client.0.smithi097.stdout:lsof 90262 ubuntu 7w FIFO 0,13 0t0 230417 pipe 2024-03-11T20:01:36.127 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-11T20:01:36.128 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-11T20:01:36.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:35 smithi097 ceph-mon[31898]: pgmap v72: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 868 KiB/s wr, 13 op/s 2024-03-11T20:01:36.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:35 smithi114 ceph-mon[36376]: pgmap v72: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 868 KiB/s wr, 13 op/s 2024-03-11T20:01:36.247 INFO:tasks.workunit:Stopping ['suites/fsync-tester.sh'] on client.0... 2024-03-11T20:01:36.247 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-11T20:01:36.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:35 smithi161 ceph-mon[36332]: pgmap v72: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 868 KiB/s wr, 13 op/s 2024-03-11T20:01:37.685 DEBUG:teuthology.parallel:result is None 2024-03-11T20:01:37.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-11T20:01:37.784 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-11T20:01:37.784 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-11T20:01:37.794 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi097.front.sepia.ceph.com 2024-03-11T20:01:37.812 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-11T20:01:37.821 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-11T20:01:37.821 INFO:teuthology.task.exec:Executing custom commands... 2024-03-11T20:01:37.821 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-03-11T20:01:37.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-11T20:01:37.934 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:37 smithi097 ceph-mon[31898]: pgmap v73: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 868 KiB/s wr, 13 op/s 2024-03-11T20:01:37.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-11T20:01:38.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:37 smithi114 ceph-mon[36376]: pgmap v73: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 868 KiB/s wr, 13 op/s 2024-03-11T20:01:38.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:37 smithi161 ceph-mon[36332]: pgmap v73: 129 pgs: 129 active+clean; 295 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 868 KiB/s wr, 13 op/s 2024-03-11T20:01:40.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:39 smithi097 ceph-mon[31898]: pgmap v74: 129 pgs: 129 active+clean; 516 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 MiB/s wr, 24 op/s 2024-03-11T20:01:40.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:39 smithi114 ceph-mon[36376]: pgmap v74: 129 pgs: 129 active+clean; 516 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 MiB/s wr, 24 op/s 2024-03-11T20:01:40.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:39 smithi161 ceph-mon[36332]: pgmap v74: 129 pgs: 129 active+clean; 516 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 MiB/s wr, 24 op/s 2024-03-11T20:01:42.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:41 smithi097 ceph-mon[31898]: pgmap v75: 129 pgs: 129 active+clean; 516 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 MiB/s wr, 18 op/s 2024-03-11T20:01:42.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:41 smithi114 ceph-mon[36376]: pgmap v75: 129 pgs: 129 active+clean; 516 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 MiB/s wr, 18 op/s 2024-03-11T20:01:42.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:41 smithi161 ceph-mon[36332]: pgmap v75: 129 pgs: 129 active+clean; 516 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 MiB/s wr, 18 op/s 2024-03-11T20:01:44.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:43 smithi097 ceph-mon[31898]: pgmap v76: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 MiB/s wr, 30 op/s 2024-03-11T20:01:44.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:43 smithi114 ceph-mon[36376]: pgmap v76: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 MiB/s wr, 30 op/s 2024-03-11T20:01:44.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:43 smithi161 ceph-mon[36332]: pgmap v76: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 22 MiB/s wr, 30 op/s 2024-03-11T20:01:46.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:45 smithi097 ceph-mon[31898]: pgmap v77: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 MiB/s wr, 22 op/s 2024-03-11T20:01:46.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:45 smithi114 ceph-mon[36376]: pgmap v77: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 MiB/s wr, 22 op/s 2024-03-11T20:01:46.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:45 smithi161 ceph-mon[36332]: pgmap v77: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 MiB/s wr, 22 op/s 2024-03-11T20:01:48.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:47 smithi097 ceph-mon[31898]: pgmap v78: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 MiB/s wr, 22 op/s 2024-03-11T20:01:48.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:47 smithi114 ceph-mon[36376]: pgmap v78: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 MiB/s wr, 22 op/s 2024-03-11T20:01:48.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:47 smithi161 ceph-mon[36332]: pgmap v78: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 MiB/s wr, 22 op/s 2024-03-11T20:01:50.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:49 smithi097 ceph-mon[31898]: pgmap v79: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 MiB/s wr, 26 op/s 2024-03-11T20:01:50.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:49 smithi114 ceph-mon[36376]: pgmap v79: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 MiB/s wr, 26 op/s 2024-03-11T20:01:50.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:49 smithi161 ceph-mon[36332]: pgmap v79: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 MiB/s wr, 26 op/s 2024-03-11T20:01:52.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:51 smithi097 ceph-mon[31898]: pgmap v80: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.7 KiB/s wr, 16 op/s 2024-03-11T20:01:52.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:51 smithi114 ceph-mon[36376]: pgmap v80: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.7 KiB/s wr, 16 op/s 2024-03-11T20:01:52.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:51 smithi161 ceph-mon[36332]: pgmap v80: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.7 KiB/s wr, 16 op/s 2024-03-11T20:01:54.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:53 smithi097 ceph-mon[31898]: pgmap v81: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 KiB/s wr, 20 op/s 2024-03-11T20:01:54.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:53 smithi114 ceph-mon[36376]: pgmap v81: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 KiB/s wr, 20 op/s 2024-03-11T20:01:54.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:53 smithi161 ceph-mon[36332]: pgmap v81: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.4 KiB/s wr, 20 op/s 2024-03-11T20:01:56.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:55 smithi097 ceph-mon[31898]: pgmap v82: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-11T20:01:56.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:55 smithi114 ceph-mon[36376]: pgmap v82: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-11T20:01:56.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:55 smithi161 ceph-mon[36332]: pgmap v82: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-11T20:01:58.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:57 smithi097 ceph-mon[31898]: pgmap v83: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-11T20:01:58.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:57 smithi114 ceph-mon[36376]: pgmap v83: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-11T20:01:58.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:57 smithi161 ceph-mon[36332]: pgmap v83: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-03-11T20:01:59.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:01:59 smithi097 ceph-mon[31898]: pgmap v84: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-11T20:02:00.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:01:59 smithi114 ceph-mon[36376]: pgmap v84: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-11T20:02:00.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:01:59 smithi161 ceph-mon[36332]: pgmap v84: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-11T20:02:02.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:01 smithi097 ceph-mon[31898]: pgmap v85: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-03-11T20:02:02.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:01 smithi114 ceph-mon[36376]: pgmap v85: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-03-11T20:02:02.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:01 smithi161 ceph-mon[36332]: pgmap v85: 129 pgs: 129 active+clean; 259 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-03-11T20:02:04.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:03 smithi097 ceph-mon[31898]: pgmap v86: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-11T20:02:04.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:03 smithi114 ceph-mon[36376]: pgmap v86: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-11T20:02:04.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:03 smithi161 ceph-mon[36332]: pgmap v86: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-11T20:02:06.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:05 smithi097 ceph-mon[31898]: pgmap v87: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:02:06.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:05 smithi114 ceph-mon[36376]: pgmap v87: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:02:06.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:05 smithi161 ceph-mon[36332]: pgmap v87: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:02:08.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:07 smithi097 ceph-mon[31898]: pgmap v88: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:02:08.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:07 smithi114 ceph-mon[36376]: pgmap v88: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:02:08.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:07 smithi161 ceph-mon[36332]: pgmap v88: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:02:10.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:09 smithi097 ceph-mon[31898]: pgmap v89: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:02:10.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:09 smithi114 ceph-mon[36376]: pgmap v89: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:02:10.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:09 smithi161 ceph-mon[36332]: pgmap v89: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:02:12.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:11 smithi097 ceph-mon[31898]: pgmap v90: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:02:12.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:11 smithi114 ceph-mon[36376]: pgmap v90: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:02:12.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:11 smithi161 ceph-mon[36332]: pgmap v90: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:02:14.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:13 smithi097 ceph-mon[31898]: pgmap v91: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-11T20:02:14.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:13 smithi114 ceph-mon[36376]: pgmap v91: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-11T20:02:14.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:13 smithi161 ceph-mon[36332]: pgmap v91: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-11T20:02:15.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:14 smithi114 ceph-mon[36376]: pgmap v92: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-11T20:02:15.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:14 smithi161 ceph-mon[36332]: pgmap v92: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-11T20:02:15.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:14 smithi097 ceph-mon[31898]: pgmap v92: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-11T20:02:18.115 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:17 smithi097 ceph-mon[31898]: pgmap v93: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-11T20:02:18.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:17 smithi114 ceph-mon[36376]: pgmap v93: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-11T20:02:18.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:17 smithi161 ceph-mon[36332]: pgmap v93: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-11T20:02:20.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:19 smithi097 ceph-mon[31898]: pgmap v94: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:02:20.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:19 smithi114 ceph-mon[36376]: pgmap v94: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:02:20.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:19 smithi161 ceph-mon[36332]: pgmap v94: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:02:22.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:21 smithi097 ceph-mon[31898]: pgmap v95: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:02:22.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:21 smithi114 ceph-mon[36376]: pgmap v95: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:02:22.280 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:21 smithi161 ceph-mon[36332]: pgmap v95: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:02:24.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:23 smithi097 ceph-mon[31898]: pgmap v96: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-11T20:02:24.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:23 smithi114 ceph-mon[36376]: pgmap v96: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-11T20:02:24.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:23 smithi161 ceph-mon[36332]: pgmap v96: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-11T20:02:26.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:25 smithi097 ceph-mon[31898]: pgmap v97: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:26.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:25 smithi114 ceph-mon[36376]: pgmap v97: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:26.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:25 smithi161 ceph-mon[36332]: pgmap v97: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:27 smithi097 ceph-mon[31898]: pgmap v98: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:28.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:27 smithi114 ceph-mon[36376]: pgmap v98: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:28.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:27 smithi161 ceph-mon[36332]: pgmap v98: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:29.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:28 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:02:29.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:28 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:02:29.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:28 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:29 smithi097 ceph-mon[31898]: pgmap v99: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:29 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:29 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:29 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:02:30.181 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:29 smithi161 ceph-mon[36332]: pgmap v99: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:30.181 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:29 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:02:30.181 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:29 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:02:30.181 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:29 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:02:30.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:29 smithi114 ceph-mon[36376]: pgmap v99: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:30.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:29 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:02:30.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:29 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:02:30.234 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:29 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:02:32.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:31 smithi097 ceph-mon[31898]: pgmap v100: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:32.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:31 smithi114 ceph-mon[36376]: pgmap v100: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:32.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:31 smithi161 ceph-mon[36332]: pgmap v100: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:34.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:33 smithi097 ceph-mon[31898]: pgmap v101: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:34.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:33 smithi114 ceph-mon[36376]: pgmap v101: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:34.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:33 smithi161 ceph-mon[36332]: pgmap v101: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:36.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:35 smithi097 ceph-mon[31898]: pgmap v102: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:36.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:35 smithi114 ceph-mon[36376]: pgmap v102: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:36.282 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:35 smithi161 ceph-mon[36332]: pgmap v102: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:38.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:37 smithi097 ceph-mon[31898]: pgmap v103: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:38.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:37 smithi114 ceph-mon[36376]: pgmap v103: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:38.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:37 smithi161 ceph-mon[36332]: pgmap v103: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:40.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:39 smithi097 ceph-mon[31898]: pgmap v104: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:40.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:39 smithi114 ceph-mon[36376]: pgmap v104: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:40.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:39 smithi161 ceph-mon[36332]: pgmap v104: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:42.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:41 smithi097 ceph-mon[31898]: pgmap v105: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:42.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:41 smithi114 ceph-mon[36376]: pgmap v105: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:42.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:41 smithi161 ceph-mon[36332]: pgmap v105: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:44.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:43 smithi097 ceph-mon[31898]: pgmap v106: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:44.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:43 smithi114 ceph-mon[36376]: pgmap v106: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:44.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:43 smithi161 ceph-mon[36332]: pgmap v106: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:46.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:45 smithi097 ceph-mon[31898]: pgmap v107: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:46.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:45 smithi114 ceph-mon[36376]: pgmap v107: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:46.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:45 smithi161 ceph-mon[36332]: pgmap v107: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:48.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:47 smithi097 ceph-mon[31898]: pgmap v108: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:48.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:47 smithi114 ceph-mon[36376]: pgmap v108: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:48.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:47 smithi161 ceph-mon[36332]: pgmap v108: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:50.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:49 smithi097 ceph-mon[31898]: pgmap v109: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:50.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:49 smithi114 ceph-mon[36376]: pgmap v109: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:50.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:49 smithi161 ceph-mon[36332]: pgmap v109: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:51 smithi097 ceph-mon[31898]: pgmap v110: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:52.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:51 smithi114 ceph-mon[36376]: pgmap v110: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:52.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:51 smithi161 ceph-mon[36332]: pgmap v110: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:54.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:53 smithi097 ceph-mon[31898]: pgmap v111: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:54.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:53 smithi114 ceph-mon[36376]: pgmap v111: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:54.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:53 smithi161 ceph-mon[36332]: pgmap v111: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:02:56.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:55 smithi097 ceph-mon[31898]: pgmap v112: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:56.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:55 smithi114 ceph-mon[36376]: pgmap v112: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:56.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:55 smithi161 ceph-mon[36332]: pgmap v112: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:57.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:56 smithi114 ceph-mon[36376]: pgmap v113: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:57.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:56 smithi161 ceph-mon[36332]: pgmap v113: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:02:57.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:56 smithi097 ceph-mon[31898]: pgmap v113: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:00.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:02:59 smithi097 ceph-mon[31898]: pgmap v114: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:00.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:02:59 smithi114 ceph-mon[36376]: pgmap v114: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:00.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:02:59 smithi161 ceph-mon[36332]: pgmap v114: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:02.081 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:01 smithi161 ceph-mon[36332]: pgmap v115: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:02.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:01 smithi097 ceph-mon[31898]: pgmap v115: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:02.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:01 smithi114 ceph-mon[36376]: pgmap v115: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:04.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:03 smithi097 ceph-mon[31898]: pgmap v116: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:03:04.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:03 smithi114 ceph-mon[36376]: pgmap v116: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:03:04.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:03 smithi161 ceph-mon[36332]: pgmap v116: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:03:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:05 smithi097 ceph-mon[31898]: pgmap v117: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:03:06.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:05 smithi114 ceph-mon[36376]: pgmap v117: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:03:06.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:05 smithi161 ceph-mon[36332]: pgmap v117: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:03:08.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:07 smithi097 ceph-mon[31898]: pgmap v118: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:03:08.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:07 smithi114 ceph-mon[36376]: pgmap v118: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:03:08.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:07 smithi161 ceph-mon[36332]: pgmap v118: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:03:10.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:09 smithi097 ceph-mon[31898]: pgmap v119: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:03:10.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:09 smithi114 ceph-mon[36376]: pgmap v119: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:03:10.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:09 smithi161 ceph-mon[36332]: pgmap v119: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:03:12.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:11 smithi097 ceph-mon[31898]: pgmap v120: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:03:12.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:11 smithi114 ceph-mon[36376]: pgmap v120: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:03:12.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:11 smithi161 ceph-mon[36332]: pgmap v120: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:03:14.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:13 smithi097 ceph-mon[31898]: pgmap v121: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:03:14.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:13 smithi114 ceph-mon[36376]: pgmap v121: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:03:14.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:13 smithi161 ceph-mon[36332]: pgmap v121: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:03:16.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:15 smithi097 ceph-mon[31898]: pgmap v122: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:03:16.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:15 smithi114 ceph-mon[36376]: pgmap v122: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:03:16.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:15 smithi161 ceph-mon[36332]: pgmap v122: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:03:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:17 smithi097 ceph-mon[31898]: pgmap v123: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:03:18.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:17 smithi114 ceph-mon[36376]: pgmap v123: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:03:18.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:17 smithi161 ceph-mon[36332]: pgmap v123: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:03:20.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:19 smithi097 ceph-mon[31898]: pgmap v124: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-11T20:03:20.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:19 smithi114 ceph-mon[36376]: pgmap v124: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-11T20:03:20.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:19 smithi161 ceph-mon[36332]: pgmap v124: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-11T20:03:22.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:21 smithi097 ceph-mon[31898]: pgmap v125: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:03:22.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:21 smithi114 ceph-mon[36376]: pgmap v125: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:03:22.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:21 smithi161 ceph-mon[36332]: pgmap v125: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:03:24.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:23 smithi097 ceph-mon[31898]: pgmap v126: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:03:24.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:23 smithi114 ceph-mon[36376]: pgmap v126: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:03:24.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:23 smithi161 ceph-mon[36332]: pgmap v126: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:03:26.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:25 smithi097 ceph-mon[31898]: pgmap v127: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:26.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:25 smithi114 ceph-mon[36376]: pgmap v127: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:26.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:25 smithi161 ceph-mon[36332]: pgmap v127: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:27 smithi097 ceph-mon[31898]: pgmap v128: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:28.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:27 smithi114 ceph-mon[36376]: pgmap v128: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:28.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:27 smithi161 ceph-mon[36332]: pgmap v128: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:30.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:29 smithi097 ceph-mon[31898]: pgmap v129: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:30.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:29 smithi114 ceph-mon[36376]: pgmap v129: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:30.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:29 smithi161 ceph-mon[36332]: pgmap v129: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:31.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:30 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:03:31.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:30 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:03:31.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:30 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:03:32.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:31 smithi097 ceph-mon[31898]: pgmap v130: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:32.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:31 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:03:32.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:31 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:03:32.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:31 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:03:32.180 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:31 smithi161 ceph-mon[36332]: pgmap v130: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:32.180 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:31 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:03:32.180 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:31 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:03:32.180 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:31 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:03:32.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:31 smithi114 ceph-mon[36376]: pgmap v130: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:32.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:31 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:03:32.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:31 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:03:32.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:31 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:03:34.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:33 smithi097 ceph-mon[31898]: pgmap v131: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:34.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:33 smithi114 ceph-mon[36376]: pgmap v131: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:34.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:33 smithi161 ceph-mon[36332]: pgmap v131: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:35.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:34 smithi097 ceph-mon[31898]: pgmap v132: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:35.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:34 smithi114 ceph-mon[36376]: pgmap v132: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:35.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:34 smithi161 ceph-mon[36332]: pgmap v132: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:38.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:37 smithi097 ceph-mon[31898]: pgmap v133: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:38.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:37 smithi114 ceph-mon[36376]: pgmap v133: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:38.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:37 smithi161 ceph-mon[36332]: pgmap v133: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:40.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:39 smithi097 ceph-mon[31898]: pgmap v134: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:40.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:39 smithi114 ceph-mon[36376]: pgmap v134: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:40.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:39 smithi161 ceph-mon[36332]: pgmap v134: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:42.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:41 smithi097 ceph-mon[31898]: pgmap v135: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:42.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:41 smithi114 ceph-mon[36376]: pgmap v135: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:42.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:41 smithi161 ceph-mon[36332]: pgmap v135: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:43 smithi097 ceph-mon[31898]: pgmap v136: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:44.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:43 smithi114 ceph-mon[36376]: pgmap v136: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:44.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:43 smithi161 ceph-mon[36332]: pgmap v136: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:46.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:45 smithi097 ceph-mon[31898]: pgmap v137: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:46.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:45 smithi114 ceph-mon[36376]: pgmap v137: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:46.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:45 smithi161 ceph-mon[36332]: pgmap v137: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:48.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:47 smithi097 ceph-mon[31898]: pgmap v138: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:48.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:47 smithi114 ceph-mon[36376]: pgmap v138: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:48.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:47 smithi161 ceph-mon[36332]: pgmap v138: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:50.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:49 smithi097 ceph-mon[31898]: pgmap v139: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:50.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:49 smithi114 ceph-mon[36376]: pgmap v139: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:50.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:49 smithi161 ceph-mon[36332]: pgmap v139: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:52.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:51 smithi097 ceph-mon[31898]: pgmap v140: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:52.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:51 smithi114 ceph-mon[36376]: pgmap v140: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:52.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:51 smithi161 ceph-mon[36332]: pgmap v140: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:54.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:53 smithi097 ceph-mon[31898]: pgmap v141: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:54.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:53 smithi114 ceph-mon[36376]: pgmap v141: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:54.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:53 smithi161 ceph-mon[36332]: pgmap v141: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:03:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:55 smithi097 ceph-mon[31898]: pgmap v142: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:56.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:55 smithi114 ceph-mon[36376]: pgmap v142: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:56.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:55 smithi161 ceph-mon[36332]: pgmap v142: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:58.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:57 smithi097 ceph-mon[31898]: pgmap v143: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:58.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:57 smithi114 ceph-mon[36376]: pgmap v143: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:03:58.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:57 smithi161 ceph-mon[36332]: pgmap v143: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:00.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:03:59 smithi097 ceph-mon[31898]: pgmap v144: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:00.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:03:59 smithi114 ceph-mon[36376]: pgmap v144: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:00.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:03:59 smithi161 ceph-mon[36332]: pgmap v144: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:02.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:01 smithi097 ceph-mon[31898]: pgmap v145: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:02.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:01 smithi114 ceph-mon[36376]: pgmap v145: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:02.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:01 smithi161 ceph-mon[36332]: pgmap v145: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:04.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:03 smithi097 ceph-mon[31898]: pgmap v146: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:04:04.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:03 smithi114 ceph-mon[36376]: pgmap v146: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:04:04.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:03 smithi161 ceph-mon[36332]: pgmap v146: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:04:06.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:05 smithi097 ceph-mon[31898]: pgmap v147: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:04:06.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:05 smithi114 ceph-mon[36376]: pgmap v147: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:04:06.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:05 smithi161 ceph-mon[36332]: pgmap v147: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:04:08.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:07 smithi097 ceph-mon[31898]: pgmap v148: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:04:08.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:07 smithi114 ceph-mon[36376]: pgmap v148: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:04:08.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:07 smithi161 ceph-mon[36332]: pgmap v148: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:04:10.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:09 smithi114 ceph-mon[36376]: pgmap v149: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:04:10.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:09 smithi161 ceph-mon[36332]: pgmap v149: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:04:10.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:09 smithi097 ceph-mon[31898]: pgmap v149: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:04:11.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:10 smithi114 ceph-mon[36376]: pgmap v150: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:04:11.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:10 smithi161 ceph-mon[36332]: pgmap v150: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:04:11.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:10 smithi097 ceph-mon[31898]: pgmap v150: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:04:14.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:13 smithi097 ceph-mon[31898]: pgmap v151: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:04:14.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:13 smithi114 ceph-mon[36376]: pgmap v151: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:04:14.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:13 smithi161 ceph-mon[36332]: pgmap v151: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:04:16.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:15 smithi097 ceph-mon[31898]: pgmap v152: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:04:16.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:15 smithi114 ceph-mon[36376]: pgmap v152: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:04:16.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:15 smithi161 ceph-mon[36332]: pgmap v152: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:04:18.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:17 smithi097 ceph-mon[31898]: pgmap v153: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:04:18.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:17 smithi114 ceph-mon[36376]: pgmap v153: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:04:18.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:17 smithi161 ceph-mon[36332]: pgmap v153: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:04:20.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:19 smithi097 ceph-mon[31898]: pgmap v154: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-11T20:04:20.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:19 smithi114 ceph-mon[36376]: pgmap v154: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-11T20:04:20.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:19 smithi161 ceph-mon[36332]: pgmap v154: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-11T20:04:22.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:21 smithi097 ceph-mon[31898]: pgmap v155: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:04:22.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:21 smithi114 ceph-mon[36376]: pgmap v155: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:04:22.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:21 smithi161 ceph-mon[36332]: pgmap v155: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:04:24.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:23 smithi097 ceph-mon[31898]: pgmap v156: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:04:24.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:23 smithi114 ceph-mon[36376]: pgmap v156: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:04:24.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:23 smithi161 ceph-mon[36332]: pgmap v156: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:04:26.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:25 smithi097 ceph-mon[31898]: pgmap v157: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:26.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:25 smithi114 ceph-mon[36376]: pgmap v157: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:26.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:25 smithi161 ceph-mon[36332]: pgmap v157: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:28.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:27 smithi097 ceph-mon[31898]: pgmap v158: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:28.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:27 smithi114 ceph-mon[36376]: pgmap v158: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:28.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:27 smithi161 ceph-mon[36332]: pgmap v158: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:30.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:29 smithi097 ceph-mon[31898]: pgmap v159: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:30.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:29 smithi114 ceph-mon[36376]: pgmap v159: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:30.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:29 smithi161 ceph-mon[36332]: pgmap v159: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:32.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:31 smithi097 ceph-mon[31898]: pgmap v160: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:32.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:31 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:04:32.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:31 smithi114 ceph-mon[36376]: pgmap v160: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:32.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:31 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:04:32.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:31 smithi161 ceph-mon[36332]: pgmap v160: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:32.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:31 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:04:33.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:32 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:04:33.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:32 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:04:33.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:32 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:04:33.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:32 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:04:33.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:32 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:04:33.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:32 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:04:33.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:32 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:04:33.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:32 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:04:33.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:32 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:04:34.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:33 smithi097 ceph-mon[31898]: pgmap v161: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:34.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:33 smithi114 ceph-mon[36376]: pgmap v161: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:34.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:33 smithi161 ceph-mon[36332]: pgmap v161: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:36.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:35 smithi097 ceph-mon[31898]: pgmap v162: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-11T20:04:36.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:35 smithi114 ceph-mon[36376]: pgmap v162: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-11T20:04:36.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:35 smithi161 ceph-mon[36332]: pgmap v162: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-11T20:04:38.232 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:37 smithi114 ceph-mon[36376]: pgmap v163: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-11T20:04:38.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:37 smithi161 ceph-mon[36332]: pgmap v163: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-11T20:04:38.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:37 smithi097 ceph-mon[31898]: pgmap v163: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-11T20:04:40.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:39 smithi114 ceph-mon[36376]: pgmap v164: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:40.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:39 smithi161 ceph-mon[36332]: pgmap v164: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:40.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:39 smithi097 ceph-mon[31898]: pgmap v164: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:42.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:41 smithi114 ceph-mon[36376]: pgmap v165: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-11T20:04:42.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:41 smithi161 ceph-mon[36332]: pgmap v165: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-11T20:04:42.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:41 smithi097 ceph-mon[31898]: pgmap v165: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-11T20:04:43.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:42 smithi114 ceph-mon[36376]: pgmap v166: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:43.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:42 smithi161 ceph-mon[36332]: pgmap v166: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:43.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:42 smithi097 ceph-mon[31898]: pgmap v166: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:46.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:45 smithi097 ceph-mon[31898]: pgmap v167: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:46.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:45 smithi114 ceph-mon[36376]: pgmap v167: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:46.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:45 smithi161 ceph-mon[36332]: pgmap v167: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:48.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:47 smithi097 ceph-mon[31898]: pgmap v168: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:48.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:47 smithi114 ceph-mon[36376]: pgmap v168: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:48.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:47 smithi161 ceph-mon[36332]: pgmap v168: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:50.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:49 smithi097 ceph-mon[31898]: pgmap v169: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:50.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:49 smithi114 ceph-mon[36376]: pgmap v169: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:50.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:49 smithi161 ceph-mon[36332]: pgmap v169: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:52.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:51 smithi097 ceph-mon[31898]: pgmap v170: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:52.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:51 smithi114 ceph-mon[36376]: pgmap v170: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:52.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:51 smithi161 ceph-mon[36332]: pgmap v170: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:54.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:53 smithi097 ceph-mon[31898]: pgmap v171: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:54.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:53 smithi114 ceph-mon[36376]: pgmap v171: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:54.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:53 smithi161 ceph-mon[36332]: pgmap v171: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:04:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:55 smithi097 ceph-mon[31898]: pgmap v172: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:56.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:55 smithi114 ceph-mon[36376]: pgmap v172: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:56.278 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:55 smithi161 ceph-mon[36332]: pgmap v172: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:58.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:57 smithi097 ceph-mon[31898]: pgmap v173: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:58.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:57 smithi114 ceph-mon[36376]: pgmap v173: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:04:58.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:57 smithi161 ceph-mon[36332]: pgmap v173: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:00.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:04:59 smithi097 ceph-mon[31898]: pgmap v174: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:00.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:04:59 smithi114 ceph-mon[36376]: pgmap v174: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:00.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:04:59 smithi161 ceph-mon[36332]: pgmap v174: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:02.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:01 smithi097 ceph-mon[31898]: pgmap v175: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:02.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:01 smithi114 ceph-mon[36376]: pgmap v175: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:02.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:01 smithi161 ceph-mon[36332]: pgmap v175: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:04.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:03 smithi114 ceph-mon[36376]: pgmap v176: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:05:04.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:03 smithi161 ceph-mon[36332]: pgmap v176: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:05:04.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:03 smithi097 ceph-mon[31898]: pgmap v176: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:05:06.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:05 smithi114 ceph-mon[36376]: pgmap v177: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:05:06.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:05 smithi161 ceph-mon[36332]: pgmap v177: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:05:06.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:05 smithi097 ceph-mon[31898]: pgmap v177: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:05:08.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:07 smithi114 ceph-mon[36376]: pgmap v178: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:05:08.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:07 smithi161 ceph-mon[36332]: pgmap v178: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:05:08.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:07 smithi097 ceph-mon[31898]: pgmap v178: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:05:10.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:09 smithi114 ceph-mon[36376]: pgmap v179: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:05:10.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:09 smithi161 ceph-mon[36332]: pgmap v179: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:05:10.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:09 smithi097 ceph-mon[31898]: pgmap v179: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:05:11.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:10 smithi114 ceph-mon[36376]: pgmap v180: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:05:11.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:10 smithi161 ceph-mon[36332]: pgmap v180: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:05:11.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:10 smithi097 ceph-mon[31898]: pgmap v180: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-11T20:05:14.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:13 smithi097 ceph-mon[31898]: pgmap v181: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:05:14.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:13 smithi114 ceph-mon[36376]: pgmap v181: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:05:14.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:13 smithi161 ceph-mon[36332]: pgmap v181: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-11T20:05:16.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:15 smithi097 ceph-mon[31898]: pgmap v182: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:05:16.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:15 smithi114 ceph-mon[36376]: pgmap v182: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:05:16.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:15 smithi161 ceph-mon[36332]: pgmap v182: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:05:18.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:17 smithi097 ceph-mon[31898]: pgmap v183: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:05:18.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:17 smithi114 ceph-mon[36376]: pgmap v183: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:05:18.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:17 smithi161 ceph-mon[36332]: pgmap v183: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-11T20:05:20.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:19 smithi097 ceph-mon[31898]: pgmap v184: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-11T20:05:20.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:19 smithi114 ceph-mon[36376]: pgmap v184: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-11T20:05:20.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:19 smithi161 ceph-mon[36332]: pgmap v184: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-11T20:05:22.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:21 smithi097 ceph-mon[31898]: pgmap v185: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:05:22.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:21 smithi114 ceph-mon[36376]: pgmap v185: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:05:22.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:21 smithi161 ceph-mon[36332]: pgmap v185: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:05:24.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:23 smithi097 ceph-mon[31898]: pgmap v186: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:05:24.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:23 smithi114 ceph-mon[36376]: pgmap v186: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:05:24.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:23 smithi161 ceph-mon[36332]: pgmap v186: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:05:26.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:25 smithi097 ceph-mon[31898]: pgmap v187: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:26.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:25 smithi114 ceph-mon[36376]: pgmap v187: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:26.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:25 smithi161 ceph-mon[36332]: pgmap v187: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:28.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:27 smithi114 ceph-mon[36376]: pgmap v188: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:28.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:27 smithi161 ceph-mon[36332]: pgmap v188: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:28.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:27 smithi097 ceph-mon[31898]: pgmap v188: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:30.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:29 smithi114 ceph-mon[36376]: pgmap v189: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:30.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:29 smithi161 ceph-mon[36332]: pgmap v189: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:30.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:29 smithi097 ceph-mon[31898]: pgmap v189: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:32.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:31 smithi114 ceph-mon[36376]: pgmap v190: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:32.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:31 smithi161 ceph-mon[36332]: pgmap v190: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:32.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:31 smithi097 ceph-mon[31898]: pgmap v190: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:34.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:33 smithi114 ceph-mon[36376]: pgmap v191: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:34.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:33 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:05:34.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:33 smithi161 ceph-mon[36332]: pgmap v191: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:34.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:33 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:05:34.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:33 smithi097 ceph-mon[31898]: pgmap v191: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:34.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:33 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:05:35.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:34 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:05:35.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:34 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:05:35.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:34 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:05:35.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:34 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:05:35.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:34 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:05:35.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:34 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:05:35.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:34 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:05:35.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:34 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:05:35.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:34 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:05:36.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:35 smithi114 ceph-mon[36376]: pgmap v192: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:36.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:35 smithi161 ceph-mon[36332]: pgmap v192: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:36.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:35 smithi097 ceph-mon[31898]: pgmap v192: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:37.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:36 smithi114 ceph-mon[36376]: pgmap v193: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:37.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:36 smithi161 ceph-mon[36332]: pgmap v193: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:37.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:36 smithi097 ceph-mon[31898]: pgmap v193: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:40.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:39 smithi097 ceph-mon[31898]: pgmap v194: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:40.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:39 smithi114 ceph-mon[36376]: pgmap v194: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:40.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:39 smithi161 ceph-mon[36332]: pgmap v194: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:42.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:41 smithi097 ceph-mon[31898]: pgmap v195: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:42.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:41 smithi114 ceph-mon[36376]: pgmap v195: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:42.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:41 smithi161 ceph-mon[36332]: pgmap v195: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:44.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:43 smithi097 ceph-mon[31898]: pgmap v196: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:44.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:43 smithi114 ceph-mon[36376]: pgmap v196: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:44.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:43 smithi161 ceph-mon[36332]: pgmap v196: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:46.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:45 smithi097 ceph-mon[31898]: pgmap v197: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:46.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:45 smithi114 ceph-mon[36376]: pgmap v197: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:46.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:45 smithi161 ceph-mon[36332]: pgmap v197: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:48.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:47 smithi114 ceph-mon[36376]: pgmap v198: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:48.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:47 smithi161 ceph-mon[36332]: pgmap v198: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:48.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:47 smithi097 ceph-mon[31898]: pgmap v198: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:50.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:49 smithi097 ceph-mon[31898]: pgmap v199: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:50.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:49 smithi114 ceph-mon[36376]: pgmap v199: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:50.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:49 smithi161 ceph-mon[36332]: pgmap v199: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:52.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:51 smithi114 ceph-mon[36376]: pgmap v200: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:52.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:51 smithi161 ceph-mon[36332]: pgmap v200: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:52.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:51 smithi097 ceph-mon[31898]: pgmap v200: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:53.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:52 smithi161 ceph-mon[36332]: pgmap v201: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:53.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:52 smithi097 ceph-mon[31898]: pgmap v201: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:53.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:52 smithi114 ceph-mon[36376]: pgmap v201: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:05:56.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:55 smithi097 ceph-mon[31898]: pgmap v202: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:56.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:55 smithi114 ceph-mon[36376]: pgmap v202: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:56.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:55 smithi161 ceph-mon[36332]: pgmap v202: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:58.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:57 smithi097 ceph-mon[31898]: pgmap v203: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:58.239 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:57 smithi114 ceph-mon[36376]: pgmap v203: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:05:58.279 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:57 smithi161 ceph-mon[36332]: pgmap v203: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:00.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:05:59 smithi097 ceph-mon[31898]: pgmap v204: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:00.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:05:59 smithi114 ceph-mon[36376]: pgmap v204: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:00.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:05:59 smithi161 ceph-mon[36332]: pgmap v204: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:02.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:01 smithi114 ceph-mon[36376]: pgmap v205: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:02.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:01 smithi161 ceph-mon[36332]: pgmap v205: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:02.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:01 smithi097 ceph-mon[31898]: pgmap v205: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:04.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:03 smithi114 ceph-mon[36376]: pgmap v206: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:06:04.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:03 smithi161 ceph-mon[36332]: pgmap v206: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:06:04.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:03 smithi097 ceph-mon[31898]: pgmap v206: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:06:06.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:05 smithi114 ceph-mon[36376]: pgmap v207: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:06:06.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:05 smithi161 ceph-mon[36332]: pgmap v207: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:06:06.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:05 smithi097 ceph-mon[31898]: pgmap v207: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:06:08.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:07 smithi114 ceph-mon[36376]: pgmap v208: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:06:08.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:07 smithi161 ceph-mon[36332]: pgmap v208: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:06:08.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:07 smithi097 ceph-mon[31898]: pgmap v208: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:06:10.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:09 smithi114 ceph-mon[36376]: pgmap v209: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:06:10.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:09 smithi161 ceph-mon[36332]: pgmap v209: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:06:10.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:09 smithi097 ceph-mon[31898]: pgmap v209: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:06:11.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:10 smithi114 ceph-mon[36376]: osdmap e95: 12 total, 12 up, 12 in 2024-03-11T20:06:11.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:10 smithi114 ceph-mon[36376]: pgmap v211: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-11T20:06:11.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:10 smithi161 ceph-mon[36332]: osdmap e95: 12 total, 12 up, 12 in 2024-03-11T20:06:11.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:10 smithi161 ceph-mon[36332]: pgmap v211: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-11T20:06:11.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:10 smithi097 ceph-mon[31898]: osdmap e95: 12 total, 12 up, 12 in 2024-03-11T20:06:11.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:10 smithi097 ceph-mon[31898]: pgmap v211: 129 pgs: 129 active+clean; 259 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-11T20:06:14.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:13 smithi097 ceph-mon[31898]: pgmap v212: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:06:14.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:13 smithi097 ceph-mon[31898]: osdmap e96: 12 total, 12 up, 12 in 2024-03-11T20:06:14.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:13 smithi114 ceph-mon[36376]: pgmap v212: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:06:14.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:13 smithi114 ceph-mon[36376]: osdmap e96: 12 total, 12 up, 12 in 2024-03-11T20:06:14.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:13 smithi161 ceph-mon[36332]: pgmap v212: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:06:14.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:13 smithi161 ceph-mon[36332]: osdmap e96: 12 total, 12 up, 12 in 2024-03-11T20:06:16.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:15 smithi114 ceph-mon[36376]: pgmap v214: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:06:16.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:15 smithi161 ceph-mon[36332]: pgmap v214: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:06:16.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:15 smithi097 ceph-mon[31898]: pgmap v214: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:06:18.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:17 smithi114 ceph-mon[36376]: pgmap v215: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-11T20:06:18.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:17 smithi161 ceph-mon[36332]: pgmap v215: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-11T20:06:18.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:17 smithi097 ceph-mon[31898]: pgmap v215: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-11T20:06:20.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:19 smithi097 ceph-mon[31898]: pgmap v216: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 229 B/s wr, 11 op/s 2024-03-11T20:06:20.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:19 smithi114 ceph-mon[36376]: pgmap v216: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 229 B/s wr, 11 op/s 2024-03-11T20:06:20.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:19 smithi161 ceph-mon[36332]: pgmap v216: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 229 B/s wr, 11 op/s 2024-03-11T20:06:22.231 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:21 smithi114 ceph-mon[36376]: pgmap v217: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:06:22.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:21 smithi161 ceph-mon[36332]: pgmap v217: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:06:22.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:21 smithi097 ceph-mon[31898]: pgmap v217: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:06:24.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:23 smithi114 ceph-mon[36376]: pgmap v218: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-11T20:06:24.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:23 smithi161 ceph-mon[36332]: pgmap v218: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-11T20:06:24.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:23 smithi097 ceph-mon[31898]: pgmap v218: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-11T20:06:26.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:25 smithi114 ceph-mon[36376]: pgmap v219: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 92 B/s wr, 9 op/s 2024-03-11T20:06:26.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:25 smithi161 ceph-mon[36332]: pgmap v219: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 92 B/s wr, 9 op/s 2024-03-11T20:06:26.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:25 smithi097 ceph-mon[31898]: pgmap v219: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 92 B/s wr, 9 op/s 2024-03-11T20:06:28.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:27 smithi114 ceph-mon[36376]: pgmap v220: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:06:28.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:27 smithi161 ceph-mon[36332]: pgmap v220: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:06:28.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:27 smithi097 ceph-mon[31898]: pgmap v220: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:06:29.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:28 smithi114 ceph-mon[36376]: pgmap v221: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:06:29.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:28 smithi161 ceph-mon[36332]: pgmap v221: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:06:29.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:28 smithi097 ceph-mon[31898]: pgmap v221: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:06:32.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:31 smithi114 ceph-mon[36376]: pgmap v222: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:32.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:31 smithi161 ceph-mon[36332]: pgmap v222: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:32.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:31 smithi097 ceph-mon[31898]: pgmap v222: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:34.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:33 smithi114 ceph-mon[36376]: pgmap v223: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:34.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:33 smithi161 ceph-mon[36332]: pgmap v223: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:34.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:33 smithi097 ceph-mon[31898]: pgmap v223: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:35.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:34 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:06:35.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:34 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:06:35.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:34 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:06:36.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:35 smithi114 ceph-mon[36376]: pgmap v224: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:36.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:35 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:06:36.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:35 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:06:36.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:35 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:06:36.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:35 smithi161 ceph-mon[36332]: pgmap v224: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:36.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:35 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:06:36.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:35 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:06:36.277 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:35 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:06:36.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:35 smithi097 ceph-mon[31898]: pgmap v224: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:36.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:35 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:06:36.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:35 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:06:36.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:35 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:06:38.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:37 smithi114 ceph-mon[36376]: pgmap v225: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:38.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:37 smithi161 ceph-mon[36332]: pgmap v225: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:38.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:37 smithi097 ceph-mon[31898]: pgmap v225: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:40.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:39 smithi097 ceph-mon[31898]: pgmap v226: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:40.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:39 smithi114 ceph-mon[36376]: pgmap v226: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:40.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:39 smithi161 ceph-mon[36332]: pgmap v226: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:42.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:41 smithi114 ceph-mon[36376]: pgmap v227: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:42.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:41 smithi161 ceph-mon[36332]: pgmap v227: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:42.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:41 smithi097 ceph-mon[31898]: pgmap v227: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:44.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:43 smithi114 ceph-mon[36376]: pgmap v228: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:44.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:43 smithi161 ceph-mon[36332]: pgmap v228: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:44.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:43 smithi097 ceph-mon[31898]: pgmap v228: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:46.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:45 smithi114 ceph-mon[36376]: pgmap v229: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:46.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:45 smithi161 ceph-mon[36332]: pgmap v229: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:46.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:45 smithi097 ceph-mon[31898]: pgmap v229: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:48.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:47 smithi114 ceph-mon[36376]: pgmap v230: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:48.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:47 smithi161 ceph-mon[36332]: pgmap v230: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:48.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:47 smithi097 ceph-mon[31898]: pgmap v230: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:50.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:49 smithi097 ceph-mon[31898]: pgmap v231: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:50.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:49 smithi114 ceph-mon[36376]: pgmap v231: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:50.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:49 smithi161 ceph-mon[36332]: pgmap v231: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:52.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:51 smithi114 ceph-mon[36376]: pgmap v232: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:52.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:51 smithi161 ceph-mon[36332]: pgmap v232: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:52.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:51 smithi097 ceph-mon[31898]: pgmap v232: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:53.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:52 smithi114 ceph-mon[36376]: pgmap v233: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:53.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:52 smithi161 ceph-mon[36332]: pgmap v233: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:53.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:52 smithi097 ceph-mon[31898]: pgmap v233: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:06:56.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:55 smithi114 ceph-mon[36376]: pgmap v234: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:56.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:55 smithi161 ceph-mon[36332]: pgmap v234: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:56.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:55 smithi097 ceph-mon[31898]: pgmap v234: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:58.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:57 smithi114 ceph-mon[36376]: pgmap v235: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:58.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:57 smithi161 ceph-mon[36332]: pgmap v235: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:06:58.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:57 smithi097 ceph-mon[31898]: pgmap v235: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:00.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:06:59 smithi114 ceph-mon[36376]: pgmap v236: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:00.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:06:59 smithi161 ceph-mon[36332]: pgmap v236: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:00.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:06:59 smithi097 ceph-mon[31898]: pgmap v236: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:02.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:01 smithi114 ceph-mon[36376]: pgmap v237: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:02.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:01 smithi161 ceph-mon[36332]: pgmap v237: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:02.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:01 smithi097 ceph-mon[31898]: pgmap v237: 129 pgs: 129 active+clean; 259 MiB data, 925 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:04.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:03 smithi114 ceph-mon[36376]: pgmap v238: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:07:04.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:03 smithi161 ceph-mon[36332]: pgmap v238: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:07:04.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:03 smithi097 ceph-mon[31898]: pgmap v238: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:07:05.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:05 smithi161 ceph-mon[36332]: pgmap v239: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:07:05.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:05 smithi097 ceph-mon[31898]: pgmap v239: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:07:05.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:05 smithi114 ceph-mon[36376]: pgmap v239: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:07:08.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:07 smithi114 ceph-mon[36376]: pgmap v240: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:07:08.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:07 smithi161 ceph-mon[36332]: pgmap v240: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:07:08.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:07 smithi097 ceph-mon[31898]: pgmap v240: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:07:10.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:09 smithi114 ceph-mon[36376]: pgmap v241: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:07:10.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:09 smithi161 ceph-mon[36332]: pgmap v241: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:07:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:09 smithi097 ceph-mon[31898]: pgmap v241: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:07:11.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:10 smithi114 ceph-mon[36376]: osdmap e97: 12 total, 12 up, 12 in 2024-03-11T20:07:11.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:10 smithi161 ceph-mon[36332]: osdmap e97: 12 total, 12 up, 12 in 2024-03-11T20:07:11.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:10 smithi097 ceph-mon[31898]: osdmap e97: 12 total, 12 up, 12 in 2024-03-11T20:07:12.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:11 smithi114 ceph-mon[36376]: pgmap v243: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-11T20:07:12.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:11 smithi161 ceph-mon[36332]: pgmap v243: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-11T20:07:12.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:11 smithi097 ceph-mon[31898]: pgmap v243: 129 pgs: 129 active+clean; 259 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-11T20:07:13.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:12 smithi114 ceph-mon[36376]: pgmap v244: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-11T20:07:13.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:12 smithi161 ceph-mon[36332]: pgmap v244: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-11T20:07:13.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:12 smithi097 ceph-mon[31898]: pgmap v244: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-11T20:07:15.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:14 smithi097 ceph-mon[31898]: osdmap e98: 12 total, 12 up, 12 in 2024-03-11T20:07:15.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:14 smithi114 ceph-mon[36376]: osdmap e98: 12 total, 12 up, 12 in 2024-03-11T20:07:15.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:14 smithi161 ceph-mon[36332]: osdmap e98: 12 total, 12 up, 12 in 2024-03-11T20:07:16.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:15 smithi097 ceph-mon[31898]: pgmap v246: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:07:16.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:15 smithi114 ceph-mon[36376]: pgmap v246: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:07:16.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:15 smithi161 ceph-mon[36332]: pgmap v246: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:07:18.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:17 smithi097 ceph-mon[31898]: pgmap v247: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-11T20:07:18.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:17 smithi114 ceph-mon[36376]: pgmap v247: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-11T20:07:18.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:17 smithi161 ceph-mon[36332]: pgmap v247: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-11T20:07:20.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:19 smithi097 ceph-mon[31898]: pgmap v248: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 229 B/s wr, 11 op/s 2024-03-11T20:07:20.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:19 smithi114 ceph-mon[36376]: pgmap v248: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 229 B/s wr, 11 op/s 2024-03-11T20:07:20.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:19 smithi161 ceph-mon[36332]: pgmap v248: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 229 B/s wr, 11 op/s 2024-03-11T20:07:22.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:21 smithi097 ceph-mon[31898]: pgmap v249: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:07:22.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:21 smithi114 ceph-mon[36376]: pgmap v249: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:07:22.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:21 smithi161 ceph-mon[36332]: pgmap v249: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:07:24.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:23 smithi097 ceph-mon[31898]: pgmap v250: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-11T20:07:24.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:23 smithi114 ceph-mon[36376]: pgmap v250: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-11T20:07:24.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:23 smithi161 ceph-mon[36332]: pgmap v250: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-11T20:07:26.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:25 smithi097 ceph-mon[31898]: pgmap v251: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-11T20:07:26.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:25 smithi114 ceph-mon[36376]: pgmap v251: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-11T20:07:26.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:25 smithi161 ceph-mon[36332]: pgmap v251: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-11T20:07:28.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:27 smithi114 ceph-mon[36376]: pgmap v252: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:07:28.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:27 smithi161 ceph-mon[36332]: pgmap v252: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:07:28.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:27 smithi097 ceph-mon[31898]: pgmap v252: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:07:30.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:29 smithi114 ceph-mon[36376]: pgmap v253: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:07:30.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:29 smithi161 ceph-mon[36332]: pgmap v253: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:07:30.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:29 smithi097 ceph-mon[31898]: pgmap v253: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:07:32.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:31 smithi114 ceph-mon[36376]: pgmap v254: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:32.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:31 smithi161 ceph-mon[36332]: pgmap v254: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:32.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:31 smithi097 ceph-mon[31898]: pgmap v254: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:34.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:33 smithi114 ceph-mon[36376]: pgmap v255: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:34.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:33 smithi161 ceph-mon[36332]: pgmap v255: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:34.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:33 smithi097 ceph-mon[31898]: pgmap v255: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:36.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:35 smithi114 ceph-mon[36376]: pgmap v256: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:36.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:35 smithi161 ceph-mon[36332]: pgmap v256: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:36.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:35 smithi097 ceph-mon[31898]: pgmap v256: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:37.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:36 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:07:37.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:36 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:07:37.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:36 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:07:38.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:37 smithi114 ceph-mon[36376]: pgmap v257: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:38.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:37 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:07:38.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:37 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:07:38.230 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:37 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:07:38.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:37 smithi161 ceph-mon[36332]: pgmap v257: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:38.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:37 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:07:38.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:37 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:07:38.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:37 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:07:38.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:37 smithi097 ceph-mon[31898]: pgmap v257: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:38.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:37 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:07:38.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:37 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:07:38.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:37 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:07:40.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:39 smithi114 ceph-mon[36376]: pgmap v258: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:40.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:39 smithi161 ceph-mon[36332]: pgmap v258: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:40.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:39 smithi097 ceph-mon[31898]: pgmap v258: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:42.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:41 smithi114 ceph-mon[36376]: pgmap v259: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:42.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:41 smithi161 ceph-mon[36332]: pgmap v259: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:42.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:41 smithi097 ceph-mon[31898]: pgmap v259: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:43.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:42 smithi114 ceph-mon[36376]: pgmap v260: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:43.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:42 smithi161 ceph-mon[36332]: pgmap v260: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:43.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:42 smithi097 ceph-mon[31898]: pgmap v260: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:46.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:45 smithi114 ceph-mon[36376]: pgmap v261: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:46.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:45 smithi161 ceph-mon[36332]: pgmap v261: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:46.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:45 smithi097 ceph-mon[31898]: pgmap v261: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:48.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:47 smithi114 ceph-mon[36376]: pgmap v262: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:48.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:47 smithi161 ceph-mon[36332]: pgmap v262: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:48.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:47 smithi097 ceph-mon[31898]: pgmap v262: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:49.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:49 smithi161 ceph-mon[36332]: pgmap v263: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:49.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:49 smithi097 ceph-mon[31898]: pgmap v263: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:49.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:49 smithi114 ceph-mon[36376]: pgmap v263: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:52.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:51 smithi114 ceph-mon[36376]: pgmap v264: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:52.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:51 smithi161 ceph-mon[36332]: pgmap v264: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:52.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:51 smithi097 ceph-mon[31898]: pgmap v264: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:54.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:53 smithi114 ceph-mon[36376]: pgmap v265: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:54.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:53 smithi161 ceph-mon[36332]: pgmap v265: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:54.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:53 smithi097 ceph-mon[31898]: pgmap v265: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:55.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:54 smithi114 ceph-mon[36376]: pgmap v266: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:55.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:54 smithi161 ceph-mon[36332]: pgmap v266: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:55.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:54 smithi097 ceph-mon[31898]: pgmap v266: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:58.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:57 smithi114 ceph-mon[36376]: pgmap v267: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:58.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:57 smithi161 ceph-mon[36332]: pgmap v267: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:58.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:57 smithi097 ceph-mon[31898]: pgmap v267: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:07:59.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:07:59 smithi161 ceph-mon[36332]: pgmap v268: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:59.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:07:59 smithi097 ceph-mon[31898]: pgmap v268: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:07:59.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:07:59 smithi114 ceph-mon[36376]: pgmap v268: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:02.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:01 smithi114 ceph-mon[36376]: pgmap v269: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:02.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:01 smithi161 ceph-mon[36332]: pgmap v269: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:02.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:01 smithi097 ceph-mon[31898]: pgmap v269: 129 pgs: 129 active+clean; 2.3 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:03.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:03 smithi161 ceph-mon[36332]: pgmap v270: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:08:03.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:03 smithi097 ceph-mon[31898]: pgmap v270: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:08:03.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:03 smithi114 ceph-mon[36376]: pgmap v270: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:08:06.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:05 smithi114 ceph-mon[36376]: pgmap v271: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:08:06.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:05 smithi161 ceph-mon[36332]: pgmap v271: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:08:06.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:05 smithi097 ceph-mon[31898]: pgmap v271: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:08:07.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:06 smithi114 ceph-mon[36376]: pgmap v272: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:08:07.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:06 smithi161 ceph-mon[36332]: pgmap v272: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:08:07.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:06 smithi097 ceph-mon[31898]: pgmap v272: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:08:10.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:09 smithi114 ceph-mon[36376]: pgmap v273: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:08:10.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:09 smithi161 ceph-mon[36332]: pgmap v273: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:08:10.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:09 smithi097 ceph-mon[31898]: pgmap v273: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:08:11.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:10 smithi161 ceph-mon[36332]: osdmap e99: 12 total, 12 up, 12 in 2024-03-11T20:08:11.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:10 smithi161 ceph-mon[36332]: pgmap v275: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-11T20:08:11.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:10 smithi097 ceph-mon[31898]: osdmap e99: 12 total, 12 up, 12 in 2024-03-11T20:08:11.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:10 smithi097 ceph-mon[31898]: pgmap v275: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-11T20:08:11.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:10 smithi114 ceph-mon[36376]: osdmap e99: 12 total, 12 up, 12 in 2024-03-11T20:08:11.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:10 smithi114 ceph-mon[36376]: pgmap v275: 129 pgs: 129 active+clean; 2.3 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-11T20:08:13.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:13 smithi161 ceph-mon[36332]: pgmap v276: 129 pgs: 129 active+clean; 2.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:08:13.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:13 smithi097 ceph-mon[31898]: pgmap v276: 129 pgs: 129 active+clean; 2.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:08:13.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:13 smithi114 ceph-mon[36376]: pgmap v276: 129 pgs: 129 active+clean; 2.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:08:15.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:14 smithi097 ceph-mon[31898]: osdmap e100: 12 total, 12 up, 12 in 2024-03-11T20:08:15.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:14 smithi114 ceph-mon[36376]: osdmap e100: 12 total, 12 up, 12 in 2024-03-11T20:08:15.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:14 smithi161 ceph-mon[36332]: osdmap e100: 12 total, 12 up, 12 in 2024-03-11T20:08:16.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:15 smithi097 ceph-mon[31898]: pgmap v278: 129 pgs: 129 active+clean; 2.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:08:16.229 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:15 smithi114 ceph-mon[36376]: pgmap v278: 129 pgs: 129 active+clean; 2.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:08:16.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:15 smithi161 ceph-mon[36332]: pgmap v278: 129 pgs: 129 active+clean; 2.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:08:17.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:17 smithi161 ceph-mon[36332]: pgmap v279: 129 pgs: 129 active+clean; 2.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-11T20:08:17.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:17 smithi097 ceph-mon[31898]: pgmap v279: 129 pgs: 129 active+clean; 2.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-11T20:08:17.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:17 smithi114 ceph-mon[36376]: pgmap v279: 129 pgs: 129 active+clean; 2.3 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-11T20:08:19.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:19 smithi161 ceph-mon[36332]: pgmap v280: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2024-03-11T20:08:19.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:19 smithi097 ceph-mon[31898]: pgmap v280: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2024-03-11T20:08:19.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:19 smithi114 ceph-mon[36376]: pgmap v280: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2024-03-11T20:08:21.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:21 smithi161 ceph-mon[36332]: pgmap v281: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:08:21.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:21 smithi097 ceph-mon[31898]: pgmap v281: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:08:21.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:21 smithi114 ceph-mon[36376]: pgmap v281: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:08:23.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:23 smithi161 ceph-mon[36332]: pgmap v282: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-11T20:08:23.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:23 smithi097 ceph-mon[31898]: pgmap v282: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-11T20:08:23.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:23 smithi114 ceph-mon[36376]: pgmap v282: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-11T20:08:25.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:25 smithi161 ceph-mon[36332]: pgmap v283: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 92 B/s wr, 9 op/s 2024-03-11T20:08:25.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:25 smithi097 ceph-mon[31898]: pgmap v283: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 92 B/s wr, 9 op/s 2024-03-11T20:08:25.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:25 smithi114 ceph-mon[36376]: pgmap v283: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 92 B/s wr, 9 op/s 2024-03-11T20:08:27.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:27 smithi161 ceph-mon[36332]: pgmap v284: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:08:27.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:27 smithi097 ceph-mon[31898]: pgmap v284: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:08:27.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:27 smithi114 ceph-mon[36376]: pgmap v284: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-11T20:08:29.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:29 smithi161 ceph-mon[36332]: pgmap v285: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:08:29.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:29 smithi097 ceph-mon[31898]: pgmap v285: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:08:29.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:29 smithi114 ceph-mon[36376]: pgmap v285: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-11T20:08:31.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:31 smithi161 ceph-mon[36332]: pgmap v286: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:31.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:31 smithi097 ceph-mon[31898]: pgmap v286: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:31.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:31 smithi114 ceph-mon[36376]: pgmap v286: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:33.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:33 smithi161 ceph-mon[36332]: pgmap v287: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:33.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:33 smithi097 ceph-mon[31898]: pgmap v287: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:33.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:33 smithi114 ceph-mon[36376]: pgmap v287: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:35.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:35 smithi161 ceph-mon[36332]: pgmap v288: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:35.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:35 smithi097 ceph-mon[31898]: pgmap v288: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:35.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:35 smithi114 ceph-mon[36376]: pgmap v288: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:37.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:37 smithi161 ceph-mon[36332]: pgmap v289: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:37.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:37 smithi097 ceph-mon[31898]: pgmap v289: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:37.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:37 smithi114 ceph-mon[36376]: pgmap v289: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:38.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:38 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:08:38.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:38 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:08:38.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:38 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:08:39.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:39 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:08:39.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:39 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:08:39.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:39 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:08:39.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:39 smithi161 ceph-mon[36332]: pgmap v290: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:39.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:39 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:08:39.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:39 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:08:39.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:39 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:08:39.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:39 smithi097 ceph-mon[31898]: pgmap v290: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:39.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:39 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:08:39.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:39 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:08:39.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:39 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:08:39.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:39 smithi114 ceph-mon[36376]: pgmap v290: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:41.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:41 smithi161 ceph-mon[36332]: pgmap v291: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:41.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:41 smithi097 ceph-mon[31898]: pgmap v291: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:41.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:41 smithi114 ceph-mon[36376]: pgmap v291: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:43.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:43 smithi161 ceph-mon[36332]: pgmap v292: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:43.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:43 smithi097 ceph-mon[31898]: pgmap v292: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:43.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:43 smithi114 ceph-mon[36376]: pgmap v292: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:45.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:45 smithi161 ceph-mon[36332]: pgmap v293: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:45.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:45 smithi097 ceph-mon[31898]: pgmap v293: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:45.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:45 smithi114 ceph-mon[36376]: pgmap v293: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:47.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:47 smithi161 ceph-mon[36332]: pgmap v294: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:47.281 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:47 smithi114 ceph-mon[36376]: pgmap v294: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:47.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:47 smithi097 ceph-mon[31898]: pgmap v294: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:49.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:49 smithi161 ceph-mon[36332]: pgmap v295: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:49.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:49 smithi097 ceph-mon[31898]: pgmap v295: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:49.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:49 smithi114 ceph-mon[36376]: pgmap v295: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:51.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:51 smithi161 ceph-mon[36332]: pgmap v296: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:51.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:51 smithi097 ceph-mon[31898]: pgmap v296: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:51.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:51 smithi114 ceph-mon[36376]: pgmap v296: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:53.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:53 smithi161 ceph-mon[36332]: pgmap v297: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:53.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:53 smithi097 ceph-mon[31898]: pgmap v297: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:53.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:53 smithi114 ceph-mon[36376]: pgmap v297: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:55.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:55 smithi161 ceph-mon[36332]: pgmap v298: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:55.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:55 smithi097 ceph-mon[31898]: pgmap v298: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:55.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:55 smithi114 ceph-mon[36376]: pgmap v298: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:57.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:57 smithi161 ceph-mon[36332]: pgmap v299: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:57.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:57 smithi097 ceph-mon[31898]: pgmap v299: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:57.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:57 smithi114 ceph-mon[36376]: pgmap v299: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:08:59.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:08:59 smithi161 ceph-mon[36332]: pgmap v300: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:59.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:08:59 smithi097 ceph-mon[31898]: pgmap v300: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:08:59.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:08:59 smithi114 ceph-mon[36376]: pgmap v300: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:01.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:01 smithi161 ceph-mon[36332]: pgmap v301: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:01.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:01 smithi097 ceph-mon[31898]: pgmap v301: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:01.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:01 smithi114 ceph-mon[36376]: pgmap v301: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:03.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:03 smithi161 ceph-mon[36332]: pgmap v302: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:09:03.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:03 smithi097 ceph-mon[31898]: pgmap v302: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:09:03.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:03 smithi114 ceph-mon[36376]: pgmap v302: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:09:05.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:05 smithi161 ceph-mon[36332]: pgmap v303: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:09:05.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:05 smithi097 ceph-mon[31898]: pgmap v303: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:09:05.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:05 smithi114 ceph-mon[36376]: pgmap v303: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:09:07.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:07 smithi161 ceph-mon[36332]: pgmap v304: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:09:07.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:07 smithi097 ceph-mon[31898]: pgmap v304: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:09:07.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:07 smithi114 ceph-mon[36376]: pgmap v304: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:09:09.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:09 smithi161 ceph-mon[36332]: pgmap v305: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:09:09.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:09 smithi097 ceph-mon[31898]: pgmap v305: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:09:09.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:09 smithi114 ceph-mon[36376]: pgmap v305: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:09:11.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:11 smithi097 ceph-mon[31898]: osdmap e101: 12 total, 12 up, 12 in 2024-03-11T20:09:11.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:11 smithi097 ceph-mon[31898]: pgmap v307: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-11T20:09:11.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:11 smithi114 ceph-mon[36376]: osdmap e101: 12 total, 12 up, 12 in 2024-03-11T20:09:11.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:11 smithi114 ceph-mon[36376]: pgmap v307: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-11T20:09:11.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:11 smithi161 ceph-mon[36332]: osdmap e101: 12 total, 12 up, 12 in 2024-03-11T20:09:11.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:11 smithi161 ceph-mon[36332]: pgmap v307: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-11T20:09:13.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:13 smithi161 ceph-mon[36332]: pgmap v308: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:09:13.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:13 smithi097 ceph-mon[31898]: pgmap v308: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:09:13.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:13 smithi114 ceph-mon[36376]: pgmap v308: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:09:15.228 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:14 smithi114 ceph-mon[36376]: osdmap e102: 12 total, 12 up, 12 in 2024-03-11T20:09:15.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:14 smithi161 ceph-mon[36332]: osdmap e102: 12 total, 12 up, 12 in 2024-03-11T20:09:15.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:14 smithi097 ceph-mon[31898]: osdmap e102: 12 total, 12 up, 12 in 2024-03-11T20:09:16.228 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:15 smithi114 ceph-mon[36376]: pgmap v310: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:09:16.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:15 smithi161 ceph-mon[36332]: pgmap v310: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:09:16.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:15 smithi097 ceph-mon[31898]: pgmap v310: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-11T20:09:17.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:17 smithi161 ceph-mon[36332]: pgmap v311: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-11T20:09:17.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:17 smithi097 ceph-mon[31898]: pgmap v311: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-11T20:09:17.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:17 smithi114 ceph-mon[36376]: pgmap v311: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-11T20:09:19.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:19 smithi161 ceph-mon[36332]: pgmap v312: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2024-03-11T20:09:19.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:19 smithi097 ceph-mon[31898]: pgmap v312: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2024-03-11T20:09:19.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:19 smithi114 ceph-mon[36376]: pgmap v312: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2024-03-11T20:09:20.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-11T20:09:20.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.284+0000 7f666dc00640 1 -- 172.21.15.97:0/3995876481 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6668106470 msgr2=0x7f666810e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:20.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.284+0000 7f666dc00640 1 --2- 172.21.15.97:0/3995876481 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6668106470 0x7f666810e040 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6660009cd0 tx=0x7f666002c0f0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.284+0000 7f666dc00640 1 -- 172.21.15.97:0/3995876481 shutdown_connections 2024-03-11T20:09:20.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.284+0000 7f666dc00640 1 --2- 172.21.15.97:0/3995876481 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6668106470 0x7f666810e040 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.284+0000 7f666dc00640 1 --2- 172.21.15.97:0/3995876481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6668101d50 0x7f6668105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.284+0000 7f666dc00640 1 --2- 172.21.15.97:0/3995876481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66681013a0 0x7f6668101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.284+0000 7f666dc00640 1 -- 172.21.15.97:0/3995876481 >> 172.21.15.97:0/3995876481 conn(0x7f66680fcc40 msgr2=0x7f66680ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:20.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.284+0000 7f666dc00640 1 -- 172.21.15.97:0/3995876481 shutdown_connections 2024-03-11T20:09:20.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.284+0000 7f666dc00640 1 -- 172.21.15.97:0/3995876481 wait complete. 2024-03-11T20:09:20.277 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.285+0000 7f666dc00640 1 Processor -- start 2024-03-11T20:09:20.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.285+0000 7f666dc00640 1 -- start start 2024-03-11T20:09:20.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.285+0000 7f666dc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66681013a0 0x7f6668197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:20.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.285+0000 7f666dc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6668106470 0x7f6668197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:20.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f666dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666819bfb0 0x7f666819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:20.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f666dc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666819c9a0 con 0x7f6668106470 2024-03-11T20:09:20.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f666dc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666819cb10 con 0x7f666819bfb0 2024-03-11T20:09:20.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f666dc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666819cc80 con 0x7f66681013a0 2024-03-11T20:09:20.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f6666a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66681013a0 0x7f6668197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:20.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f6666000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6668106470 0x7f6668197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:20.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f6666a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66681013a0 0x7f6668197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:45748/0 (socket says 172.21.15.97:45748) 2024-03-11T20:09:20.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f6666a00640 1 -- 172.21.15.97:0/3362748995 learned_addr learned my addr 172.21.15.97:0/3362748995 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:09:20.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f6666000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6668106470 0x7f6668197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44094/0 (socket says 172.21.15.97:44094) 2024-03-11T20:09:20.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f6667400640 1 --2- 172.21.15.97:0/3362748995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666819bfb0 0x7f666819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:20.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f6666a00640 1 -- 172.21.15.97:0/3362748995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666819bfb0 msgr2=0x7f666819c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:20.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f6666a00640 1 --2- 172.21.15.97:0/3362748995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666819bfb0 0x7f666819c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f6666a00640 1 -- 172.21.15.97:0/3362748995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6668106470 msgr2=0x7f6668197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:20.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f6666a00640 1 --2- 172.21.15.97:0/3362748995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6668106470 0x7f6668197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.286+0000 7f6666a00640 1 -- 172.21.15.97:0/3362748995 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6660009900 con 0x7f66681013a0 2024-03-11T20:09:20.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.287+0000 7f6666a00640 1 --2- 172.21.15.97:0/3362748995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66681013a0 0x7f6668197740 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f665800e9f0 tx=0x7f665800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:20.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.287+0000 7f665f400640 1 -- 172.21.15.97:0/3362748995 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6658009700 con 0x7f66681013a0 2024-03-11T20:09:20.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.287+0000 7f666dc00640 1 -- 172.21.15.97:0/3362748995 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66681a4260 con 0x7f66681013a0 2024-03-11T20:09:20.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.287+0000 7f665f400640 1 -- 172.21.15.97:0/3362748995 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f665800cd80 con 0x7f66681013a0 2024-03-11T20:09:20.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.287+0000 7f665f400640 1 -- 172.21.15.97:0/3362748995 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6658018920 con 0x7f66681013a0 2024-03-11T20:09:20.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.287+0000 7f666dc00640 1 -- 172.21.15.97:0/3362748995 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66681a4710 con 0x7f66681013a0 2024-03-11T20:09:20.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.288+0000 7f666dc00640 1 -- 172.21.15.97:0/3362748995 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f666804eee0 con 0x7f66681013a0 2024-03-11T20:09:20.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.290+0000 7f665f400640 1 -- 172.21.15.97:0/3362748995 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6658018a80 con 0x7f66681013a0 2024-03-11T20:09:20.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.291+0000 7f665f400640 1 --2- 172.21.15.97:0/3362748995 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f664c0a1c40 0x7f664c0a4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:20.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.291+0000 7f665f400640 1 -- 172.21.15.97:0/3362748995 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7f6658014070 con 0x7f66681013a0 2024-03-11T20:09:20.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.292+0000 7f6666000640 1 --2- 172.21.15.97:0/3362748995 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f664c0a1c40 0x7f664c0a4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:20.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.292+0000 7f6666000640 1 --2- 172.21.15.97:0/3362748995 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f664c0a1c40 0x7f664c0a4100 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f66681002e0 tx=0x7f6650005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:20.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.294+0000 7f665f400640 1 -- 172.21.15.97:0/3362748995 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6658096fa0 con 0x7f66681013a0 2024-03-11T20:09:20.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.513+0000 7f666dc00640 1 -- 172.21.15.97:0/3362748995 --> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f6668101e20 con 0x7f664c0a1c40 2024-03-11T20:09:20.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.527+0000 7f665f400640 1 -- 172.21.15.97:0/3362748995 <== mgr.15396 v2:172.21.15.97:6800/1085760108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7f6668101e20 con 0x7f664c0a1c40 2024-03-11T20:09:20.517 INFO:teuthology.orchestra.run.smithi097.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-11T00:00:00", "created": "2024-03-11T19:59:17", "first": "2024-03-11T20:00:00", "last": "2024-03-11T20:09:00", "last_pruned": "2024-03-11T20:09:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-11T20:09:20.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.530+0000 7f666dc00640 1 -- 172.21.15.97:0/3362748995 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f664c0a1c40 msgr2=0x7f664c0a4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:20.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.530+0000 7f666dc00640 1 --2- 172.21.15.97:0/3362748995 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f664c0a1c40 0x7f664c0a4100 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f66681002e0 tx=0x7f6650005f90 comp rx=0 tx=0).stop 2024-03-11T20:09:20.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.530+0000 7f666dc00640 1 -- 172.21.15.97:0/3362748995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66681013a0 msgr2=0x7f6668197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:20.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.530+0000 7f666dc00640 1 --2- 172.21.15.97:0/3362748995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66681013a0 0x7f6668197740 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f665800e9f0 tx=0x7f665800eeb0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.530+0000 7f666dc00640 1 -- 172.21.15.97:0/3362748995 shutdown_connections 2024-03-11T20:09:20.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.530+0000 7f666dc00640 1 --2- 172.21.15.97:0/3362748995 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f664c0a1c40 0x7f664c0a4100 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.530+0000 7f666dc00640 1 --2- 172.21.15.97:0/3362748995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f666819bfb0 0x7f666819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.530+0000 7f666dc00640 1 --2- 172.21.15.97:0/3362748995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6668106470 0x7f6668197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.530+0000 7f666dc00640 1 --2- 172.21.15.97:0/3362748995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66681013a0 0x7f6668197740 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.530+0000 7f666dc00640 1 -- 172.21.15.97:0/3362748995 >> 172.21.15.97:0/3362748995 conn(0x7f66680fcc40 msgr2=0x7f66680fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:20.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.530+0000 7f666dc00640 1 -- 172.21.15.97:0/3362748995 shutdown_connections 2024-03-11T20:09:20.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.530+0000 7f666dc00640 1 -- 172.21.15.97:0/3362748995 wait complete. 2024-03-11T20:09:20.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-11T20:09:20.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.750+0000 7f9eeb200640 1 -- 172.21.15.97:0/3416269025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee41013a0 msgr2=0x7f9ee4101780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:20.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.750+0000 7f9eeb200640 1 --2- 172.21.15.97:0/3416269025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee41013a0 0x7f9ee4101780 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009930 tx=0x7f9ed402bc30 comp rx=0 tx=0).stop 2024-03-11T20:09:20.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.750+0000 7f9eeb200640 1 -- 172.21.15.97:0/3416269025 shutdown_connections 2024-03-11T20:09:20.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.750+0000 7f9eeb200640 1 --2- 172.21.15.97:0/3416269025 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ee4106470 0x7f9ee410e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.750+0000 7f9eeb200640 1 --2- 172.21.15.97:0/3416269025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ee4101d50 0x7f9ee4105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.750+0000 7f9eeb200640 1 --2- 172.21.15.97:0/3416269025 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee41013a0 0x7f9ee4101780 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.750+0000 7f9eeb200640 1 -- 172.21.15.97:0/3416269025 >> 172.21.15.97:0/3416269025 conn(0x7f9ee40fcc40 msgr2=0x7f9ee40ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:20.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.750+0000 7f9eeb200640 1 -- 172.21.15.97:0/3416269025 shutdown_connections 2024-03-11T20:09:20.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.750+0000 7f9eeb200640 1 -- 172.21.15.97:0/3416269025 wait complete. 2024-03-11T20:09:20.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.751+0000 7f9eeb200640 1 Processor -- start 2024-03-11T20:09:20.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.751+0000 7f9eeb200640 1 -- start start 2024-03-11T20:09:20.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.751+0000 7f9eeb200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ee41013a0 0x7f9ee419f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:20.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.751+0000 7f9eeb200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee4106470 0x7f9ee419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:20.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.751+0000 7f9eeb200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ee4199800 0x7f9ee4199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:20.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.751+0000 7f9eeb200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee419a2b0 con 0x7f9ee4106470 2024-03-11T20:09:20.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.751+0000 7f9eeb200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee419a420 con 0x7f9ee41013a0 2024-03-11T20:09:20.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.751+0000 7f9eeb200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee419a590 con 0x7f9ee4199800 2024-03-11T20:09:20.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.752+0000 7f9ee3400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee4106470 0x7f9ee419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:20.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.752+0000 7f9ee3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ee41013a0 0x7f9ee419f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.752+0000 7f9ee3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ee41013a0 0x7f9ee419f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:48784/0 (socket says 172.21.15.97:48784) 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.752+0000 7f9ee3e00640 1 -- 172.21.15.97:0/2781408898 learned_addr learned my addr 172.21.15.97:0/2781408898 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.752+0000 7f9ee9200640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ee4199800 0x7f9ee4199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.752+0000 7f9ee3e00640 1 -- 172.21.15.97:0/2781408898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ee4199800 msgr2=0x7f9ee4199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.752+0000 7f9ee3e00640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ee4199800 0x7f9ee4199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.752+0000 7f9ee3e00640 1 -- 172.21.15.97:0/2781408898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee4106470 msgr2=0x7f9ee419fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.752+0000 7f9ee3e00640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee4106470 0x7f9ee419fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.752+0000 7f9ee3e00640 1 -- 172.21.15.97:0/2781408898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed4009560 con 0x7f9ee41013a0 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.752+0000 7f9ee9200640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ee4199800 0x7f9ee4199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.752+0000 7f9ee3e00640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ee41013a0 0x7f9ee419f640 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009230 tx=0x7f9ed402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.753+0000 7f9ee0c00640 1 -- 172.21.15.97:0/2781408898 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4004350 con 0x7f9ee41013a0 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.753+0000 7f9ee0c00640 1 -- 172.21.15.97:0/2781408898 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ed40044b0 con 0x7f9ee41013a0 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.753+0000 7f9eeb200640 1 -- 172.21.15.97:0/2781408898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee41a6490 con 0x7f9ee41013a0 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.753+0000 7f9ee0c00640 1 -- 172.21.15.97:0/2781408898 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4003d00 con 0x7f9ee41013a0 2024-03-11T20:09:20.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.753+0000 7f9eeb200640 1 -- 172.21.15.97:0/2781408898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee41a6970 con 0x7f9ee41013a0 2024-03-11T20:09:20.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.754+0000 7f9eeb200640 1 -- 172.21.15.97:0/2781408898 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee4102870 con 0x7f9ee41013a0 2024-03-11T20:09:20.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.756+0000 7f9ee0c00640 1 -- 172.21.15.97:0/2781408898 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9ed4004df0 con 0x7f9ee41013a0 2024-03-11T20:09:20.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.757+0000 7f9ee0c00640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9ec40a1c40 0x7f9ec40a4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:20.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.757+0000 7f9ee0c00640 1 -- 172.21.15.97:0/2781408898 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7f9ed40efbc0 con 0x7f9ee41013a0 2024-03-11T20:09:20.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.757+0000 7f9ee3400640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9ec40a1c40 0x7f9ec40a4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:20.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.758+0000 7f9ee3400640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9ec40a1c40 0x7f9ec40a4100 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9ee41006b0 tx=0x7f9ecc009500 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:20.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.762+0000 7f9ee0c00640 1 -- 172.21.15.97:0/2781408898 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9ed40b8750 con 0x7f9ee41013a0 2024-03-11T20:09:20.972 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.982+0000 7f9eeb200640 1 -- 172.21.15.97:0/2781408898 --> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9ee4101dc0 con 0x7f9ec40a1c40 2024-03-11T20:09:20.985 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.995+0000 7f9ee0c00640 1 -- 172.21.15.97:0/2781408898 <== mgr.15396 v2:172.21.15.97:6800/1085760108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f9ee4101dc0 con 0x7f9ec40a1c40 2024-03-11T20:09:20.985 INFO:teuthology.orchestra.run.smithi097.stdout:/ 1m 6m3h 2024-03-11T20:09:20.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.998+0000 7f9eeb200640 1 -- 172.21.15.97:0/2781408898 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9ec40a1c40 msgr2=0x7f9ec40a4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:20.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.998+0000 7f9eeb200640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9ec40a1c40 0x7f9ec40a4100 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9ee41006b0 tx=0x7f9ecc009500 comp rx=0 tx=0).stop 2024-03-11T20:09:20.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.998+0000 7f9eeb200640 1 -- 172.21.15.97:0/2781408898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ee41013a0 msgr2=0x7f9ee419f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:20.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.998+0000 7f9eeb200640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ee41013a0 0x7f9ee419f640 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009230 tx=0x7f9ed402c650 comp rx=0 tx=0).stop 2024-03-11T20:09:20.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.998+0000 7f9eeb200640 1 -- 172.21.15.97:0/2781408898 shutdown_connections 2024-03-11T20:09:20.988 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.998+0000 7f9eeb200640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9ec40a1c40 0x7f9ec40a4100 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.998+0000 7f9eeb200640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ee4199800 0x7f9ee4199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.998+0000 7f9eeb200640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee4106470 0x7f9ee419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.998+0000 7f9eeb200640 1 --2- 172.21.15.97:0/2781408898 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ee41013a0 0x7f9ee419f640 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:20.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.998+0000 7f9eeb200640 1 -- 172.21.15.97:0/2781408898 >> 172.21.15.97:0/2781408898 conn(0x7f9ee40fcc40 msgr2=0x7f9ee40fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:20.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.998+0000 7f9eeb200640 1 -- 172.21.15.97:0/2781408898 shutdown_connections 2024-03-11T20:09:20.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:20.998+0000 7f9eeb200640 1 -- 172.21.15.97:0/2781408898 wait complete. 2024-03-11T20:09:21.007 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-11T20:09:21.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T20:09:21.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.237+0000 7f192b400640 1 -- 172.21.15.97:0/723279175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19241080f0 msgr2=0x7f1924114600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:21.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.237+0000 7f192b400640 1 --2- 172.21.15.97:0/723279175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19241080f0 0x7f1924114600 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f191c009cd0 tx=0x7f191c02c0e0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.237+0000 7f192b400640 1 -- 172.21.15.97:0/723279175 shutdown_connections 2024-03-11T20:09:21.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.237+0000 7f192b400640 1 --2- 172.21.15.97:0/723279175 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19241080f0 0x7f1924114600 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.237+0000 7f192b400640 1 --2- 172.21.15.97:0/723279175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1924107730 0x7f1924107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.237+0000 7f192b400640 1 --2- 172.21.15.97:0/723279175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f192410d740 0x7f192410db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.237+0000 7f192b400640 1 -- 172.21.15.97:0/723279175 >> 172.21.15.97:0/723279175 conn(0x7f19240756d0 msgr2=0x7f1924077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.237+0000 7f192b400640 1 -- 172.21.15.97:0/723279175 shutdown_connections 2024-03-11T20:09:21.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.237+0000 7f192b400640 1 -- 172.21.15.97:0/723279175 wait complete. 2024-03-11T20:09:21.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.238+0000 7f192b400640 1 Processor -- start 2024-03-11T20:09:21.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.239+0000 7f192b400640 1 -- start start 2024-03-11T20:09:21.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.239+0000 7f192b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1924107730 0x7f192419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:21.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.239+0000 7f192b400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19241080f0 0x7f192419e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:21.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.239+0000 7f192b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f192410d740 0x7f19241a2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:21.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.239+0000 7f192b400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19241a2c80 con 0x7f19241080f0 2024-03-11T20:09:21.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.239+0000 7f192b400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19241a2df0 con 0x7f192410d740 2024-03-11T20:09:21.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.239+0000 7f192b400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19241a2fb0 con 0x7f1924107730 2024-03-11T20:09:21.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.239+0000 7f1923e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19241080f0 0x7f192419e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.239+0000 7f1928a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1924107730 0x7f192419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.239+0000 7f1928a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1924107730 0x7f192419de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:45796/0 (socket says 172.21.15.97:45796) 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.239+0000 7f1923e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19241080f0 0x7f192419e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44130/0 (socket says 172.21.15.97:44130) 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.239+0000 7f1928a00640 1 -- 172.21.15.97:0/1564985140 learned_addr learned my addr 172.21.15.97:0/1564985140 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.239+0000 7f1929400640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f192410d740 0x7f19241a2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.240+0000 7f1928a00640 1 -- 172.21.15.97:0/1564985140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f192410d740 msgr2=0x7f19241a2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.240+0000 7f1928a00640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f192410d740 0x7f19241a2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.240+0000 7f1928a00640 1 -- 172.21.15.97:0/1564985140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19241080f0 msgr2=0x7f192419e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.240+0000 7f1928a00640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19241080f0 0x7f192419e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.240+0000 7f1928a00640 1 -- 172.21.15.97:0/1564985140 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1918009490 con 0x7f1924107730 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.240+0000 7f1923e00640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19241080f0 0x7f192419e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.240+0000 7f1928a00640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1924107730 0x7f192419de10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f191800ea60 tx=0x7f191800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.240+0000 7f1921600640 1 -- 172.21.15.97:0/1564985140 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191800cde0 con 0x7f1924107730 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.241+0000 7f192b400640 1 -- 172.21.15.97:0/1564985140 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191c009900 con 0x7f1924107730 2024-03-11T20:09:21.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.241+0000 7f1921600640 1 -- 172.21.15.97:0/1564985140 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19180035f0 con 0x7f1924107730 2024-03-11T20:09:21.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.241+0000 7f1921600640 1 -- 172.21.15.97:0/1564985140 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1918010690 con 0x7f1924107730 2024-03-11T20:09:21.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.241+0000 7f192b400640 1 -- 172.21.15.97:0/1564985140 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19241aa930 con 0x7f1924107730 2024-03-11T20:09:21.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.242+0000 7f192b400640 1 -- 172.21.15.97:0/1564985140 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19240683c0 con 0x7f1924107730 2024-03-11T20:09:21.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.244+0000 7f1921600640 1 -- 172.21.15.97:0/1564985140 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f1918003af0 con 0x7f1924107730 2024-03-11T20:09:21.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.245+0000 7f1921600640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f190c0a1ba0 0x7f190c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:21.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.245+0000 7f1921600640 1 -- 172.21.15.97:0/1564985140 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7f1918014070 con 0x7f1924107730 2024-03-11T20:09:21.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.245+0000 7f1923e00640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f190c0a1ba0 0x7f190c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:21.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.246+0000 7f1923e00640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f190c0a1ba0 0x7f190c0a4060 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1924077d80 tx=0x7f1914005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:21.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.249+0000 7f1921600640 1 -- 172.21.15.97:0/1564985140 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1918096ac0 con 0x7f1924107730 2024-03-11T20:09:21.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:21 smithi161 ceph-mon[36332]: from='client.25756 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:09:21.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:21 smithi161 ceph-mon[36332]: pgmap v313: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:09:21.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:21 smithi097 ceph-mon[31898]: from='client.25756 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:09:21.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:21 smithi097 ceph-mon[31898]: pgmap v313: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:09:21.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.449+0000 7f192b400640 1 -- 172.21.15.97:0/1564985140 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f192410fca0 con 0x7f1924107730 2024-03-11T20:09:21.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.451+0000 7f1921600640 1 -- 172.21.15.97:0/1564985140 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+25400 (secure 0 0 0) 0x7f191809a180 con 0x7f1924107730 2024-03-11T20:09:21.442 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T20:09:21.442 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":102,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T20:09:13.920214+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/206978801":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/3511489140":"2024-03-12T19:59:16.643130+0000","172.21.15.97:6801/3389151509":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/3879069067":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6800/3389151509":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/839620487":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T20:09:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.454+0000 7f192b400640 1 -- 172.21.15.97:0/1564985140 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f190c0a1ba0 msgr2=0x7f190c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.454+0000 7f192b400640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f190c0a1ba0 0x7f190c0a4060 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1924077d80 tx=0x7f1914005f90 comp rx=0 tx=0).stop 2024-03-11T20:09:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.454+0000 7f192b400640 1 -- 172.21.15.97:0/1564985140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1924107730 msgr2=0x7f192419de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.454+0000 7f192b400640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1924107730 0x7f192419de10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f191800ea60 tx=0x7f191800ef20 comp rx=0 tx=0).stop 2024-03-11T20:09:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.454+0000 7f192b400640 1 -- 172.21.15.97:0/1564985140 shutdown_connections 2024-03-11T20:09:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.454+0000 7f192b400640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f190c0a1ba0 0x7f190c0a4060 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.454+0000 7f192b400640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f192410d740 0x7f19241a2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.454+0000 7f192b400640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f19241080f0 0x7f192419e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.454+0000 7f192b400640 1 --2- 172.21.15.97:0/1564985140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1924107730 0x7f192419de10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.454+0000 7f192b400640 1 -- 172.21.15.97:0/1564985140 >> 172.21.15.97:0/1564985140 conn(0x7f19240756d0 msgr2=0x7f1924075e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:21.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.454+0000 7f192b400640 1 -- 172.21.15.97:0/1564985140 shutdown_connections 2024-03-11T20:09:21.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.454+0000 7f192b400640 1 -- 172.21.15.97:0/1564985140 wait complete. 2024-03-11T20:09:21.464 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:50:29.051985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:55:43.424076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:55:44.652160+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T19:55:48.598077+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T20:09:21.465 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T20:09:21.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:21 smithi114 ceph-mon[36376]: from='client.25756 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:09:21.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:21 smithi114 ceph-mon[36376]: pgmap v313: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-11T20:09:21.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.691+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2847850387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ef0103f40 msgr2=0x7f9ef0107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:21.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.691+0000 7f9ef4a00640 1 --2- 172.21.15.97:0/2847850387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ef0103f40 0x7f9ef0107f80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9edc009930 tx=0x7f9edc02bc20 comp rx=0 tx=0).stop 2024-03-11T20:09:21.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.691+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2847850387 shutdown_connections 2024-03-11T20:09:21.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.691+0000 7f9ef4a00640 1 --2- 172.21.15.97:0/2847850387 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef0108660 0x7f9ef0110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.691+0000 7f9ef4a00640 1 --2- 172.21.15.97:0/2847850387 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ef0103f40 0x7f9ef0107f80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.691+0000 7f9ef4a00640 1 --2- 172.21.15.97:0/2847850387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ef0103590 0x7f9ef0103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.691+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2847850387 >> 172.21.15.97:0/2847850387 conn(0x7f9ef00fedd0 msgr2=0x7f9ef01011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:21.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.691+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2847850387 shutdown_connections 2024-03-11T20:09:21.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.691+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2847850387 wait complete. 2024-03-11T20:09:21.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.692+0000 7f9ef4a00640 1 Processor -- start 2024-03-11T20:09:21.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.692+0000 7f9ef4a00640 1 -- start start 2024-03-11T20:09:21.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.692+0000 7f9ef4a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef0103590 0x7f9ef019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:21.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.692+0000 7f9ef4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ef0108660 0x7f9ef019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:21.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.692+0000 7f9ef4a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ef0199890 0x7f9ef0199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:21.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.692+0000 7f9ef4a00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef019a340 con 0x7f9ef0199890 2024-03-11T20:09:21.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.692+0000 7f9ef4a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef019a4b0 con 0x7f9ef0108660 2024-03-11T20:09:21.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.692+0000 7f9ef4a00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef019a620 con 0x7f9ef0103590 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.693+0000 7f9eee800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ef0199890 0x7f9ef0199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.693+0000 7f9eee800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ef0199890 0x7f9ef0199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44156/0 (socket says 172.21.15.97:44156) 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.693+0000 7f9eee800640 1 -- 172.21.15.97:0/2598377434 learned_addr learned my addr 172.21.15.97:0/2598377434 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.693+0000 7f9eed400640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ef0108660 0x7f9ef019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.693+0000 7f9eede00640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef0103590 0x7f9ef019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.693+0000 7f9eee800640 1 -- 172.21.15.97:0/2598377434 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef0103590 msgr2=0x7f9ef019f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.693+0000 7f9eee800640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef0103590 0x7f9ef019f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.693+0000 7f9eee800640 1 -- 172.21.15.97:0/2598377434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ef0108660 msgr2=0x7f9ef019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.693+0000 7f9eee800640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ef0108660 0x7f9ef019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.693+0000 7f9eee800640 1 -- 172.21.15.97:0/2598377434 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc009560 con 0x7f9ef0199890 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.693+0000 7f9eed400640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ef0108660 0x7f9ef019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.694+0000 7f9eee800640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ef0199890 0x7f9ef0199d40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9ee800eb10 tx=0x7f9ee800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.694+0000 7f9ee6a00640 1 -- 172.21.15.97:0/2598377434 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8004b20 con 0x7f9ef0199890 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.694+0000 7f9ee6a00640 1 -- 172.21.15.97:0/2598377434 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ee8004c80 con 0x7f9ef0199890 2024-03-11T20:09:21.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.694+0000 7f9ee6a00640 1 -- 172.21.15.97:0/2598377434 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8010650 con 0x7f9ef0199890 2024-03-11T20:09:21.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.694+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2598377434 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef01a6580 con 0x7f9ef0199890 2024-03-11T20:09:21.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.694+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2598377434 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef01a6a60 con 0x7f9ef0199890 2024-03-11T20:09:21.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.695+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2598377434 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef0104a60 con 0x7f9ef0199890 2024-03-11T20:09:21.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.697+0000 7f9ee6a00640 1 -- 172.21.15.97:0/2598377434 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9ee8005020 con 0x7f9ef0199890 2024-03-11T20:09:21.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.699+0000 7f9ee6a00640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9ed40a1c20 0x7f9ed40a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:21.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.699+0000 7f9ee6a00640 1 -- 172.21.15.97:0/2598377434 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7f9ee8014070 con 0x7f9ef0199890 2024-03-11T20:09:21.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.700+0000 7f9eede00640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9ed40a1c20 0x7f9ed40a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:21.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.700+0000 7f9eede00640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9ed40a1c20 0x7f9ed40a40e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009bd0 tx=0x7f9ee0005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:21.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.704+0000 7f9ee6a00640 1 -- 172.21.15.97:0/2598377434 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9ee8096b50 con 0x7f9ef0199890 2024-03-11T20:09:21.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.907+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2598377434 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ef0103f40 con 0x7f9ef0199890 2024-03-11T20:09:21.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.908+0000 7f9ee6a00640 1 -- 172.21.15.97:0/2598377434 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ee809a210 con 0x7f9ef0199890 2024-03-11T20:09:21.899 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-11T20:09:21.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.910+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2598377434 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9ed40a1c20 msgr2=0x7f9ed40a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:21.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.910+0000 7f9ef4a00640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9ed40a1c20 0x7f9ed40a40e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009bd0 tx=0x7f9ee0005f90 comp rx=0 tx=0).stop 2024-03-11T20:09:21.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.911+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2598377434 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ef0199890 msgr2=0x7f9ef0199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:21.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.911+0000 7f9ef4a00640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ef0199890 0x7f9ef0199d40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9ee800eb10 tx=0x7f9ee800c2d0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.911+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2598377434 shutdown_connections 2024-03-11T20:09:21.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.911+0000 7f9ef4a00640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f9ed40a1c20 0x7f9ed40a40e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.911+0000 7f9ef4a00640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ef0199890 0x7f9ef0199d40 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.911+0000 7f9ef4a00640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ef0108660 0x7f9ef019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.911+0000 7f9ef4a00640 1 --2- 172.21.15.97:0/2598377434 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ef0103590 0x7f9ef019f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:21.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.911+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2598377434 >> 172.21.15.97:0/2598377434 conn(0x7f9ef00fedd0 msgr2=0x7f9ef0100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:21.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.911+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2598377434 shutdown_connections 2024-03-11T20:09:21.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:21.911+0000 7f9ef4a00640 1 -- 172.21.15.97:0/2598377434 wait complete. 2024-03-11T20:09:21.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T20:09:22.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.142+0000 7fa6a4400640 1 -- 172.21.15.97:0/1762309660 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa69c108660 msgr2=0x7fa69c110230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:22.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.142+0000 7fa6a4400640 1 --2- 172.21.15.97:0/1762309660 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa69c108660 0x7fa69c110230 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009930 tx=0x7fa68c02bc30 comp rx=0 tx=0).stop 2024-03-11T20:09:22.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.142+0000 7fa6a4400640 1 -- 172.21.15.97:0/1762309660 shutdown_connections 2024-03-11T20:09:22.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.142+0000 7fa6a4400640 1 --2- 172.21.15.97:0/1762309660 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa69c108660 0x7fa69c110230 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.142+0000 7fa6a4400640 1 --2- 172.21.15.97:0/1762309660 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa69c103f40 0x7fa69c107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.142+0000 7fa6a4400640 1 --2- 172.21.15.97:0/1762309660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa69c103590 0x7fa69c103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.142+0000 7fa6a4400640 1 -- 172.21.15.97:0/1762309660 >> 172.21.15.97:0/1762309660 conn(0x7fa69c0fee30 msgr2=0x7fa69c101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:22.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.142+0000 7fa6a4400640 1 -- 172.21.15.97:0/1762309660 shutdown_connections 2024-03-11T20:09:22.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.142+0000 7fa6a4400640 1 -- 172.21.15.97:0/1762309660 wait complete. 2024-03-11T20:09:22.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.143+0000 7fa6a4400640 1 Processor -- start 2024-03-11T20:09:22.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.143+0000 7fa6a4400640 1 -- start start 2024-03-11T20:09:22.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a4400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa69c103590 0x7fa69c19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:22.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a4400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa69c108660 0x7fa69c19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:22.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a4400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa69c1997e0 0x7fa69c199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:22.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a4400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c19a290 con 0x7fa69c1997e0 2024-03-11T20:09:22.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a4400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c19a400 con 0x7fa69c103590 2024-03-11T20:09:22.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a4400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69c19a570 con 0x7fa69c108660 2024-03-11T20:09:22.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a1a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa69c103590 0x7fa69c19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:22.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a2400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa69c1997e0 0x7fa69c199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:22.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a1a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa69c103590 0x7fa69c19f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:48850/0 (socket says 172.21.15.97:48850) 2024-03-11T20:09:22.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a1a00640 1 -- 172.21.15.97:0/252629186 learned_addr learned my addr 172.21.15.97:0/252629186 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:09:22.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a1000640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa69c108660 0x7fa69c19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:22.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a1a00640 1 -- 172.21.15.97:0/252629186 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa69c108660 msgr2=0x7fa69c19fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:22.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a1a00640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa69c108660 0x7fa69c19fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a1a00640 1 -- 172.21.15.97:0/252629186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa69c1997e0 msgr2=0x7fa69c199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:22.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a1a00640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa69c1997e0 0x7fa69c199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a1a00640 1 -- 172.21.15.97:0/252629186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68c009560 con 0x7fa69c103590 2024-03-11T20:09:22.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.144+0000 7fa6a2400640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa69c1997e0 0x7fa69c199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:09:22.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.145+0000 7fa6a1a00640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa69c103590 0x7fa69c19f670 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa69400ebc0 tx=0x7fa69400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:22.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.145+0000 7fa69a000640 1 -- 172.21.15.97:0/252629186 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6940099b0 con 0x7fa69c103590 2024-03-11T20:09:22.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.145+0000 7fa69a000640 1 -- 172.21.15.97:0/252629186 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa69400eec0 con 0x7fa69c103590 2024-03-11T20:09:22.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.145+0000 7fa6a4400640 1 -- 172.21.15.97:0/252629186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa69c1a6520 con 0x7fa69c103590 2024-03-11T20:09:22.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.145+0000 7fa69a000640 1 -- 172.21.15.97:0/252629186 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694018b60 con 0x7fa69c103590 2024-03-11T20:09:22.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.145+0000 7fa6a4400640 1 -- 172.21.15.97:0/252629186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69c1a6a00 con 0x7fa69c103590 2024-03-11T20:09:22.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.147+0000 7fa6a4400640 1 -- 172.21.15.97:0/252629186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69c104a60 con 0x7fa69c103590 2024-03-11T20:09:22.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.148+0000 7fa69a000640 1 -- 172.21.15.97:0/252629186 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fa694018cc0 con 0x7fa69c103590 2024-03-11T20:09:22.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.150+0000 7fa69a000640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fa6800a1ba0 0x7fa6800a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:22.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.150+0000 7fa69a000640 1 -- 172.21.15.97:0/252629186 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7fa694014070 con 0x7fa69c103590 2024-03-11T20:09:22.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.150+0000 7fa6a1000640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fa6800a1ba0 0x7fa6800a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:22.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.151+0000 7fa6a1000640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fa6800a1ba0 0x7fa6800a4060 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa69c109be0 tx=0x7fa688005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:22.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.155+0000 7fa69a000640 1 -- 172.21.15.97:0/252629186 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa694052d20 con 0x7fa69c103590 2024-03-11T20:09:22.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.354+0000 7fa6a4400640 1 -- 172.21.15.97:0/252629186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa69c103f40 con 0x7fa69c103590 2024-03-11T20:09:22.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.355+0000 7fa69a000640 1 -- 172.21.15.97:0/252629186 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa694096dd0 con 0x7fa69c103590 2024-03-11T20:09:22.345 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T20:09:22.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.358+0000 7fa6a4400640 1 -- 172.21.15.97:0/252629186 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fa6800a1ba0 msgr2=0x7fa6800a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:22.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.358+0000 7fa6a4400640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fa6800a1ba0 0x7fa6800a4060 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa69c109be0 tx=0x7fa688005f90 comp rx=0 tx=0).stop 2024-03-11T20:09:22.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.358+0000 7fa6a4400640 1 -- 172.21.15.97:0/252629186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa69c103590 msgr2=0x7fa69c19f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:22.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.358+0000 7fa6a4400640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa69c103590 0x7fa69c19f670 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa69400ebc0 tx=0x7fa69400c6a0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.358+0000 7fa6a4400640 1 -- 172.21.15.97:0/252629186 shutdown_connections 2024-03-11T20:09:22.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.358+0000 7fa6a4400640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fa6800a1ba0 0x7fa6800a4060 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.358+0000 7fa6a4400640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa69c1997e0 0x7fa69c199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.358+0000 7fa6a4400640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa69c108660 0x7fa69c19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.358+0000 7fa6a4400640 1 --2- 172.21.15.97:0/252629186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa69c103590 0x7fa69c19f670 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.358+0000 7fa6a4400640 1 -- 172.21.15.97:0/252629186 >> 172.21.15.97:0/252629186 conn(0x7fa69c0fee30 msgr2=0x7fa69c101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:22.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.358+0000 7fa6a4400640 1 -- 172.21.15.97:0/252629186 shutdown_connections 2024-03-11T20:09:22.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.358+0000 7fa6a4400640 1 -- 172.21.15.97:0/252629186 wait complete. 2024-03-11T20:09:22.365 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:22 smithi097 ceph-mon[31898]: from='client.25775 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:09:22.365 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:22 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/1564985140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:09:22.365 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:22 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2598377434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T20:09:22.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T20:09:22.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:22 smithi114 ceph-mon[36376]: from='client.25775 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:09:22.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:22 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/1564985140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:09:22.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:22 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2598377434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T20:09:22.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:22 smithi161 ceph-mon[36332]: from='client.25775 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:09:22.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:22 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/1564985140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:09:22.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:22 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2598377434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T20:09:22.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.595+0000 7fc068800640 1 -- 172.21.15.97:0/3463704008 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc060107720 msgr2=0x7fc060107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:22.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.595+0000 7fc068800640 1 --2- 172.21.15.97:0/3463704008 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc060107720 0x7fc060107ba0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc054009930 tx=0x7fc05402bc20 comp rx=0 tx=0).stop 2024-03-11T20:09:22.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.595+0000 7fc068800640 1 -- 172.21.15.97:0/3463704008 shutdown_connections 2024-03-11T20:09:22.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.595+0000 7fc068800640 1 --2- 172.21.15.97:0/3463704008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0601080e0 0x7fc0601145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.595+0000 7fc068800640 1 --2- 172.21.15.97:0/3463704008 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc060107720 0x7fc060107ba0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.595+0000 7fc068800640 1 --2- 172.21.15.97:0/3463704008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc06010d730 0x7fc06010db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.595+0000 7fc068800640 1 -- 172.21.15.97:0/3463704008 >> 172.21.15.97:0/3463704008 conn(0x7fc0600756c0 msgr2=0x7fc060077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:22.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.595+0000 7fc068800640 1 -- 172.21.15.97:0/3463704008 shutdown_connections 2024-03-11T20:09:22.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.595+0000 7fc068800640 1 -- 172.21.15.97:0/3463704008 wait complete. 2024-03-11T20:09:22.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.596+0000 7fc068800640 1 Processor -- start 2024-03-11T20:09:22.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.596+0000 7fc068800640 1 -- start start 2024-03-11T20:09:22.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.596+0000 7fc068800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc060107720 0x7fc06019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:22.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.596+0000 7fc068800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0601080e0 0x7fc06019e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:22.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.596+0000 7fc068800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc06010d730 0x7fc0601a26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:22.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.596+0000 7fc068800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0601a2cb0 con 0x7fc06010d730 2024-03-11T20:09:22.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.596+0000 7fc068800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0601a2e20 con 0x7fc060107720 2024-03-11T20:09:22.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.596+0000 7fc068800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0601a2fe0 con 0x7fc0601080e0 2024-03-11T20:09:22.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.597+0000 7fc066800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc06010d730 0x7fc0601a26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:22.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.597+0000 7fc065e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc060107720 0x7fc06019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:22.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.597+0000 7fc065400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0601080e0 0x7fc06019e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:22.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.597+0000 7fc065e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc060107720 0x7fc06019ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:48870/0 (socket says 172.21.15.97:48870) 2024-03-11T20:09:22.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.597+0000 7fc065e00640 1 -- 172.21.15.97:0/4203304889 learned_addr learned my addr 172.21.15.97:0/4203304889 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:09:22.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.597+0000 7fc065400640 1 -- 172.21.15.97:0/4203304889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc060107720 msgr2=0x7fc06019ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:22.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.597+0000 7fc065400640 1 --2- 172.21.15.97:0/4203304889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc060107720 0x7fc06019ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.597+0000 7fc065400640 1 -- 172.21.15.97:0/4203304889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc06010d730 msgr2=0x7fc0601a26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:22.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.597+0000 7fc065400640 1 --2- 172.21.15.97:0/4203304889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc06010d730 0x7fc0601a26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.597+0000 7fc065400640 1 -- 172.21.15.97:0/4203304889 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc054009560 con 0x7fc0601080e0 2024-03-11T20:09:22.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.598+0000 7fc065400640 1 --2- 172.21.15.97:0/4203304889 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0601080e0 0x7fc06019e330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc054036040 tx=0x7fc054004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:22.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.598+0000 7fc05aa00640 1 -- 172.21.15.97:0/4203304889 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05402dea0 con 0x7fc0601080e0 2024-03-11T20:09:22.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.598+0000 7fc05aa00640 1 -- 172.21.15.97:0/4203304889 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc054004440 con 0x7fc0601080e0 2024-03-11T20:09:22.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.598+0000 7fc068800640 1 -- 172.21.15.97:0/4203304889 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0601a3260 con 0x7fc0601080e0 2024-03-11T20:09:22.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.598+0000 7fc05aa00640 1 -- 172.21.15.97:0/4203304889 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc054003c20 con 0x7fc0601080e0 2024-03-11T20:09:22.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.598+0000 7fc068800640 1 -- 172.21.15.97:0/4203304889 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0601aab40 con 0x7fc0601080e0 2024-03-11T20:09:22.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.599+0000 7fc068800640 1 -- 172.21.15.97:0/4203304889 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc060108e60 con 0x7fc0601080e0 2024-03-11T20:09:22.591 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.601+0000 7fc05aa00640 1 -- 172.21.15.97:0/4203304889 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc054004df0 con 0x7fc0601080e0 2024-03-11T20:09:22.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.603+0000 7fc05aa00640 1 --2- 172.21.15.97:0/4203304889 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc0480a1ba0 0x7fc0480a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:22.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.603+0000 7fc05aa00640 1 -- 172.21.15.97:0/4203304889 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7fc0540f07c0 con 0x7fc0601080e0 2024-03-11T20:09:22.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.603+0000 7fc065e00640 1 --2- 172.21.15.97:0/4203304889 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc0480a1ba0 0x7fc0480a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:22.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.604+0000 7fc065e00640 1 --2- 172.21.15.97:0/4203304889 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc0480a1ba0 0x7fc0480a4060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc04c005fd0 tx=0x7fc04c00b480 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:22.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.605+0000 7fc05aa00640 1 -- 172.21.15.97:0/4203304889 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc0540b9350 con 0x7fc0601080e0 2024-03-11T20:09:22.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.805+0000 7fc068800640 1 -- 172.21.15.97:0/4203304889 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc06010fcb0 con 0x7fc0601080e0 2024-03-11T20:09:22.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.806+0000 7fc05aa00640 1 -- 172.21.15.97:0/4203304889 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc0540430a0 con 0x7fc0601080e0 2024-03-11T20:09:22.796 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-11T20:09:22.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.808+0000 7fc068800640 1 -- 172.21.15.97:0/4203304889 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc0480a1ba0 msgr2=0x7fc0480a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:22.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.809+0000 7fc068800640 1 --2- 172.21.15.97:0/4203304889 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc0480a1ba0 0x7fc0480a4060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc04c005fd0 tx=0x7fc04c00b480 comp rx=0 tx=0).stop 2024-03-11T20:09:22.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.809+0000 7fc068800640 1 -- 172.21.15.97:0/4203304889 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0601080e0 msgr2=0x7fc06019e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:22.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.809+0000 7fc068800640 1 --2- 172.21.15.97:0/4203304889 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0601080e0 0x7fc06019e330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc054036040 tx=0x7fc054004040 comp rx=0 tx=0).stop 2024-03-11T20:09:22.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.809+0000 7fc068800640 1 -- 172.21.15.97:0/4203304889 shutdown_connections 2024-03-11T20:09:22.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.809+0000 7fc068800640 1 --2- 172.21.15.97:0/4203304889 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fc0480a1ba0 0x7fc0480a4060 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.809+0000 7fc068800640 1 --2- 172.21.15.97:0/4203304889 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc06010d730 0x7fc0601a26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.809+0000 7fc068800640 1 --2- 172.21.15.97:0/4203304889 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0601080e0 0x7fc06019e330 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.809+0000 7fc068800640 1 --2- 172.21.15.97:0/4203304889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc060107720 0x7fc06019ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:22.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.809+0000 7fc068800640 1 -- 172.21.15.97:0/4203304889 >> 172.21.15.97:0/4203304889 conn(0x7fc0600756c0 msgr2=0x7fc060075d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:22.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.809+0000 7fc068800640 1 -- 172.21.15.97:0/4203304889 shutdown_connections 2024-03-11T20:09:22.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:22.809+0000 7fc068800640 1 -- 172.21.15.97:0/4203304889 wait complete. 2024-03-11T20:09:22.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T20:09:23.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.047+0000 7fe053c00640 1 -- 172.21.15.97:0/3222272609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe04c107720 msgr2=0x7fe04c107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:23.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.047+0000 7fe053c00640 1 --2- 172.21.15.97:0/3222272609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe04c107720 0x7fe04c107ba0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe038009930 tx=0x7fe03802bc20 comp rx=0 tx=0).stop 2024-03-11T20:09:23.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.047+0000 7fe053c00640 1 -- 172.21.15.97:0/3222272609 shutdown_connections 2024-03-11T20:09:23.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.047+0000 7fe053c00640 1 --2- 172.21.15.97:0/3222272609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe04c1080e0 0x7fe04c1145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.047+0000 7fe053c00640 1 --2- 172.21.15.97:0/3222272609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe04c107720 0x7fe04c107ba0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.047+0000 7fe053c00640 1 --2- 172.21.15.97:0/3222272609 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe04c10d730 0x7fe04c10db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.047+0000 7fe053c00640 1 -- 172.21.15.97:0/3222272609 >> 172.21.15.97:0/3222272609 conn(0x7fe04c0756c0 msgr2=0x7fe04c077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:23.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.047+0000 7fe053c00640 1 -- 172.21.15.97:0/3222272609 shutdown_connections 2024-03-11T20:09:23.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.047+0000 7fe053c00640 1 -- 172.21.15.97:0/3222272609 wait complete. 2024-03-11T20:09:23.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.049+0000 7fe053c00640 1 Processor -- start 2024-03-11T20:09:23.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.049+0000 7fe053c00640 1 -- start start 2024-03-11T20:09:23.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.049+0000 7fe053c00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe04c107720 0x7fe04c19de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:23.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.049+0000 7fe053c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe04c1080e0 0x7fe04c19e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:23.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.049+0000 7fe053c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe04c10d730 0x7fe04c1a2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:23.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.049+0000 7fe053c00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c1a2d00 con 0x7fe04c107720 2024-03-11T20:09:23.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.049+0000 7fe053c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c1a2e70 con 0x7fe04c1080e0 2024-03-11T20:09:23.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.049+0000 7fe053c00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c1a3030 con 0x7fe04c10d730 2024-03-11T20:09:23.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.050+0000 7fe051200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe04c107720 0x7fe04c19de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:23.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.050+0000 7fe04be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe04c1080e0 0x7fe04c19e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:23.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.050+0000 7fe04be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe04c1080e0 0x7fe04c19e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:48890/0 (socket says 172.21.15.97:48890) 2024-03-11T20:09:23.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.050+0000 7fe04be00640 1 -- 172.21.15.97:0/2888796755 learned_addr learned my addr 172.21.15.97:0/2888796755 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:09:23.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.050+0000 7fe051c00640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe04c10d730 0x7fe04c1a2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:23.045 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.050+0000 7fe051200640 1 -- 172.21.15.97:0/2888796755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe04c10d730 msgr2=0x7fe04c1a2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:23.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.050+0000 7fe051200640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe04c10d730 0x7fe04c1a2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.050+0000 7fe051200640 1 -- 172.21.15.97:0/2888796755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe04c1080e0 msgr2=0x7fe04c19e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:23.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.050+0000 7fe051200640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe04c1080e0 0x7fe04c19e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.050+0000 7fe051200640 1 -- 172.21.15.97:0/2888796755 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe038009560 con 0x7fe04c107720 2024-03-11T20:09:23.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.050+0000 7fe04be00640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe04c1080e0 0x7fe04c19e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:09:23.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.050+0000 7fe051200640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe04c107720 0x7fe04c19de90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe04000e9f0 tx=0x7fe04000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:23.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.051+0000 7fe049600640 1 -- 172.21.15.97:0/2888796755 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04000ccc0 con 0x7fe04c107720 2024-03-11T20:09:23.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.051+0000 7fe049600640 1 -- 172.21.15.97:0/2888796755 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe04000ce20 con 0x7fe04c107720 2024-03-11T20:09:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.051+0000 7fe053c00640 1 -- 172.21.15.97:0/2888796755 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe04c1a3310 con 0x7fe04c107720 2024-03-11T20:09:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.051+0000 7fe049600640 1 -- 172.21.15.97:0/2888796755 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe040018920 con 0x7fe04c107720 2024-03-11T20:09:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.051+0000 7fe053c00640 1 -- 172.21.15.97:0/2888796755 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04c1aab90 con 0x7fe04c107720 2024-03-11T20:09:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.052+0000 7fe053c00640 1 -- 172.21.15.97:0/2888796755 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe04c0683c0 con 0x7fe04c107720 2024-03-11T20:09:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.054+0000 7fe049600640 1 -- 172.21.15.97:0/2888796755 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe040018a80 con 0x7fe04c107720 2024-03-11T20:09:23.047 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.055+0000 7fe049600640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe0340a1c20 0x7fe0340a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:23.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.056+0000 7fe049600640 1 -- 172.21.15.97:0/2888796755 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7fe040014070 con 0x7fe04c107720 2024-03-11T20:09:23.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.056+0000 7fe04be00640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe0340a1c20 0x7fe0340a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:23.048 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.056+0000 7fe04be00640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe0340a1c20 0x7fe0340a40e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe038006010 tx=0x7fe038005a20 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:23.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.059+0000 7fe049600640 1 -- 172.21.15.97:0/2888796755 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe040052d20 con 0x7fe04c107720 2024-03-11T20:09:23.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:23 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/252629186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T20:09:23.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:23 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/4203304889' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T20:09:23.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:23 smithi097 ceph-mon[31898]: pgmap v314: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-11T20:09:23.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.261+0000 7fe053c00640 1 -- 172.21.15.97:0/2888796755 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe04c10fcb0 con 0x7fe04c107720 2024-03-11T20:09:23.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.262+0000 7fe049600640 1 -- 172.21.15.97:0/2888796755 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe040096930 con 0x7fe04c107720 2024-03-11T20:09:23.252 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T20:09:23.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.265+0000 7fe053c00640 1 -- 172.21.15.97:0/2888796755 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe0340a1c20 msgr2=0x7fe0340a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:23.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.265+0000 7fe053c00640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe0340a1c20 0x7fe0340a40e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe038006010 tx=0x7fe038005a20 comp rx=0 tx=0).stop 2024-03-11T20:09:23.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.265+0000 7fe053c00640 1 -- 172.21.15.97:0/2888796755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe04c107720 msgr2=0x7fe04c19de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:23.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.265+0000 7fe053c00640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe04c107720 0x7fe04c19de90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe04000e9f0 tx=0x7fe04000eeb0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.265+0000 7fe053c00640 1 -- 172.21.15.97:0/2888796755 shutdown_connections 2024-03-11T20:09:23.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.265+0000 7fe053c00640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe0340a1c20 0x7fe0340a40e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.265+0000 7fe053c00640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe04c10d730 0x7fe04c1a2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.265+0000 7fe053c00640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe04c1080e0 0x7fe04c19e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.265+0000 7fe053c00640 1 --2- 172.21.15.97:0/2888796755 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe04c107720 0x7fe04c19de90 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.265+0000 7fe053c00640 1 -- 172.21.15.97:0/2888796755 >> 172.21.15.97:0/2888796755 conn(0x7fe04c0756c0 msgr2=0x7fe04c075d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:23.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.265+0000 7fe053c00640 1 -- 172.21.15.97:0/2888796755 shutdown_connections 2024-03-11T20:09:23.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.265+0000 7fe053c00640 1 -- 172.21.15.97:0/2888796755 wait complete. 2024-03-11T20:09:23.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-11T20:09:23.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:23 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/252629186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T20:09:23.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:23 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/4203304889' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T20:09:23.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:23 smithi114 ceph-mon[36376]: pgmap v314: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-11T20:09:23.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.500+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2076233827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f8108650 msgr2=0x7fe5f8110220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:23.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.500+0000 7fe5fdc00640 1 --2- 172.21.15.97:0/2076233827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f8108650 0x7fe5f8110220 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009cd0 tx=0x7fe5f002c0e0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.501+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2076233827 shutdown_connections 2024-03-11T20:09:23.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.501+0000 7fe5fdc00640 1 --2- 172.21.15.97:0/2076233827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f8108650 0x7fe5f8110220 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.501+0000 7fe5fdc00640 1 --2- 172.21.15.97:0/2076233827 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5f8103f30 0x7fe5f8107f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.501+0000 7fe5fdc00640 1 --2- 172.21.15.97:0/2076233827 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5f8103580 0x7fe5f8103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.501+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2076233827 >> 172.21.15.97:0/2076233827 conn(0x7fe5f80fee40 msgr2=0x7fe5f8101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:23.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.501+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2076233827 shutdown_connections 2024-03-11T20:09:23.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.501+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2076233827 wait complete. 2024-03-11T20:09:23.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.502+0000 7fe5fdc00640 1 Processor -- start 2024-03-11T20:09:23.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.502+0000 7fe5fdc00640 1 -- start start 2024-03-11T20:09:23.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.502+0000 7fe5fdc00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5f8103580 0x7fe5f819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:23.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.502+0000 7fe5fdc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f8108650 0x7fe5f819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:23.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.502+0000 7fe5fdc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5f8199800 0x7fe5f8199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:23.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.502+0000 7fe5fdc00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f819a2b0 con 0x7fe5f8103580 2024-03-11T20:09:23.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.502+0000 7fe5fdc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f819a420 con 0x7fe5f8108650 2024-03-11T20:09:23.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.502+0000 7fe5fdc00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f819a590 con 0x7fe5f8199800 2024-03-11T20:09:23.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.502+0000 7fe5f6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5f8103580 0x7fe5f819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:23.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.502+0000 7fe5f6a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5f8103580 0x7fe5f819f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44222/0 (socket says 172.21.15.97:44222) 2024-03-11T20:09:23.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.502+0000 7fe5f6a00640 1 -- 172.21.15.97:0/2664978075 learned_addr learned my addr 172.21.15.97:0/2664978075 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:09:23.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.502+0000 7fe5f6000640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f8108650 0x7fe5f819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:23.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.502+0000 7fe5f7400640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5f8199800 0x7fe5f8199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:23.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.503+0000 7fe5f6000640 1 -- 172.21.15.97:0/2664978075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5f8199800 msgr2=0x7fe5f8199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:23.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.503+0000 7fe5f6000640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5f8199800 0x7fe5f8199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.503+0000 7fe5f6000640 1 -- 172.21.15.97:0/2664978075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5f8103580 msgr2=0x7fe5f819f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:23.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.503+0000 7fe5f6000640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5f8103580 0x7fe5f819f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:23.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.503+0000 7fe5f6000640 1 -- 172.21.15.97:0/2664978075 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f0009900 con 0x7fe5f8108650 2024-03-11T20:09:23.494 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.503+0000 7fe5f6000640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f8108650 0x7fe5f819fb80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe5e400e9c0 tx=0x7fe5e400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:23.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.503+0000 7fe5ef400640 1 -- 172.21.15.97:0/2664978075 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e400ccc0 con 0x7fe5f8108650 2024-03-11T20:09:23.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.503+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2664978075 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f81a64f0 con 0x7fe5f8108650 2024-03-11T20:09:23.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.503+0000 7fe5ef400640 1 -- 172.21.15.97:0/2664978075 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5e400ce20 con 0x7fe5f8108650 2024-03-11T20:09:23.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.503+0000 7fe5ef400640 1 -- 172.21.15.97:0/2664978075 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4018880 con 0x7fe5f8108650 2024-03-11T20:09:23.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.504+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2664978075 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f81a6a00 con 0x7fe5f8108650 2024-03-11T20:09:23.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.504+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2664978075 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f8104a50 con 0x7fe5f8108650 2024-03-11T20:09:23.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.507+0000 7fe5ef400640 1 -- 172.21.15.97:0/2664978075 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe5e40189e0 con 0x7fe5f8108650 2024-03-11T20:09:23.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.508+0000 7fe5ef400640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe5dc0a1ba0 0x7fe5dc0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:09:23.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.509+0000 7fe5ef400640 1 -- 172.21.15.97:0/2664978075 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7fe5e4014070 con 0x7fe5f8108650 2024-03-11T20:09:23.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.509+0000 7fe5f6a00640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe5dc0a1ba0 0x7fe5dc0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:09:23.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.509+0000 7fe5f6a00640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe5dc0a1ba0 0x7fe5dc0a4060 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800bca0 tx=0x7fe5e800b410 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:09:23.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.510+0000 7fe5ef400640 1 -- 172.21.15.97:0/2664978075 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe5e4096e80 con 0x7fe5f8108650 2024-03-11T20:09:23.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:23 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/252629186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T20:09:23.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:23 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/4203304889' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T20:09:23.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:23 smithi161 ceph-mon[36332]: pgmap v314: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-11T20:09:23.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.761+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2664978075 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7fe5f810b8e0 con 0x7fe5f8108650 2024-03-11T20:09:23.762 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.772+0000 7fe5ef400640 1 -- 172.21.15.97:0/2664978075 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) v1 ==== 81+0+735001 (secure 0 0 0) 0x7fe5e409a540 con 0x7fe5f8108650 2024-03-11T20:09:23.775 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T20:09:23.775 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T20:09:23.775 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 26, 2024-03-11T20:09:23.775 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.775 INFO:teuthology.orchestra.run.smithi097.stdout: "active_gid": 15396, 2024-03-11T20:09:23.775 INFO:teuthology.orchestra.run.smithi097.stdout: "active_name": "x", 2024-03-11T20:09:23.775 INFO:teuthology.orchestra.run.smithi097.stdout: "active_addrs": { 2024-03-11T20:09:23.776 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-03-11T20:09:23.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:23.776 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-03-11T20:09:23.776 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6800", 2024-03-11T20:09:23.776 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1085760108 2024-03-11T20:09:23.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:23.776 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v1", 2024-03-11T20:09:23.776 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:6801", 2024-03-11T20:09:23.776 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1085760108 2024-03-11T20:09:23.776 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:23.776 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T20:09:23.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.777 INFO:teuthology.orchestra.run.smithi097.stdout: "active_addr": "172.21.15.97:6801/1085760108", 2024-03-11T20:09:23.777 INFO:teuthology.orchestra.run.smithi097.stdout: "active_change": "2024-03-11T19:59:16.643393+0000", 2024-03-11T20:09:23.777 INFO:teuthology.orchestra.run.smithi097.stdout: "active_mgr_features": 4540701547738038271, 2024-03-11T20:09:23.777 INFO:teuthology.orchestra.run.smithi097.stdout: "available": true, 2024-03-11T20:09:23.777 INFO:teuthology.orchestra.run.smithi097.stdout: "standbys": [ 2024-03-11T20:09:23.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:23.777 INFO:teuthology.orchestra.run.smithi097.stdout: "gid": 25502, 2024-03-11T20:09:23.777 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "y", 2024-03-11T20:09:23.777 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_features": 4540701547738038271, 2024-03-11T20:09:23.777 INFO:teuthology.orchestra.run.smithi097.stdout: "available_modules": [ 2024-03-11T20:09:23.778 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:23.778 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "alerts", 2024-03-11T20:09:23.778 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:23.778 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:23.778 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:23.778 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:23.778 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:23.778 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:23.778 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.778 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.778 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:23.778 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.778 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.779 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.779 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "How frequently to reexamine health status", 2024-03-11T20:09:23.779 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.779 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.779 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.779 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:23.779 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:23.779 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.779 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.780 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.780 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.780 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.780 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.780 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:23.780 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:23.780 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:23.780 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:23.781 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:23.781 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:23.781 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:23.781 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:23.781 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.781 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.781 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.781 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.782 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:23.782 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:23.782 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.782 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.782 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.782 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.783 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.783 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.783 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.783 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.783 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.783 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.783 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.784 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.784 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:23.784 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:23.784 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.784 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.784 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.784 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:23.785 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.785 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.785 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:23.785 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:23.785 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:23.785 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:23.785 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:23.785 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:23.786 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:23.786 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:23.786 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.786 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.786 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.786 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.787 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:23.787 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:23.787 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.787 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.787 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.787 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.787 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.787 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.788 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.788 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.788 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.788 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.788 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.788 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.788 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_destination": { 2024-03-11T20:09:23.789 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_destination", 2024-03-11T20:09:23.789 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.789 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.789 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.789 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.789 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.789 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.790 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.790 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Email address to send alerts to", 2024-03-11T20:09:23.790 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.790 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.790 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.790 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_from_name": { 2024-03-11T20:09:23.791 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_from_name", 2024-03-11T20:09:23.791 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.791 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.791 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.791 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "Ceph", 2024-03-11T20:09:23.791 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.791 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.792 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.792 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Email From: name", 2024-03-11T20:09:23.792 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.792 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.792 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.792 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_host": { 2024-03-11T20:09:23.792 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_host", 2024-03-11T20:09:23.793 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.793 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.793 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.793 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.793 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.793 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.793 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.793 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "SMTP server", 2024-03-11T20:09:23.794 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.794 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.794 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.794 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_password": { 2024-03-11T20:09:23.794 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_password", 2024-03-11T20:09:23.794 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.795 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.795 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.795 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.795 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.795 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.795 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.795 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Password to authenticate with", 2024-03-11T20:09:23.795 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.795 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.796 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.796 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_port": { 2024-03-11T20:09:23.796 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_port", 2024-03-11T20:09:23.796 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.796 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.796 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.796 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "465", 2024-03-11T20:09:23.796 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.796 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.797 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.797 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "SMTP port", 2024-03-11T20:09:23.797 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.797 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.797 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.797 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_sender": { 2024-03-11T20:09:23.797 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_sender", 2024-03-11T20:09:23.797 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.797 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.798 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.798 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.798 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.798 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.798 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.798 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "SMTP envelope sender", 2024-03-11T20:09:23.798 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.798 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.799 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.799 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_ssl": { 2024-03-11T20:09:23.799 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_ssl", 2024-03-11T20:09:23.799 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.799 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.800 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.800 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.800 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.800 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.800 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.800 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-11T20:09:23.800 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.800 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.801 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.801 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_user": { 2024-03-11T20:09:23.801 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_user", 2024-03-11T20:09:23.801 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.801 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.801 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.801 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.801 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.801 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.801 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.801 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "User to authenticate as", 2024-03-11T20:09:23.802 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.802 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.802 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.802 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:23.802 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:23.802 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.802 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:23.802 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "balancer", 2024-03-11T20:09:23.802 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:23.802 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:23.802 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:23.803 INFO:teuthology.orchestra.run.smithi097.stdout: "active": { 2024-03-11T20:09:23.803 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "active", 2024-03-11T20:09:23.803 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.803 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.803 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.803 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.803 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.803 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.803 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.803 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "automatically balance PGs across cluster", 2024-03-11T20:09:23.804 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.804 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.804 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.804 INFO:teuthology.orchestra.run.smithi097.stdout: "begin_time": { 2024-03-11T20:09:23.804 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "begin_time", 2024-03-11T20:09:23.804 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.804 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.804 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.804 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0000", 2024-03-11T20:09:23.805 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.805 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.805 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.805 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "beginning time of day to automatically balance", 2024-03-11T20:09:23.805 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T20:09:23.805 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.805 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.805 INFO:teuthology.orchestra.run.smithi097.stdout: "begin_weekday": { 2024-03-11T20:09:23.805 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "begin_weekday", 2024-03-11T20:09:23.805 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:23.805 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.806 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.806 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:23.806 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "0", 2024-03-11T20:09:23.806 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "6", 2024-03-11T20:09:23.806 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.806 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-11T20:09:23.806 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T20:09:23.806 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.806 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.806 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_compat_max_iterations": { 2024-03-11T20:09:23.806 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crush_compat_max_iterations", 2024-03-11T20:09:23.807 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:23.807 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.807 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.807 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "25", 2024-03-11T20:09:23.807 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1", 2024-03-11T20:09:23.807 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "250", 2024-03-11T20:09:23.807 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.807 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-11T20:09:23.807 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.807 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.807 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.807 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_compat_metrics": { 2024-03-11T20:09:23.808 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crush_compat_metrics", 2024-03-11T20:09:23.808 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.808 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.808 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.808 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "pgs,objects,bytes", 2024-03-11T20:09:23.808 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.808 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.808 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.808 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-11T20:09:23.808 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-11T20:09:23.808 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.808 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.809 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_compat_step": { 2024-03-11T20:09:23.809 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crush_compat_step", 2024-03-11T20:09:23.809 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:23.809 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.809 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.809 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0.5", 2024-03-11T20:09:23.809 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "0.001", 2024-03-11T20:09:23.809 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "0.999", 2024-03-11T20:09:23.809 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.809 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "aggressiveness of optimization", 2024-03-11T20:09:23.809 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-11T20:09:23.809 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.809 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.810 INFO:teuthology.orchestra.run.smithi097.stdout: "end_time": { 2024-03-11T20:09:23.810 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "end_time", 2024-03-11T20:09:23.810 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.810 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.810 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.810 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "2359", 2024-03-11T20:09:23.810 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.810 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.810 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.810 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "ending time of day to automatically balance", 2024-03-11T20:09:23.810 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T20:09:23.810 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.811 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.811 INFO:teuthology.orchestra.run.smithi097.stdout: "end_weekday": { 2024-03-11T20:09:23.811 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "end_weekday", 2024-03-11T20:09:23.811 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:23.811 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.811 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.811 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:23.811 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "0", 2024-03-11T20:09:23.811 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "6", 2024-03-11T20:09:23.811 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.811 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-11T20:09:23.811 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T20:09:23.812 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.812 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.812 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:23.812 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:23.812 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.812 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.812 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.812 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.812 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.812 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.812 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:23.812 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:23.813 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:23.813 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:23.813 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:23.813 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:23.813 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:23.813 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:23.813 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.813 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.813 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.813 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.814 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:23.814 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:23.814 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.814 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.814 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.814 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.814 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.814 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.815 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.815 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.815 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.815 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.815 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.815 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:23.815 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:23.815 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.815 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.815 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.816 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:23.816 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.816 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.816 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:23.816 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:23.816 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:23.816 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:23.816 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:23.816 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:23.816 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:23.816 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:23.816 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.817 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.817 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.817 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.817 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:23.817 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:23.817 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.817 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.817 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.817 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.817 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.818 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.818 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.818 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.818 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.818 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.818 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.818 INFO:teuthology.orchestra.run.smithi097.stdout: "min_score": { 2024-03-11T20:09:23.819 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "min_score", 2024-03-11T20:09:23.819 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:23.819 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.819 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.819 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:23.819 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.819 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.819 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.820 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-11T20:09:23.820 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.820 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.820 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.820 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.820 INFO:teuthology.orchestra.run.smithi097.stdout: "mode": { 2024-03-11T20:09:23.820 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mode", 2024-03-11T20:09:23.821 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.821 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.821 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.821 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "upmap", 2024-03-11T20:09:23.821 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.821 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.821 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:23.822 INFO:teuthology.orchestra.run.smithi097.stdout: "crush-compat", 2024-03-11T20:09:23.822 INFO:teuthology.orchestra.run.smithi097.stdout: "none", 2024-03-11T20:09:23.822 INFO:teuthology.orchestra.run.smithi097.stdout: "read", 2024-03-11T20:09:23.822 INFO:teuthology.orchestra.run.smithi097.stdout: "upmap", 2024-03-11T20:09:23.822 INFO:teuthology.orchestra.run.smithi097.stdout: "upmap-read" 2024-03-11T20:09:23.822 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:23.822 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Balancer mode", 2024-03-11T20:09:23.823 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.823 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.823 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.823 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_ids": { 2024-03-11T20:09:23.823 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "pool_ids", 2024-03-11T20:09:23.823 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.824 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.824 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.824 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.824 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.824 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.824 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.824 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-11T20:09:23.825 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.825 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.825 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.825 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:23.825 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:23.825 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:23.826 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.826 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.826 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:23.826 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.826 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.826 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.826 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-11T20:09:23.827 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.827 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.827 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.827 INFO:teuthology.orchestra.run.smithi097.stdout: "upmap_max_deviation": { 2024-03-11T20:09:23.827 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "upmap_max_deviation", 2024-03-11T20:09:23.827 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.827 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.828 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.828 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5", 2024-03-11T20:09:23.828 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1", 2024-03-11T20:09:23.828 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.828 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.828 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-11T20:09:23.828 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-11T20:09:23.829 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.829 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.829 INFO:teuthology.orchestra.run.smithi097.stdout: "upmap_max_optimizations": { 2024-03-11T20:09:23.829 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "upmap_max_optimizations", 2024-03-11T20:09:23.829 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:23.829 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.829 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.830 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:23.830 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.830 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.830 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.830 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-11T20:09:23.830 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.830 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.830 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.831 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:23.831 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:23.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:23.831 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cephadm", 2024-03-11T20:09:23.831 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:23.831 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:23.832 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:23.832 INFO:teuthology.orchestra.run.smithi097.stdout: "agent_down_multiplier": { 2024-03-11T20:09:23.832 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "agent_down_multiplier", 2024-03-11T20:09:23.832 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:23.832 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.832 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.832 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "3.0", 2024-03-11T20:09:23.833 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.833 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.833 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.833 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-11T20:09:23.833 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.833 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.833 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.834 INFO:teuthology.orchestra.run.smithi097.stdout: "agent_refresh_rate": { 2024-03-11T20:09:23.834 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "agent_refresh_rate", 2024-03-11T20:09:23.834 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:23.834 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.834 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.834 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "20", 2024-03-11T20:09:23.834 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.835 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.835 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.835 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-11T20:09:23.835 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.835 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.835 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.835 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.835 INFO:teuthology.orchestra.run.smithi097.stdout: "agent_starting_port": { 2024-03-11T20:09:23.836 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "agent_starting_port", 2024-03-11T20:09:23.836 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.836 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.836 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.836 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "4721", 2024-03-11T20:09:23.836 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.836 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.837 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.837 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-11T20:09:23.837 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.837 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.837 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.837 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.838 INFO:teuthology.orchestra.run.smithi097.stdout: "allow_ptrace": { 2024-03-11T20:09:23.838 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "allow_ptrace", 2024-03-11T20:09:23.838 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.838 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.838 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.838 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.838 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.838 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.839 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.839 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-11T20:09:23.839 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-11T20:09:23.839 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.839 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.839 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.839 INFO:teuthology.orchestra.run.smithi097.stdout: "autotune_interval": { 2024-03-11T20:09:23.839 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "autotune_interval", 2024-03-11T20:09:23.839 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:23.839 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.840 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.840 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:23.840 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.840 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.840 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.840 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-11T20:09:23.840 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.840 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.840 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.840 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.840 INFO:teuthology.orchestra.run.smithi097.stdout: "autotune_memory_target_ratio": { 2024-03-11T20:09:23.840 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "autotune_memory_target_ratio", 2024-03-11T20:09:23.841 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:23.841 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.841 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.841 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0.7", 2024-03-11T20:09:23.841 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.841 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.841 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.841 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-11T20:09:23.841 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.841 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.841 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.841 INFO:teuthology.orchestra.run.smithi097.stdout: "cephadm_log_destination": { 2024-03-11T20:09:23.842 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cephadm_log_destination", 2024-03-11T20:09:23.842 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.842 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.842 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.842 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.842 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.842 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.842 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:23.842 INFO:teuthology.orchestra.run.smithi097.stdout: "file", 2024-03-11T20:09:23.842 INFO:teuthology.orchestra.run.smithi097.stdout: "file,syslog", 2024-03-11T20:09:23.842 INFO:teuthology.orchestra.run.smithi097.stdout: "syslog" 2024-03-11T20:09:23.842 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:23.843 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-11T20:09:23.843 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.843 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.843 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.843 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.843 INFO:teuthology.orchestra.run.smithi097.stdout: "cgroups_split": { 2024-03-11T20:09:23.843 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cgroups_split", 2024-03-11T20:09:23.843 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.843 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.843 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.843 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.843 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.844 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.844 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.844 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-11T20:09:23.844 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.844 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.844 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.844 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.844 INFO:teuthology.orchestra.run.smithi097.stdout: "config_checks_enabled": { 2024-03-11T20:09:23.844 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "config_checks_enabled", 2024-03-11T20:09:23.844 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.844 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.844 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.844 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.845 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.845 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.845 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.845 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-11T20:09:23.845 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.845 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.845 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.845 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.845 INFO:teuthology.orchestra.run.smithi097.stdout: "config_dashboard": { 2024-03-11T20:09:23.845 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "config_dashboard", 2024-03-11T20:09:23.845 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.845 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.845 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.846 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.846 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.846 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.846 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.846 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-11T20:09:23.846 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.846 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.846 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.846 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_alertmanager": { 2024-03-11T20:09:23.846 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_alertmanager", 2024-03-11T20:09:23.846 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.847 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.847 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.847 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-11T20:09:23.847 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.847 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.847 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.847 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Prometheus container image", 2024-03-11T20:09:23.847 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.847 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.847 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.847 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_base": { 2024-03-11T20:09:23.847 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_base", 2024-03-11T20:09:23.848 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.848 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.848 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.848 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-11T20:09:23.848 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.848 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.848 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.848 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Container image name, without the tag", 2024-03-11T20:09:23.848 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.848 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.848 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.849 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_elasticsearch": { 2024-03-11T20:09:23.849 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_elasticsearch", 2024-03-11T20:09:23.849 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.849 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.849 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.849 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-11T20:09:23.849 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.849 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.849 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.849 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "elasticsearch container image", 2024-03-11T20:09:23.849 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.849 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.849 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.850 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.850 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_grafana": { 2024-03-11T20:09:23.850 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_grafana", 2024-03-11T20:09:23.850 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.850 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.850 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.850 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-11T20:09:23.850 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.850 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.850 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.850 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Prometheus container image", 2024-03-11T20:09:23.850 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.851 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.851 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.851 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_haproxy": { 2024-03-11T20:09:23.851 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_haproxy", 2024-03-11T20:09:23.851 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.851 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.851 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.851 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-11T20:09:23.852 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.852 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.852 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.852 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "HAproxy container image", 2024-03-11T20:09:23.852 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.852 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.852 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.852 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.852 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_jaeger_agent": { 2024-03-11T20:09:23.852 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_jaeger_agent", 2024-03-11T20:09:23.852 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.852 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.853 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.853 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-11T20:09:23.853 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.853 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.853 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.853 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Jaeger agent container image", 2024-03-11T20:09:23.853 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.853 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.853 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.854 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_jaeger_collector": { 2024-03-11T20:09:23.854 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_jaeger_collector", 2024-03-11T20:09:23.854 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.854 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.854 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.854 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-11T20:09:23.854 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.855 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.855 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.855 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Jaeger collector container image", 2024-03-11T20:09:23.855 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.855 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.859 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.860 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_jaeger_query": { 2024-03-11T20:09:23.860 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_jaeger_query", 2024-03-11T20:09:23.860 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.860 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.860 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.860 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-11T20:09:23.860 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.860 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.860 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.860 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Jaeger query container image", 2024-03-11T20:09:23.861 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.861 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.861 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.861 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.861 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_keepalived": { 2024-03-11T20:09:23.861 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_keepalived", 2024-03-11T20:09:23.861 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.861 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.861 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.861 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-11T20:09:23.862 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.862 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.862 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.862 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Keepalived container image", 2024-03-11T20:09:23.862 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.862 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.862 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.862 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_loki": { 2024-03-11T20:09:23.862 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_loki", 2024-03-11T20:09:23.863 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.863 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.863 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.863 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-11T20:09:23.863 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.863 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.863 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.864 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Loki container image", 2024-03-11T20:09:23.864 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.864 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.864 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.864 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.864 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_node_exporter": { 2024-03-11T20:09:23.864 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_node_exporter", 2024-03-11T20:09:23.865 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.865 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.865 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.865 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-11T20:09:23.865 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.865 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.865 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.866 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Prometheus container image", 2024-03-11T20:09:23.866 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.866 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.866 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.866 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.866 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_nvmeof": { 2024-03-11T20:09:23.866 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_nvmeof", 2024-03-11T20:09:23.867 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.867 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.867 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.867 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-11T20:09:23.867 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.867 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.867 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.868 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Nvme-of container image", 2024-03-11T20:09:23.868 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.868 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.868 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.868 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.868 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_prometheus": { 2024-03-11T20:09:23.868 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_prometheus", 2024-03-11T20:09:23.868 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.869 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.869 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.869 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-11T20:09:23.869 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.869 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.869 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.869 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Prometheus container image", 2024-03-11T20:09:23.869 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.870 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.870 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.870 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.870 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_promtail": { 2024-03-11T20:09:23.870 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_promtail", 2024-03-11T20:09:23.870 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.870 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.871 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.871 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-11T20:09:23.871 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.871 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.871 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.871 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Promtail container image", 2024-03-11T20:09:23.871 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.872 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.872 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.872 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.872 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_snmp_gateway": { 2024-03-11T20:09:23.872 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_snmp_gateway", 2024-03-11T20:09:23.872 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.872 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.872 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.873 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-11T20:09:23.873 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.873 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.873 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.873 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "SNMP Gateway container image", 2024-03-11T20:09:23.873 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.873 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.873 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.873 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.874 INFO:teuthology.orchestra.run.smithi097.stdout: "container_init": { 2024-03-11T20:09:23.874 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_init", 2024-03-11T20:09:23.874 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.874 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.874 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.874 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.874 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.874 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.874 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.875 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Run podman/docker with `--init`", 2024-03-11T20:09:23.875 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.875 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.875 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.875 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.875 INFO:teuthology.orchestra.run.smithi097.stdout: "daemon_cache_timeout": { 2024-03-11T20:09:23.875 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "daemon_cache_timeout", 2024-03-11T20:09:23.876 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:23.876 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.876 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.876 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:23.876 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.876 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.877 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.877 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-11T20:09:23.877 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.877 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.877 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.877 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.877 INFO:teuthology.orchestra.run.smithi097.stdout: "default_cephadm_command_timeout": { 2024-03-11T20:09:23.877 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "default_cephadm_command_timeout", 2024-03-11T20:09:23.878 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.878 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.878 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.878 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "900", 2024-03-11T20:09:23.878 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.878 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.878 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.878 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-11T20:09:23.878 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.879 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.879 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.879 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.879 INFO:teuthology.orchestra.run.smithi097.stdout: "default_registry": { 2024-03-11T20:09:23.879 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "default_registry", 2024-03-11T20:09:23.879 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.879 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.879 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.880 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "docker.io", 2024-03-11T20:09:23.880 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.880 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.880 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.880 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-11T20:09:23.880 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.880 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.881 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.881 INFO:teuthology.orchestra.run.smithi097.stdout: "device_cache_timeout": { 2024-03-11T20:09:23.881 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "device_cache_timeout", 2024-03-11T20:09:23.881 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:23.881 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.881 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.881 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "1800", 2024-03-11T20:09:23.881 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.881 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.882 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.882 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "seconds to cache device inventory", 2024-03-11T20:09:23.882 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.882 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.882 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.882 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.882 INFO:teuthology.orchestra.run.smithi097.stdout: "device_enhanced_scan": { 2024-03-11T20:09:23.882 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "device_enhanced_scan", 2024-03-11T20:09:23.882 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.882 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.882 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.882 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.882 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.883 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.883 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.883 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-11T20:09:23.883 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.883 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.883 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.883 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.883 INFO:teuthology.orchestra.run.smithi097.stdout: "facts_cache_timeout": { 2024-03-11T20:09:23.883 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "facts_cache_timeout", 2024-03-11T20:09:23.883 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:23.883 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.883 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.884 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:23.884 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.884 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.884 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.884 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "seconds to cache host facts data", 2024-03-11T20:09:23.884 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.884 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.884 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.884 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.884 INFO:teuthology.orchestra.run.smithi097.stdout: "host_check_interval": { 2024-03-11T20:09:23.884 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "host_check_interval", 2024-03-11T20:09:23.884 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:23.884 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to perform a host check", 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: "hw_monitoring": { 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "hw_monitoring", 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.885 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.886 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.886 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.886 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.886 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.886 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.886 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-11T20:09:23.886 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.886 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.886 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.886 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.886 INFO:teuthology.orchestra.run.smithi097.stdout: "inventory_list_all": { 2024-03-11T20:09:23.886 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "inventory_list_all", 2024-03-11T20:09:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-11T20:09:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:23.888 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:23.888 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.888 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.888 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.888 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.888 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.888 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.888 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:23.888 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:23.888 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:23.888 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:23.888 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:23.888 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: "log_refresh_metadata": { 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_refresh_metadata", 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.889 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.890 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.890 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.890 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-11T20:09:23.890 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.890 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.890 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.890 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.890 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:23.890 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:23.890 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-11T20:09:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:23.891 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:23.892 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.892 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.892 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.892 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:23.892 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.892 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.892 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:23.892 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:23.892 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:23.893 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:23.893 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:23.893 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:23.893 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:23.893 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:23.893 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.893 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.893 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.894 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.894 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.894 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:23.894 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:23.894 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.894 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.894 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.895 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.895 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.895 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.895 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.895 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.895 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.895 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.896 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.896 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.896 INFO:teuthology.orchestra.run.smithi097.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-11T20:09:23.896 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-11T20:09:23.896 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.896 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.896 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.897 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.897 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.897 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.897 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.897 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-11T20:09:23.897 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.897 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.897 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.897 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.897 INFO:teuthology.orchestra.run.smithi097.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-11T20:09:23.897 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-11T20:09:23.898 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.898 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.898 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.898 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "*", 2024-03-11T20:09:23.898 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.898 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.898 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.898 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-11T20:09:23.898 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.898 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.898 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.899 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.899 INFO:teuthology.orchestra.run.smithi097.stdout: "max_count_per_host": { 2024-03-11T20:09:23.899 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_count_per_host", 2024-03-11T20:09:23.899 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.899 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.899 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.899 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:23.899 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.899 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.899 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.899 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "max number of daemons per service per host", 2024-03-11T20:09:23.899 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.900 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.900 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.900 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.900 INFO:teuthology.orchestra.run.smithi097.stdout: "max_osd_draining_count": { 2024-03-11T20:09:23.900 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_osd_draining_count", 2024-03-11T20:09:23.900 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.900 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.900 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.901 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:23.901 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.901 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.901 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.901 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-11T20:09:23.901 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.901 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.901 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.902 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.902 INFO:teuthology.orchestra.run.smithi097.stdout: "migration_current": { 2024-03-11T20:09:23.902 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "migration_current", 2024-03-11T20:09:23.902 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.902 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.902 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.902 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.903 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.903 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.903 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.903 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "internal - do not modify", 2024-03-11T20:09:23.903 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.903 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.903 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.904 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.904 INFO:teuthology.orchestra.run.smithi097.stdout: "mode": { 2024-03-11T20:09:23.904 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mode", 2024-03-11T20:09:23.904 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.904 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.904 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.904 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "root", 2024-03-11T20:09:23.905 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.905 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.905 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:23.905 INFO:teuthology.orchestra.run.smithi097.stdout: "cephadm-package", 2024-03-11T20:09:23.905 INFO:teuthology.orchestra.run.smithi097.stdout: "root" 2024-03-11T20:09:23.905 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:23.905 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "mode for remote execution of cephadm", 2024-03-11T20:09:23.905 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.906 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.906 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.906 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.906 INFO:teuthology.orchestra.run.smithi097.stdout: "oob_default_addr": { 2024-03-11T20:09:23.906 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "oob_default_addr", 2024-03-11T20:09:23.906 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.906 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.907 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.907 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "169.254.1.1", 2024-03-11T20:09:23.907 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.907 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.907 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.907 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-11T20:09:23.907 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.907 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.908 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.908 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.908 INFO:teuthology.orchestra.run.smithi097.stdout: "prometheus_alerts_path": { 2024-03-11T20:09:23.908 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "prometheus_alerts_path", 2024-03-11T20:09:23.908 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.908 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.908 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.909 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-11T20:09:23.909 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.909 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.909 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.909 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-11T20:09:23.909 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.909 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.909 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.910 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.910 INFO:teuthology.orchestra.run.smithi097.stdout: "registry_insecure": { 2024-03-11T20:09:23.910 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "registry_insecure", 2024-03-11T20:09:23.910 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.910 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.910 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.910 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.910 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.910 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.911 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.911 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-11T20:09:23.911 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.911 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.911 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.911 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.911 INFO:teuthology.orchestra.run.smithi097.stdout: "registry_password": { 2024-03-11T20:09:23.912 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "registry_password", 2024-03-11T20:09:23.912 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.912 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.912 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.912 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.912 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.912 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.912 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.913 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-11T20:09:23.913 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.913 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.913 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.913 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.913 INFO:teuthology.orchestra.run.smithi097.stdout: "registry_url": { 2024-03-11T20:09:23.913 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "registry_url", 2024-03-11T20:09:23.914 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.914 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.914 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.914 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.914 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.915 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.915 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.915 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-11T20:09:23.915 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.915 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.915 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.915 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.916 INFO:teuthology.orchestra.run.smithi097.stdout: "registry_username": { 2024-03-11T20:09:23.916 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "registry_username", 2024-03-11T20:09:23.916 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.916 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.916 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.916 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.916 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.916 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.917 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.917 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-11T20:09:23.917 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.917 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.917 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.917 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.917 INFO:teuthology.orchestra.run.smithi097.stdout: "secure_monitoring_stack": { 2024-03-11T20:09:23.917 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "secure_monitoring_stack", 2024-03-11T20:09:23.917 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.917 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.918 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.918 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.918 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.918 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.918 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.918 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-11T20:09:23.918 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.918 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.918 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.918 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.918 INFO:teuthology.orchestra.run.smithi097.stdout: "service_discovery_port": { 2024-03-11T20:09:23.918 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "service_discovery_port", 2024-03-11T20:09:23.918 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.919 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.919 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.919 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8765", 2024-03-11T20:09:23.919 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.919 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.919 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.919 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "cephadm service discovery port", 2024-03-11T20:09:23.919 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.919 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.919 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.919 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.919 INFO:teuthology.orchestra.run.smithi097.stdout: "ssh_config_file": { 2024-03-11T20:09:23.920 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ssh_config_file", 2024-03-11T20:09:23.920 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.920 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.920 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.920 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.920 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.920 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.920 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.920 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-11T20:09:23.920 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.920 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.920 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.921 INFO:teuthology.orchestra.run.smithi097.stdout: "use_agent": { 2024-03-11T20:09:23.921 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "use_agent", 2024-03-11T20:09:23.921 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.921 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.921 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.921 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.921 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.921 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.921 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.921 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-11T20:09:23.922 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.922 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.922 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.922 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.922 INFO:teuthology.orchestra.run.smithi097.stdout: "use_repo_digest": { 2024-03-11T20:09:23.922 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "use_repo_digest", 2024-03-11T20:09:23.922 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.922 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.922 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.922 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.922 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.923 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.923 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.923 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-11T20:09:23.923 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.923 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.923 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.923 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.923 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_on_failed_host_check": { 2024-03-11T20:09:23.923 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_on_failed_host_check", 2024-03-11T20:09:23.923 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.923 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.923 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.924 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.924 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.924 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.924 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.924 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "raise a health warning if the host check fails", 2024-03-11T20:09:23.924 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.924 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.924 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.924 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.924 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_on_stray_daemons": { 2024-03-11T20:09:23.924 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_on_stray_daemons", 2024-03-11T20:09:23.924 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.924 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_on_stray_hosts": { 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_on_stray_hosts", 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.925 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.926 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.926 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.926 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.926 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.926 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.926 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-11T20:09:23.926 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.926 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.926 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.926 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:23.926 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:23.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.927 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:23.927 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crash", 2024-03-11T20:09:23.927 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:23.927 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:23.927 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:23.927 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:23.927 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:23.927 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.927 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.927 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.927 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.927 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.929 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:23.929 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:23.929 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.929 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.929 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.929 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.929 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.929 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.929 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.929 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.930 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.930 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.930 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.930 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:23.930 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:23.931 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.931 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.931 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.931 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:23.931 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.931 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.931 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:23.931 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:23.932 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:23.932 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:23.932 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:23.932 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:23.932 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:23.932 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:23.932 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.933 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.933 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.933 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.933 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:23.933 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:23.933 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.934 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.934 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.934 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.934 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.934 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.934 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.934 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.935 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.935 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.935 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.935 INFO:teuthology.orchestra.run.smithi097.stdout: "retain_interval": { 2024-03-11T20:09:23.935 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "retain_interval", 2024-03-11T20:09:23.936 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:23.936 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.936 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.936 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "31536000", 2024-03-11T20:09:23.936 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.936 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.936 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.936 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-11T20:09:23.937 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.937 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.937 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.937 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_recent_interval": { 2024-03-11T20:09:23.937 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_recent_interval", 2024-03-11T20:09:23.937 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:23.938 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.938 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.938 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "1209600", 2024-03-11T20:09:23.938 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.938 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.938 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.938 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-11T20:09:23.939 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.939 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.939 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.939 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:23.939 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:23.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:23.939 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "dashboard", 2024-03-11T20:09:23.939 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:23.940 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:23.940 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:23.940 INFO:teuthology.orchestra.run.smithi097.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-11T20:09:23.940 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-11T20:09:23.940 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.940 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.940 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.940 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:23.941 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.941 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.941 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.941 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.941 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.941 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.941 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.941 INFO:teuthology.orchestra.run.smithi097.stdout: "ALERTMANAGER_API_HOST": { 2024-03-11T20:09:23.941 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-11T20:09:23.941 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.941 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.942 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.942 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.942 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.942 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.942 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.942 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.942 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.942 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.942 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.942 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.942 INFO:teuthology.orchestra.run.smithi097.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-11T20:09:23.942 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-11T20:09:23.943 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.943 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.943 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.943 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.943 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.943 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.943 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.943 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.943 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.943 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.943 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.943 INFO:teuthology.orchestra.run.smithi097.stdout: "AUDIT_API_ENABLED": { 2024-03-11T20:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "AUDIT_API_ENABLED", 2024-03-11T20:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.945 INFO:teuthology.orchestra.run.smithi097.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-11T20:09:23.945 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-11T20:09:23.945 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.945 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.945 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.945 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.945 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.945 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.945 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.945 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.945 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.945 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.946 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.946 INFO:teuthology.orchestra.run.smithi097.stdout: "ENABLE_BROWSABLE_API": { 2024-03-11T20:09:23.946 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-11T20:09:23.946 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.946 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.946 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.946 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.946 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.946 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.946 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.947 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.948 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.948 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.948 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.948 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.948 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-11T20:09:23.948 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-11T20:09:23.948 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.948 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.948 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.948 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.948 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.948 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.949 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.949 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.949 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.949 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.949 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.949 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.949 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-11T20:09:23.949 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-11T20:09:23.949 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.949 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.949 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.949 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.950 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.950 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.950 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.950 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.950 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.950 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.950 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.950 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.950 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-11T20:09:23.950 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-11T20:09:23.950 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.950 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.951 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.951 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.951 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.951 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.951 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.951 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.951 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.951 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.951 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.951 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.951 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-11T20:09:23.951 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-11T20:09:23.951 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-11T20:09:23.952 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.953 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.953 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.953 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.953 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.953 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.953 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.953 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.953 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.953 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.953 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.953 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-11T20:09:23.953 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-11T20:09:23.954 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.954 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.954 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.954 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.954 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.954 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.954 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.954 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.954 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.954 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.954 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.955 INFO:teuthology.orchestra.run.smithi097.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-11T20:09:23.955 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-11T20:09:23.955 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.955 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.955 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.955 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.955 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.955 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.955 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.955 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.955 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.955 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.955 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.956 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_API_PASSWORD": { 2024-03-11T20:09:23.956 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-11T20:09:23.956 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.956 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.956 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.956 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "admin", 2024-03-11T20:09:23.956 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.956 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.956 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.956 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.956 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.956 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.957 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.958 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.958 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_API_URL": { 2024-03-11T20:09:23.958 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_API_URL", 2024-03-11T20:09:23.958 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.958 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.958 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.958 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.958 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.958 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.958 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.958 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.959 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.959 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.959 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.959 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_API_USERNAME": { 2024-03-11T20:09:23.959 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-11T20:09:23.959 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.959 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.959 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.959 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "admin", 2024-03-11T20:09:23.959 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.959 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.959 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.960 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.961 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.961 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.961 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.961 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.961 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-11T20:09:23.961 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-11T20:09:23.961 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.961 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.961 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.961 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.961 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.961 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.962 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.962 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.962 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.962 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.962 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.962 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.962 INFO:teuthology.orchestra.run.smithi097.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-11T20:09:23.962 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-11T20:09:23.962 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.962 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.962 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.962 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.963 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.963 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.963 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.963 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.963 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.963 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.963 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.963 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.963 INFO:teuthology.orchestra.run.smithi097.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-11T20:09:23.963 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-11T20:09:23.963 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.963 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.964 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.964 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.964 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.964 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.964 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.964 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.964 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.964 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.964 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.964 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.964 INFO:teuthology.orchestra.run.smithi097.stdout: "PROMETHEUS_API_HOST": { 2024-03-11T20:09:23.964 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-11T20:09:23.964 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.965 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.965 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.965 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.965 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.965 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.965 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.965 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.965 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.965 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.965 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.965 INFO:teuthology.orchestra.run.smithi097.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-11T20:09:23.965 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-11T20:09:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.966 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-11T20:09:23.967 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-11T20:09:23.967 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.967 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.967 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.967 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.967 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.967 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.967 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.967 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.967 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.967 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.967 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.968 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-11T20:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-11T20:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.970 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.970 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.970 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-11T20:09:23.970 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-11T20:09:23.970 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.970 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.970 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.970 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.970 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.970 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.970 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.970 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.971 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.971 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.971 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.971 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.971 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-11T20:09:23.971 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-11T20:09:23.971 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.971 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.971 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.971 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.971 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.971 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.972 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.972 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.972 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.972 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.972 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.972 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-11T20:09:23.972 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-11T20:09:23.972 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.972 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.972 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.972 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.972 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.973 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.973 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.973 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.973 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.973 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.973 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.973 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-11T20:09:23.973 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-11T20:09:23.973 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.973 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.973 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.973 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_ENABLED": { 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.974 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.975 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.975 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.975 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.975 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.975 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.975 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.975 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.975 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.975 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-11T20:09:23.975 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-11T20:09:23.975 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.975 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.976 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.976 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-11T20:09:23.976 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.976 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.976 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.976 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.976 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.976 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.976 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.976 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-11T20:09:23.976 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-11T20:09:23.976 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.977 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.977 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.977 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:23.977 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.977 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.977 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.977 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.977 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.977 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.977 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.977 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-11T20:09:23.978 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-11T20:09:23.978 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.978 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.978 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.978 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8", 2024-03-11T20:09:23.978 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.978 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.978 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.978 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.978 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.978 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.979 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.979 INFO:teuthology.orchestra.run.smithi097.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-11T20:09:23.979 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-11T20:09:23.979 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.979 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.979 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.979 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "45", 2024-03-11T20:09:23.979 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.979 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.979 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.979 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.980 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.980 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.980 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.980 INFO:teuthology.orchestra.run.smithi097.stdout: "RGW_API_ACCESS_KEY": { 2024-03-11T20:09:23.980 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-11T20:09:23.980 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.980 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.980 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.980 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.980 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.980 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.981 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.981 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.981 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.981 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.981 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.981 INFO:teuthology.orchestra.run.smithi097.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-11T20:09:23.981 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-11T20:09:23.981 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.981 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.981 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.981 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "admin", 2024-03-11T20:09:23.981 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.982 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.982 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.982 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.982 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.982 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.982 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.982 INFO:teuthology.orchestra.run.smithi097.stdout: "RGW_API_SECRET_KEY": { 2024-03-11T20:09:23.982 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-11T20:09:23.982 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.982 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.982 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.983 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.983 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.983 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.983 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.983 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.983 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.983 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.983 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.983 INFO:teuthology.orchestra.run.smithi097.stdout: "RGW_API_SSL_VERIFY": { 2024-03-11T20:09:23.983 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-11T20:09:23.983 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.983 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.984 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.984 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:23.984 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.984 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.984 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.984 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.984 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.984 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.984 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.984 INFO:teuthology.orchestra.run.smithi097.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-11T20:09:23.984 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-11T20:09:23.984 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.985 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.985 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.985 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:23.985 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.985 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.985 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.985 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.985 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.985 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.985 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.985 INFO:teuthology.orchestra.run.smithi097.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-11T20:09:23.985 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-11T20:09:23.986 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.986 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.986 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.986 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:23.986 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.986 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.986 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.986 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.986 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.986 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.986 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.987 INFO:teuthology.orchestra.run.smithi097.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-11T20:09:23.987 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-11T20:09:23.987 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.987 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.987 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.987 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5", 2024-03-11T20:09:23.987 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.987 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.987 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.987 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.987 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.987 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.987 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.988 INFO:teuthology.orchestra.run.smithi097.stdout: "cross_origin_url": { 2024-03-11T20:09:23.988 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cross_origin_url", 2024-03-11T20:09:23.988 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.988 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.988 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.988 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.988 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.988 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.988 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.988 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.988 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.988 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.989 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.989 INFO:teuthology.orchestra.run.smithi097.stdout: "crt_file": { 2024-03-11T20:09:23.989 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crt_file", 2024-03-11T20:09:23.989 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.989 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.989 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.989 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.989 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.989 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.989 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.989 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.989 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.990 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.990 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.990 INFO:teuthology.orchestra.run.smithi097.stdout: "debug": { 2024-03-11T20:09:23.990 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "debug", 2024-03-11T20:09:23.990 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.990 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.990 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.990 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.990 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.990 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.990 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.990 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Enable/disable debug options", 2024-03-11T20:09:23.991 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.991 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.991 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.991 INFO:teuthology.orchestra.run.smithi097.stdout: "jwt_token_ttl": { 2024-03-11T20:09:23.991 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "jwt_token_ttl", 2024-03-11T20:09:23.991 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:23.991 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.991 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.991 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "28800", 2024-03-11T20:09:23.991 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.991 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.991 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.992 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.992 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.992 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.992 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.992 INFO:teuthology.orchestra.run.smithi097.stdout: "key_file": { 2024-03-11T20:09:23.992 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "key_file", 2024-03-11T20:09:23.992 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.992 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.992 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.992 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.992 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.993 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.993 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.993 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.993 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.993 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.993 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.993 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:23.993 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:23.993 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.993 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.993 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.994 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.994 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.994 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.994 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:23.994 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:23.994 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:23.994 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:23.994 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:23.994 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:23.994 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:23.994 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:23.994 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.994 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.996 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.996 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.996 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.996 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:23.996 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:23.996 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.996 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.996 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.996 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:23.996 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.996 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.996 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:23.997 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:23.997 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:23.997 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:23.997 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:23.997 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:23.997 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:23.997 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:23.997 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.997 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.997 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.997 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.998 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:23.998 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:23.998 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:23.998 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.998 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:23.998 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:23.998 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.998 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.998 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.998 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:23.998 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.998 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:23.998 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: "motd": { 2024-03-11T20:09:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "motd", 2024-03-11T20:09:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "The message of the day", 2024-03-11T20:09:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:23.999 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.000 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.000 INFO:teuthology.orchestra.run.smithi097.stdout: "redirect_resolve_ip_addr": { 2024-03-11T20:09:24.000 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "redirect_resolve_ip_addr", 2024-03-11T20:09:24.000 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.000 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.000 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.000 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.000 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.000 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.000 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.000 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.001 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.001 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.001 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.001 INFO:teuthology.orchestra.run.smithi097.stdout: "server_addr": { 2024-03-11T20:09:24.001 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_addr", 2024-03-11T20:09:24.001 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.001 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.001 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.001 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "::", 2024-03-11T20:09:24.001 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.001 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.001 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.002 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.002 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.002 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.002 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.002 INFO:teuthology.orchestra.run.smithi097.stdout: "server_port": { 2024-03-11T20:09:24.002 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_port", 2024-03-11T20:09:24.002 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.002 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.002 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.002 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8080", 2024-03-11T20:09:24.002 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.002 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.003 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.003 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.003 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.003 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.003 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.003 INFO:teuthology.orchestra.run.smithi097.stdout: "ssl": { 2024-03-11T20:09:24.003 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ssl", 2024-03-11T20:09:24.003 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.003 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.003 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.003 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.003 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.004 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.004 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.004 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.004 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.004 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.004 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.004 INFO:teuthology.orchestra.run.smithi097.stdout: "ssl_server_port": { 2024-03-11T20:09:24.004 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ssl_server_port", 2024-03-11T20:09:24.004 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.004 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.004 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.005 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8443", 2024-03-11T20:09:24.005 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.005 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.005 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.005 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.005 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.005 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.005 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.005 INFO:teuthology.orchestra.run.smithi097.stdout: "standby_behaviour": { 2024-03-11T20:09:24.005 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "standby_behaviour", 2024-03-11T20:09:24.005 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.005 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.006 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.006 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "redirect", 2024-03-11T20:09:24.006 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.006 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.006 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.006 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.006 INFO:teuthology.orchestra.run.smithi097.stdout: "redirect" 2024-03-11T20:09:24.006 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.006 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.006 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.006 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.006 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.007 INFO:teuthology.orchestra.run.smithi097.stdout: "standby_error_status_code": { 2024-03-11T20:09:24.007 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "standby_error_status_code", 2024-03-11T20:09:24.007 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.007 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.007 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.007 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "500", 2024-03-11T20:09:24.007 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "400", 2024-03-11T20:09:24.007 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "599", 2024-03-11T20:09:24.007 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.007 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.007 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.008 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.008 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.008 INFO:teuthology.orchestra.run.smithi097.stdout: "url_prefix": { 2024-03-11T20:09:24.008 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "url_prefix", 2024-03-11T20:09:24.008 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.008 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.008 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.008 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.008 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.008 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.009 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.009 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.009 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.009 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.009 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.009 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.009 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.009 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "devicehealth", 2024-03-11T20:09:24.009 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.010 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.010 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.010 INFO:teuthology.orchestra.run.smithi097.stdout: "enable_monitoring": { 2024-03-11T20:09:24.010 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "enable_monitoring", 2024-03-11T20:09:24.010 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.010 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.010 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.010 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.010 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.010 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.010 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.011 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "monitor device health metrics", 2024-03-11T20:09:24.011 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.011 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.011 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.011 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.011 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.011 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.011 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.011 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.011 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.011 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.012 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.012 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.012 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.012 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.012 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.012 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.012 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.012 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.012 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.012 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.012 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.012 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.012 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.013 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.013 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.013 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.013 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.013 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.013 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.013 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.013 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.013 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.013 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.013 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.013 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.014 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.014 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.014 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.014 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.014 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.014 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.014 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.014 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.014 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.014 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.014 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.014 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.015 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.015 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.015 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.015 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.015 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.015 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.015 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.015 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.015 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.015 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.015 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.016 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.016 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.016 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.016 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.016 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.016 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.016 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.016 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.016 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.016 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.016 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.016 INFO:teuthology.orchestra.run.smithi097.stdout: "mark_out_threshold": { 2024-03-11T20:09:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mark_out_threshold", 2024-03-11T20:09:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "2419200", 2024-03-11T20:09:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-11T20:09:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.018 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_name": { 2024-03-11T20:09:24.018 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "pool_name", 2024-03-11T20:09:24.018 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.018 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.018 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.018 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "device_health_metrics", 2024-03-11T20:09:24.018 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.018 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.018 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.018 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-11T20:09:24.018 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.018 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.018 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.019 INFO:teuthology.orchestra.run.smithi097.stdout: "retention_period": { 2024-03-11T20:09:24.019 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "retention_period", 2024-03-11T20:09:24.019 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.019 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.019 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.019 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "15552000", 2024-03-11T20:09:24.019 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.019 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.019 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.019 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how long to retain device health metrics", 2024-03-11T20:09:24.019 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.019 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: "scrape_frequency": { 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "scrape_frequency", 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "86400", 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.020 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.021 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.021 INFO:teuthology.orchestra.run.smithi097.stdout: "self_heal": { 2024-03-11T20:09:24.021 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "self_heal", 2024-03-11T20:09:24.021 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.021 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.021 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.021 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.021 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.021 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.021 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.021 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-11T20:09:24.021 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.022 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.022 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.022 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.022 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.022 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.022 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.022 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.022 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:24.022 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.022 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.022 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.023 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to wake up and check device health", 2024-03-11T20:09:24.023 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.023 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.023 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.023 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_threshold": { 2024-03-11T20:09:24.023 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_threshold", 2024-03-11T20:09:24.023 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.023 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.023 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.023 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "7257600", 2024-03-11T20:09:24.023 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.024 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.024 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.024 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-11T20:09:24.024 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.024 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.024 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.024 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.024 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.024 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "diskprediction_local", 2024-03-11T20:09:24.024 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.024 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.025 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.025 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.025 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.025 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.025 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.025 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.025 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.025 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.025 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.025 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.025 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.025 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.025 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.026 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.026 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.026 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.026 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.026 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.026 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.026 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.026 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.026 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.026 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.026 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.027 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.027 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.027 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.027 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.027 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.027 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.027 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.027 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.027 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.027 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.027 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.028 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.028 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.028 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.028 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.028 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.028 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.028 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.028 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.028 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.028 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.028 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.028 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.029 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.029 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.029 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.029 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.029 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.029 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.029 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.029 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.029 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.029 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.029 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.029 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.030 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.030 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.030 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.030 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.030 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.030 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.030 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.030 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.030 INFO:teuthology.orchestra.run.smithi097.stdout: "predict_interval": { 2024-03-11T20:09:24.030 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "predict_interval", 2024-03-11T20:09:24.030 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "86400", 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: "predictor_model": { 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "predictor_model", 2024-03-11T20:09:24.031 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.032 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.032 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.032 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "prophetstor", 2024-03-11T20:09:24.032 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.032 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.032 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.032 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.032 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.032 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.032 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.032 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.032 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.033 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.033 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.033 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.033 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:24.033 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.033 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.033 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.033 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.033 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.033 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.033 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.033 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.033 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.034 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "influx", 2024-03-11T20:09:24.034 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": false, 2024-03-11T20:09:24.034 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "influxdb python module not found", 2024-03-11T20:09:24.034 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.034 INFO:teuthology.orchestra.run.smithi097.stdout: "batch_size": { 2024-03-11T20:09:24.034 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "batch_size", 2024-03-11T20:09:24.034 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.034 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.034 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.034 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5000", 2024-03-11T20:09:24.034 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.035 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.035 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.035 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-11T20:09:24.035 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.035 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.035 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.035 INFO:teuthology.orchestra.run.smithi097.stdout: "database": { 2024-03-11T20:09:24.035 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "database", 2024-03-11T20:09:24.035 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.035 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.035 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.035 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "ceph", 2024-03-11T20:09:24.036 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.036 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.036 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.036 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-11T20:09:24.036 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.036 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.036 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.036 INFO:teuthology.orchestra.run.smithi097.stdout: "hostname": { 2024-03-11T20:09:24.036 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "hostname", 2024-03-11T20:09:24.036 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.036 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.036 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.037 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.037 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.037 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.037 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.037 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "InfluxDB server hostname", 2024-03-11T20:09:24.037 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.037 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.037 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.037 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:24.037 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:24.037 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.038 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.038 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.038 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "30", 2024-03-11T20:09:24.038 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "5", 2024-03-11T20:09:24.038 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.038 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.038 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-11T20:09:24.038 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.038 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.038 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.038 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.038 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.039 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.039 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.039 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.039 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.039 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.039 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.039 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.039 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.039 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.039 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.039 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.039 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.039 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.040 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.040 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.040 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.040 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.040 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.040 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.040 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.040 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.040 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.040 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.040 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.040 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.041 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.042 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.042 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.042 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.042 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.042 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.042 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.042 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.042 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.042 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.042 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.042 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.042 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.042 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.043 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.043 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.043 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.043 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.043 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.043 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.043 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.043 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.043 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.043 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.043 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.043 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.044 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.044 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.044 INFO:teuthology.orchestra.run.smithi097.stdout: "password": { 2024-03-11T20:09:24.044 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "password", 2024-03-11T20:09:24.044 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.044 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.044 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.044 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.044 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.044 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.044 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.044 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "password of InfluxDB server user", 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: "port": { 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "port", 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8086", 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.045 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "InfluxDB server port", 2024-03-11T20:09:24.046 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.046 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.046 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.046 INFO:teuthology.orchestra.run.smithi097.stdout: "ssl": { 2024-03-11T20:09:24.046 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ssl", 2024-03-11T20:09:24.046 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.046 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.046 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.046 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "false", 2024-03-11T20:09:24.046 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.046 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.047 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.047 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T20:09:24.047 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.047 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.047 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.047 INFO:teuthology.orchestra.run.smithi097.stdout: "threads": { 2024-03-11T20:09:24.047 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "threads", 2024-03-11T20:09:24.047 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.047 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.047 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.047 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5", 2024-03-11T20:09:24.047 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1", 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "32", 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: "username": { 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "username", 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.048 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.049 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.049 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.049 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "username of InfluxDB server user", 2024-03-11T20:09:24.049 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.049 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.049 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.049 INFO:teuthology.orchestra.run.smithi097.stdout: "verify_ssl": { 2024-03-11T20:09:24.049 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "verify_ssl", 2024-03-11T20:09:24.049 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.049 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.049 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.049 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "true", 2024-03-11T20:09:24.050 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.050 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.050 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.050 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T20:09:24.050 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.050 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.050 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.050 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.050 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.050 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "insights", 2024-03-11T20:09:24.050 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.051 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.051 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.051 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.051 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.051 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.051 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.051 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.051 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.051 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.051 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.051 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.051 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.051 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.052 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.052 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.052 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.052 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.052 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.052 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.052 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.052 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.052 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.052 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.053 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.053 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.053 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.053 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.053 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.053 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.053 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.053 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.053 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.053 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.053 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.053 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.054 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.054 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.054 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.054 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.054 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.054 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.054 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.054 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.054 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.054 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.054 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.054 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.055 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.055 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.055 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.055 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.055 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.055 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.055 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.055 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.055 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.055 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.055 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.056 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.056 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.056 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.056 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.056 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.056 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.056 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.056 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.056 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.056 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.056 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.056 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.057 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.057 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "iostat", 2024-03-11T20:09:24.057 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.057 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.057 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.057 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.057 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.057 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.057 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.057 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.057 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.057 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.057 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.058 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.058 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.058 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.058 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.058 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.058 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.058 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.058 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.058 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.058 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.058 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.058 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.059 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.059 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.059 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.059 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.059 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.059 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.059 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.059 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.059 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.059 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.059 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.059 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.060 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.060 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.060 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.060 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.060 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.060 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.060 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.060 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.060 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.060 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.060 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.060 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.062 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.062 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.062 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.062 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.062 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.062 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.062 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.063 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.063 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.063 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.063 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.063 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.063 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.063 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.063 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.063 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.063 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.063 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.063 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.064 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.064 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.064 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.064 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.064 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.064 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "k8sevents", 2024-03-11T20:09:24.064 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.064 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.064 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.064 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_event_retention_days": { 2024-03-11T20:09:24.064 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ceph_event_retention_days", 2024-03-11T20:09:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "7", 2024-03-11T20:09:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-11T20:09:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: "config_check_secs": { 2024-03-11T20:09:24.066 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "config_check_secs", 2024-03-11T20:09:24.066 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.066 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.066 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.066 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.066 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "10", 2024-03-11T20:09:24.066 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.066 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.066 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-11T20:09:24.066 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.066 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.066 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.067 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.068 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.068 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.068 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.068 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.068 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.068 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.068 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.068 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.068 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.068 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.068 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.069 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.069 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.069 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.069 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.069 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.069 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.069 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.069 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.069 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.070 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.070 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.070 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.070 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.070 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.070 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.070 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.070 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.070 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.070 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.070 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.070 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.071 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.071 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.071 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.071 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.071 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.071 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.071 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.071 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.071 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.071 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.071 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.071 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.072 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.072 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.072 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.072 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.072 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.072 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.072 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.072 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.072 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.072 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.072 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.073 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "localpool", 2024-03-11T20:09:24.073 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.073 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.073 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.073 INFO:teuthology.orchestra.run.smithi097.stdout: "failure_domain": { 2024-03-11T20:09:24.073 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "failure_domain", 2024-03-11T20:09:24.073 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.073 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.073 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.073 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "host", 2024-03-11T20:09:24.073 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.073 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.073 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.074 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "failure domain for any created local pool", 2024-03-11T20:09:24.074 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-11T20:09:24.074 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.074 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.074 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.074 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.074 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.074 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.074 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.074 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.074 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.075 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.075 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.075 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.075 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.075 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.075 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.075 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.075 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.075 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.075 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.075 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.075 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.075 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.076 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.076 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.076 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.076 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.076 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.076 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.076 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.076 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.076 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.076 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.076 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.076 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.077 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.077 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.077 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.077 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.077 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.077 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.077 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.077 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.077 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.077 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.077 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.077 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.078 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.078 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.078 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.078 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.078 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.078 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.078 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.078 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.078 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.078 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.078 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.078 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.079 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.079 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.079 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.079 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.079 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.079 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.079 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.079 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.079 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.079 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.079 INFO:teuthology.orchestra.run.smithi097.stdout: "min_size": { 2024-03-11T20:09:24.079 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "min_size", 2024-03-11T20:09:24.080 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.080 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.080 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.080 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.080 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.080 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.080 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.080 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "default min_size for any created local pool", 2024-03-11T20:09:24.080 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-11T20:09:24.080 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.080 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.080 INFO:teuthology.orchestra.run.smithi097.stdout: "num_rep": { 2024-03-11T20:09:24.081 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "num_rep", 2024-03-11T20:09:24.081 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.081 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.081 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.081 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "3", 2024-03-11T20:09:24.081 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.081 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.081 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.081 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "default replica count for any created local pool", 2024-03-11T20:09:24.081 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.081 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.081 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.082 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num": { 2024-03-11T20:09:24.082 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "pg_num", 2024-03-11T20:09:24.082 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.082 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.082 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.082 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "128", 2024-03-11T20:09:24.082 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.082 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.082 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.082 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "default pg_num for any created local pool", 2024-03-11T20:09:24.082 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.082 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.082 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.083 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.083 INFO:teuthology.orchestra.run.smithi097.stdout: "prefix": { 2024-03-11T20:09:24.083 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "prefix", 2024-03-11T20:09:24.083 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.083 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.083 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.083 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.083 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.083 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.083 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.083 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "name prefix for any created local pool", 2024-03-11T20:09:24.083 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.084 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.084 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.084 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.084 INFO:teuthology.orchestra.run.smithi097.stdout: "subtree": { 2024-03-11T20:09:24.084 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "subtree", 2024-03-11T20:09:24.084 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.084 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.084 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.084 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "rack", 2024-03-11T20:09:24.084 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.084 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.084 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.085 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-11T20:09:24.085 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-11T20:09:24.085 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.085 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.085 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.085 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.085 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.085 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mds_autoscaler", 2024-03-11T20:09:24.085 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.085 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.085 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.085 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.086 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.086 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.086 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.086 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.086 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.086 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.086 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.086 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.086 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.086 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.086 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.086 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.086 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.087 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.087 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.087 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.087 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.087 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.087 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.087 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.087 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.087 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.087 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.087 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.087 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.088 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.088 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.088 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.088 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.088 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.088 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.088 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.088 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.088 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.088 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.088 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.088 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.090 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.090 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.090 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.090 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.090 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.090 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.090 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.090 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.090 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.090 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.090 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.090 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.090 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.091 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.091 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.091 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.091 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.091 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.091 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.091 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mirroring", 2024-03-11T20:09:24.091 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.091 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.091 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.091 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.091 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.092 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.092 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.092 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.092 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.092 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.092 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.092 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.092 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.092 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.092 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.092 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.092 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.093 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.093 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.093 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.093 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.093 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.093 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.093 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.093 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.093 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.093 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.093 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.093 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.094 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.094 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.094 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.094 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.094 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.094 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.094 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.094 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.094 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.094 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.094 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.095 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.095 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.095 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.095 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.095 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.095 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.095 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.095 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.095 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.095 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.095 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.095 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.096 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.096 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.096 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.096 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.096 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.096 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.096 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.096 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.096 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.096 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.096 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "nfs", 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.097 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.098 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.098 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.098 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.098 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.098 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.098 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.098 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.098 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.098 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.098 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.098 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.098 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.099 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.099 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.099 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.099 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.099 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.099 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.099 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.099 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.099 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.099 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.099 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.099 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.100 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.100 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.100 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.100 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.100 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.100 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.100 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.100 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.100 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.100 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.100 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.100 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.101 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.101 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.101 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.101 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.101 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.101 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.101 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.101 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.101 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.101 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.101 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.101 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.101 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.102 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.102 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.102 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.102 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.102 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.102 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.102 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.102 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.102 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.102 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.102 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.103 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.103 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.103 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.103 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.103 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.103 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.103 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.103 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.103 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.103 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "orchestrator", 2024-03-11T20:09:24.104 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.104 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.104 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.104 INFO:teuthology.orchestra.run.smithi097.stdout: "fail_fs": { 2024-03-11T20:09:24.104 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "fail_fs", 2024-03-11T20:09:24.104 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.104 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.104 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.104 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.104 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.104 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.104 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.105 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-11T20:09:24.105 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.105 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.105 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.105 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.105 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.105 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.105 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.105 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.105 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.105 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.105 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.106 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.106 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.106 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.106 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.106 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.106 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.106 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.106 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.106 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.106 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.106 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.106 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.107 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.108 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.108 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.108 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.108 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.108 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.108 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.108 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.108 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.108 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.108 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.108 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.108 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.108 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.109 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.109 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.109 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.109 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.109 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.109 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.109 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.109 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.109 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.109 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.109 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.109 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.110 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.110 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.110 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.110 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.110 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.110 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.110 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.110 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.110 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.110 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.110 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.110 INFO:teuthology.orchestra.run.smithi097.stdout: "orchestrator": { 2024-03-11T20:09:24.110 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "orchestrator", 2024-03-11T20:09:24.111 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.111 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.111 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.111 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.111 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.111 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.111 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.111 INFO:teuthology.orchestra.run.smithi097.stdout: "cephadm", 2024-03-11T20:09:24.111 INFO:teuthology.orchestra.run.smithi097.stdout: "rook", 2024-03-11T20:09:24.111 INFO:teuthology.orchestra.run.smithi097.stdout: "test_orchestrator" 2024-03-11T20:09:24.111 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.111 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Orchestrator backend", 2024-03-11T20:09:24.111 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.112 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.112 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.112 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.112 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.112 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "osd_perf_query", 2024-03-11T20:09:24.112 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.112 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.112 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.112 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.112 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.112 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.113 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.114 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.114 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.114 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.114 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.114 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.114 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.114 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.114 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.114 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.114 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.114 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.115 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.115 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.115 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.115 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.115 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.115 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.115 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.115 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.115 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.115 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.115 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.116 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.116 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.116 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.116 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.116 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.116 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.116 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.116 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.116 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.116 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.116 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.116 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.116 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.117 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.117 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.117 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.117 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.117 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.117 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.117 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.117 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.117 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.117 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.117 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.117 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.118 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.118 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.118 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.118 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.118 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.118 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "osd_support", 2024-03-11T20:09:24.118 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.118 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.118 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.118 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.118 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.119 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.119 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.119 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.119 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.119 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.119 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.119 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.119 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.119 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.119 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.119 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.119 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.119 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.120 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.120 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.120 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.120 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.120 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.120 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.120 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.120 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.120 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.120 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.120 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.121 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.121 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.121 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.121 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.121 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.121 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.121 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.121 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.121 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.121 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.121 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.121 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.121 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.122 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.122 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.122 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.122 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.122 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.122 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.122 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.122 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.122 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.122 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.122 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.122 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.123 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.124 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.124 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.124 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.124 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.124 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.124 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "pg_autoscaler", 2024-03-11T20:09:24.124 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.124 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.124 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.124 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.124 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.125 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.125 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.125 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.125 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.125 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.125 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.125 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.125 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.125 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.125 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.125 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.125 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.125 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.126 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.126 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.126 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.126 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.126 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.126 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.126 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.126 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.126 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.126 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.126 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.126 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.127 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.127 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.127 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.127 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.127 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.127 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.127 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.127 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.127 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.127 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.127 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.127 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.127 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.128 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.129 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.129 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.129 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.129 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.129 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.129 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.129 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.129 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.129 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.129 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.129 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.129 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.130 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.130 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.130 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.130 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.130 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.130 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.130 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:24.130 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.130 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.130 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.130 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.131 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.131 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.131 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.131 INFO:teuthology.orchestra.run.smithi097.stdout: "threshold": { 2024-03-11T20:09:24.131 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "threshold", 2024-03-11T20:09:24.131 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.131 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.131 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.131 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "3.0", 2024-03-11T20:09:24.131 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1.0", 2024-03-11T20:09:24.132 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.132 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.132 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "scaling threshold", 2024-03-11T20:09:24.132 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-11T20:09:24.132 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.132 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.132 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.132 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.132 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "progress", 2024-03-11T20:09:24.132 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.132 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.133 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.133 INFO:teuthology.orchestra.run.smithi097.stdout: "allow_pg_recovery_event": { 2024-03-11T20:09:24.133 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "allow_pg_recovery_event", 2024-03-11T20:09:24.133 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.133 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.133 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.133 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.133 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.133 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.133 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.133 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-11T20:09:24.133 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.133 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.134 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.134 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.134 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": { 2024-03-11T20:09:24.134 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "enabled", 2024-03-11T20:09:24.134 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.134 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.134 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.134 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.134 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.134 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.134 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.135 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.135 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.135 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.135 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.135 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.135 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.135 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.135 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.135 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.135 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.135 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.135 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.136 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.136 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.136 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.136 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.136 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.136 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.136 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.136 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.136 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.136 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.136 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.136 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.136 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.137 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.137 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.137 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.137 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.137 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.137 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.137 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.137 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.137 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.137 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.137 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.137 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.138 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.138 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.138 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.138 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.138 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.138 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.138 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.138 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.138 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.138 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.138 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.138 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.139 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.139 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.139 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.139 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.139 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.139 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.139 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.139 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.139 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.139 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.139 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.139 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.139 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.140 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.140 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.140 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.140 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.140 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.140 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.140 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.140 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.140 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.140 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.140 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.140 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.140 INFO:teuthology.orchestra.run.smithi097.stdout: "max_completed_events": { 2024-03-11T20:09:24.141 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_completed_events", 2024-03-11T20:09:24.141 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.141 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.141 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.141 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "50", 2024-03-11T20:09:24.141 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.141 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.141 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.141 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "number of past completed events to remember", 2024-03-11T20:09:24.141 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.141 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.141 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.141 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.142 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.142 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.142 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.142 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.142 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.142 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5", 2024-03-11T20:09:24.142 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.142 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.142 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.142 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how long the module is going to sleep", 2024-03-11T20:09:24.142 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.142 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.142 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.143 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.143 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.143 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.143 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "prometheus", 2024-03-11T20:09:24.143 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.143 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.143 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.143 INFO:teuthology.orchestra.run.smithi097.stdout: "cache": { 2024-03-11T20:09:24.143 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cache", 2024-03-11T20:09:24.143 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.143 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.144 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.144 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.144 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.144 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.144 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.144 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.144 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.144 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.144 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.144 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.144 INFO:teuthology.orchestra.run.smithi097.stdout: "exclude_perf_counters": { 2024-03-11T20:09:24.144 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "exclude_perf_counters", 2024-03-11T20:09:24.144 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.145 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.145 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.145 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.145 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.145 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.145 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.145 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-11T20:09:24.145 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-11T20:09:24.145 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.145 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.146 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.146 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.146 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.146 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.146 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.146 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.146 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.146 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.146 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.146 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.146 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.147 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.147 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.147 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.147 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.147 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.147 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.147 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.147 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.147 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.147 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.147 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.148 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.148 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.148 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.148 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.148 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.148 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.148 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.148 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.148 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.148 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.148 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.148 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.149 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.149 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.149 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.149 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.149 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.149 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.149 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.149 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.149 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.149 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.149 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.149 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.150 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.150 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.150 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.150 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.150 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.150 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.150 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.150 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.150 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.150 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.150 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.151 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.151 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.151 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.151 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.151 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.151 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.151 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.151 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.151 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.151 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.151 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_stats_pools": { 2024-03-11T20:09:24.151 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rbd_stats_pools", 2024-03-11T20:09:24.152 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.152 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.152 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.152 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.152 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.152 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.152 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.152 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.152 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.152 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.152 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.152 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.152 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-11T20:09:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-11T20:09:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "300", 2024-03-11T20:09:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.153 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "scrape_interval": { 2024-03-11T20:09:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "scrape_interval", 2024-03-11T20:09:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "15.0", 2024-03-11T20:09:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.154 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.155 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.155 INFO:teuthology.orchestra.run.smithi097.stdout: "server_addr": { 2024-03-11T20:09:24.155 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_addr", 2024-03-11T20:09:24.155 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.155 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.155 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.155 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "::", 2024-03-11T20:09:24.155 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.155 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.155 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.155 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-11T20:09:24.155 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: "server_port": { 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_port", 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "9283", 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-11T20:09:24.156 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.157 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.157 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.157 INFO:teuthology.orchestra.run.smithi097.stdout: "stale_cache_strategy": { 2024-03-11T20:09:24.157 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "stale_cache_strategy", 2024-03-11T20:09:24.157 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.157 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.157 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.157 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "log", 2024-03-11T20:09:24.157 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.157 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.157 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.157 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.158 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.158 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.158 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.158 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.158 INFO:teuthology.orchestra.run.smithi097.stdout: "standby_behaviour": { 2024-03-11T20:09:24.158 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "standby_behaviour", 2024-03-11T20:09:24.158 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.158 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.158 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.158 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "default", 2024-03-11T20:09:24.158 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.158 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.159 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.159 INFO:teuthology.orchestra.run.smithi097.stdout: "default", 2024-03-11T20:09:24.159 INFO:teuthology.orchestra.run.smithi097.stdout: "error" 2024-03-11T20:09:24.159 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.159 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.159 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.159 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.159 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.159 INFO:teuthology.orchestra.run.smithi097.stdout: "standby_error_status_code": { 2024-03-11T20:09:24.159 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "standby_error_status_code", 2024-03-11T20:09:24.159 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.160 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.160 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.160 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "500", 2024-03-11T20:09:24.160 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "400", 2024-03-11T20:09:24.160 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "599", 2024-03-11T20:09:24.160 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.160 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.160 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.160 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.160 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.160 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.160 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.161 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rbd_support", 2024-03-11T20:09:24.161 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.161 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.161 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.161 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.161 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.161 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.161 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.161 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.161 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.161 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.162 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.162 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.162 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.162 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.162 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.162 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.162 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.162 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.162 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.162 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.162 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.162 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.163 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.163 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.163 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.163 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.163 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.163 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.163 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.163 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.163 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.163 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.163 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.164 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.164 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.164 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.164 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.164 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.164 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.164 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.164 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.164 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.164 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.164 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.164 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.164 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.165 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.165 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.165 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.165 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.165 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.165 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.165 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.165 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.165 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.165 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.165 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.165 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.166 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.166 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.166 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.166 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.166 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.166 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.166 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.166 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.166 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.166 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.166 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.166 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.167 INFO:teuthology.orchestra.run.smithi097.stdout: "max_concurrent_snap_create": { 2024-03-11T20:09:24.167 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_concurrent_snap_create", 2024-03-11T20:09:24.167 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.167 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.167 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.167 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.167 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.167 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.167 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.167 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.167 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.167 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.168 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.168 INFO:teuthology.orchestra.run.smithi097.stdout: "mirror_snapshot_schedule": { 2024-03-11T20:09:24.168 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mirror_snapshot_schedule", 2024-03-11T20:09:24.168 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.168 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.168 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.168 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.168 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.168 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.168 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.168 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.168 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: "trash_purge_schedule": { 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "trash_purge_schedule", 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.169 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.170 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.170 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.170 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.170 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.170 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "restful", 2024-03-11T20:09:24.170 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.170 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.170 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.170 INFO:teuthology.orchestra.run.smithi097.stdout: "enable_auth": { 2024-03-11T20:09:24.170 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "enable_auth", 2024-03-11T20:09:24.170 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.171 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.171 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.171 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.171 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.171 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.171 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.171 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.171 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.171 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.171 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.171 INFO:teuthology.orchestra.run.smithi097.stdout: "key_file": { 2024-03-11T20:09:24.171 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "key_file", 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.172 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.174 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.174 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.174 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.174 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.174 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.174 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.174 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.174 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.174 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.174 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.174 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.174 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.175 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.175 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.175 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.175 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.175 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.175 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.176 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.176 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.176 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.176 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.176 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.176 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.176 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.176 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.176 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.176 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.176 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.176 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.177 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.177 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.177 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.177 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.177 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.177 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.177 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.177 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.177 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.177 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.177 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "server_addr": { 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_addr", 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.178 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: "server_port": { 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_port", 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.179 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rgw", 2024-03-11T20:09:24.180 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.181 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.182 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.183 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.184 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.185 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rook", 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.186 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.187 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.188 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.189 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.190 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.190 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.190 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.190 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.190 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.190 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.190 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.190 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.190 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.190 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.190 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: "storage_class": { 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "storage_class", 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "local", 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "selftest", 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.192 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.193 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.194 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "roption1": { 2024-03-11T20:09:24.196 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "roption1", 2024-03-11T20:09:24.197 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.197 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.197 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.197 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.197 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.197 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.197 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.197 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.197 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.197 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.197 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.197 INFO:teuthology.orchestra.run.smithi097.stdout: "roption2": { 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "roption2", 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "xyz", 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.198 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption1": { 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption1", 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption2": { 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption2", 2024-03-11T20:09:24.199 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption3": { 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption3", 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.200 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption4": { 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption4", 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.201 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.202 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.202 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.202 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.202 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.202 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.202 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.202 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.202 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption5": { 2024-03-11T20:09:24.202 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption5", 2024-03-11T20:09:24.202 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.202 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.202 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.203 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.203 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.203 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.203 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.203 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.203 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.203 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.203 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.203 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption6": { 2024-03-11T20:09:24.203 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption6", 2024-03-11T20:09:24.203 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.203 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption7": { 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption7", 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.204 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1", 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "42", 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: "testkey": { 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "testkey", 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.205 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: "testlkey": { 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "testlkey", 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.206 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "testnewline": { 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "testnewline", 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.207 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "snap_schedule", 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.208 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "allow_m_granularity": { 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "allow_m_granularity", 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "allow minute scheduled snapshots", 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "dump_on_update": { 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "dump_on_update", 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "dump database to debug log on update", 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.210 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.211 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.212 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.213 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.214 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "stats", 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.215 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.216 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.217 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.218 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.219 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.220 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "status", 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.221 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.222 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.223 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.224 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.225 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "telegraf", 2024-03-11T20:09:24.226 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.226 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.226 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.226 INFO:teuthology.orchestra.run.smithi097.stdout: "address": { 2024-03-11T20:09:24.226 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "address", 2024-03-11T20:09:24.226 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.226 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.226 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.226 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-11T20:09:24.226 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.226 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.226 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.226 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "15", 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.227 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.228 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.228 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.228 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.228 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.228 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.228 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.228 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.228 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.228 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.228 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.228 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.229 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.229 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.229 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.229 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.229 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.229 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.229 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.229 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.229 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.229 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.229 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.229 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.230 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.230 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.230 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.230 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.230 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.230 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.230 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.230 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.230 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.230 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.230 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.230 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.231 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.231 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.231 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.231 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.231 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.231 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.231 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.231 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.231 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.231 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.231 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.231 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.231 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.232 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.233 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.233 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.233 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.233 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.233 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.233 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.233 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.233 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.233 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.233 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "telemetry", 2024-03-11T20:09:24.234 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.234 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.234 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.234 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_basic": { 2024-03-11T20:09:24.234 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_basic", 2024-03-11T20:09:24.234 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.234 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.234 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.234 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.234 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.234 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.234 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.234 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_crash": { 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_crash", 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-11T20:09:24.235 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_device": { 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_device", 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-11T20:09:24.236 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_ident": { 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_ident", 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.237 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.238 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.238 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_perf": { 2024-03-11T20:09:24.238 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_perf", 2024-03-11T20:09:24.238 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.238 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.238 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.238 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.238 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.238 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.238 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.238 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-11T20:09:24.238 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: "contact": { 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "contact", 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.239 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "description": { 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "description", 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.240 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "device_url": { 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "device_url", 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.241 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.242 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.242 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": { 2024-03-11T20:09:24.242 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "enabled", 2024-03-11T20:09:24.242 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.242 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.242 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.242 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.242 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.242 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.242 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.242 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.242 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.242 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "24", 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "8", 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.243 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: "last_opt_revision": { 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "last_opt_revision", 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "1", 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.244 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: "leaderboard": { 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "leaderboard", 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.245 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: "leaderboard_description": { 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "leaderboard_description", 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.246 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.247 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.248 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.249 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.249 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.249 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.249 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.249 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.249 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.249 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.249 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.249 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.249 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.249 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.249 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.249 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.250 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.251 INFO:teuthology.orchestra.run.smithi097.stdout: "organization": { 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "organization", 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "proxy": { 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "proxy", 2024-03-11T20:09:24.252 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "url": { 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "url", 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.253 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "test_orchestrator", 2024-03-11T20:09:24.254 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.255 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.256 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.257 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.258 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.259 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "volumes", 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.260 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.261 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.262 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.263 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "max_concurrent_clones": { 2024-03-11T20:09:24.264 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_concurrent_clones", 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "4", 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "periodic_async_work": { 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "periodic_async_work", 2024-03-11T20:09:24.265 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.266 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.266 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.266 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.266 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.266 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.266 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.266 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Periodically check for async work", 2024-03-11T20:09:24.266 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.266 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.266 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.266 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: "snapshot_clone_delay": { 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "snapshot_clone_delay", 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.267 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "zabbix", 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: "discovery_interval": { 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "discovery_interval", 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "100", 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.268 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: "identifier": { 2024-03-11T20:09:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "identifier", 2024-03-11T20:09:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:24.270 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.271 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.272 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.273 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.274 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.275 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.276 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.276 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.276 INFO:teuthology.orchestra.run.smithi097.stdout: "zabbix_host": { 2024-03-11T20:09:24.276 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "zabbix_host", 2024-03-11T20:09:24.276 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.276 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.276 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.276 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.276 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.276 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.276 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.276 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.276 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "zabbix_port": { 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "zabbix_port", 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10051", 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.277 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: "zabbix_sender": { 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "zabbix_sender", 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.278 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: "gid": 25504, 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "z", 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr_features": 4540701547738038271, 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: "available_modules": [ 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "alerts", 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.279 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "How frequently to reexamine health status", 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.280 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.281 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.282 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.283 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.284 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_destination": { 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_destination", 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.285 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Email address to send alerts to", 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_from_name": { 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_from_name", 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.286 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "Ceph", 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Email From: name", 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_host": { 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_host", 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.287 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "SMTP server", 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_password": { 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_password", 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Password to authenticate with", 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_port": { 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_port", 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "465", 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "SMTP port", 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_sender": { 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_sender", 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "SMTP envelope sender", 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_ssl": { 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_ssl", 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_user": { 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_user", 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "User to authenticate as", 2024-03-11T20:09:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "balancer", 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: "active": { 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "active", 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.293 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "automatically balance PGs across cluster", 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "begin_time": { 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "begin_time", 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0000", 2024-03-11T20:09:24.294 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.295 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.295 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.295 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "beginning time of day to automatically balance", 2024-03-11T20:09:24.295 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T20:09:24.295 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.295 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.295 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.295 INFO:teuthology.orchestra.run.smithi097.stdout: "begin_weekday": { 2024-03-11T20:09:24.295 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "begin_weekday", 2024-03-11T20:09:24.295 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:24.295 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.295 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.295 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "0", 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "6", 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_compat_max_iterations": { 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crush_compat_max_iterations", 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "25", 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1", 2024-03-11T20:09:24.296 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "250", 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_compat_metrics": { 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crush_compat_metrics", 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "pgs,objects,bytes", 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.297 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.298 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-11T20:09:24.298 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-11T20:09:24.298 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.298 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.298 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_compat_step": { 2024-03-11T20:09:24.298 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crush_compat_step", 2024-03-11T20:09:24.298 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.298 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.298 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.298 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0.5", 2024-03-11T20:09:24.298 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "0.001", 2024-03-11T20:09:24.298 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "0.999", 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "aggressiveness of optimization", 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "end_time": { 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "end_time", 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "2359", 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.299 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "ending time of day to automatically balance", 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "end_weekday": { 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "end_weekday", 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "0", 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "6", 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-11T20:09:24.300 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.301 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.302 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.303 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.304 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.305 INFO:teuthology.orchestra.run.smithi097.stdout: "min_score": { 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "min_score", 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "mode": { 2024-03-11T20:09:24.306 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mode", 2024-03-11T20:09:24.307 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.307 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.307 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.307 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "upmap", 2024-03-11T20:09:24.307 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.307 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.307 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.307 INFO:teuthology.orchestra.run.smithi097.stdout: "crush-compat", 2024-03-11T20:09:24.307 INFO:teuthology.orchestra.run.smithi097.stdout: "none", 2024-03-11T20:09:24.307 INFO:teuthology.orchestra.run.smithi097.stdout: "read", 2024-03-11T20:09:24.307 INFO:teuthology.orchestra.run.smithi097.stdout: "upmap", 2024-03-11T20:09:24.307 INFO:teuthology.orchestra.run.smithi097.stdout: "upmap-read" 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Balancer mode", 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_ids": { 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "pool_ids", 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.308 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.309 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: "upmap_max_deviation": { 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "upmap_max_deviation", 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5", 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1", 2024-03-11T20:09:24.310 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: "upmap_max_optimizations": { 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "upmap_max_optimizations", 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.311 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cephadm", 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: "agent_down_multiplier": { 2024-03-11T20:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "agent_down_multiplier", 2024-03-11T20:09:24.313 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.313 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.313 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.313 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "3.0", 2024-03-11T20:09:24.313 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.313 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.313 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.313 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-11T20:09:24.313 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.313 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.313 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.313 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.313 INFO:teuthology.orchestra.run.smithi097.stdout: "agent_refresh_rate": { 2024-03-11T20:09:24.314 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "agent_refresh_rate", 2024-03-11T20:09:24.314 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.314 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.314 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.314 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "20", 2024-03-11T20:09:24.314 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.314 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.314 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.314 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-11T20:09:24.314 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.314 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.314 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: "agent_starting_port": { 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "agent_starting_port", 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "4721", 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.315 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "allow_ptrace": { 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "allow_ptrace", 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.316 INFO:teuthology.orchestra.run.smithi097.stdout: "autotune_interval": { 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "autotune_interval", 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "autotune_memory_target_ratio": { 2024-03-11T20:09:24.317 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "autotune_memory_target_ratio", 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0.7", 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: "cephadm_log_destination": { 2024-03-11T20:09:24.318 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cephadm_log_destination", 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "file", 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "file,syslog", 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "syslog" 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.319 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "cgroups_split": { 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cgroups_split", 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.320 INFO:teuthology.orchestra.run.smithi097.stdout: "config_checks_enabled": { 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "config_checks_enabled", 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "config_dashboard": { 2024-03-11T20:09:24.321 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "config_dashboard", 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_alertmanager": { 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_alertmanager", 2024-03-11T20:09:24.322 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Prometheus container image", 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_base": { 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_base", 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.323 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Container image name, without the tag", 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_elasticsearch": { 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_elasticsearch", 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.324 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "elasticsearch container image", 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_grafana": { 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_grafana", 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.325 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Prometheus container image", 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_haproxy": { 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_haproxy", 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.326 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "HAproxy container image", 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_jaeger_agent": { 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_jaeger_agent", 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.327 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Jaeger agent container image", 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_jaeger_collector": { 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_jaeger_collector", 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.328 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Jaeger collector container image", 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_jaeger_query": { 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_jaeger_query", 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.329 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Jaeger query container image", 2024-03-11T20:09:24.330 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.330 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.330 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.330 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_keepalived": { 2024-03-11T20:09:24.330 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_keepalived", 2024-03-11T20:09:24.330 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.330 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.330 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.330 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-11T20:09:24.330 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.330 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.330 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Keepalived container image", 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_loki": { 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_loki", 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.331 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Loki container image", 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_node_exporter": { 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_node_exporter", 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.332 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Prometheus container image", 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_nvmeof": { 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_nvmeof", 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Nvme-of container image", 2024-03-11T20:09:24.333 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_prometheus": { 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_prometheus", 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Prometheus container image", 2024-03-11T20:09:24.334 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_promtail": { 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_promtail", 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Promtail container image", 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.335 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_snmp_gateway": { 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_snmp_gateway", 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "SNMP Gateway container image", 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.336 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: "container_init": { 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_init", 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Run podman/docker with `--init`", 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: "daemon_cache_timeout": { 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "daemon_cache_timeout", 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.338 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "default_cephadm_command_timeout": { 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "default_cephadm_command_timeout", 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "900", 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "default_registry": { 2024-03-11T20:09:24.339 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "default_registry", 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "docker.io", 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "device_cache_timeout": { 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "device_cache_timeout", 2024-03-11T20:09:24.340 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "1800", 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "seconds to cache device inventory", 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "device_enhanced_scan": { 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "device_enhanced_scan", 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.341 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: "facts_cache_timeout": { 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "facts_cache_timeout", 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.342 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "seconds to cache host facts data", 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: "host_check_interval": { 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "host_check_interval", 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.343 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to perform a host check", 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: "hw_monitoring": { 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "hw_monitoring", 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.344 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.345 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.345 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.345 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.345 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.345 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.345 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-11T20:09:24.345 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.345 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.345 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.345 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.345 INFO:teuthology.orchestra.run.smithi097.stdout: "inventory_list_all": { 2024-03-11T20:09:24.345 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "inventory_list_all", 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.346 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.347 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "log_refresh_metadata": { 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_refresh_metadata", 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-11T20:09:24.348 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-11T20:09:24.349 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.350 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.351 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.352 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "*", 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.353 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.354 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.354 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-11T20:09:24.354 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.354 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.354 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.354 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.354 INFO:teuthology.orchestra.run.smithi097.stdout: "max_count_per_host": { 2024-03-11T20:09:24.354 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_count_per_host", 2024-03-11T20:09:24.354 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.354 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.354 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.354 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.354 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "max number of daemons per service per host", 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "max_osd_draining_count": { 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_osd_draining_count", 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.355 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "migration_current": { 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "migration_current", 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "internal - do not modify", 2024-03-11T20:09:24.356 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: "mode": { 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mode", 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "root", 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: "cephadm-package", 2024-03-11T20:09:24.357 INFO:teuthology.orchestra.run.smithi097.stdout: "root" 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "mode for remote execution of cephadm", 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: "oob_default_addr": { 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "oob_default_addr", 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "169.254.1.1", 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.358 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "prometheus_alerts_path": { 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "prometheus_alerts_path", 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.359 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "registry_insecure": { 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "registry_insecure", 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-11T20:09:24.360 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.361 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.361 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.361 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.361 INFO:teuthology.orchestra.run.smithi097.stdout: "registry_password": { 2024-03-11T20:09:24.361 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "registry_password", 2024-03-11T20:09:24.361 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.361 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.361 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.361 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.361 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.361 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.361 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.361 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "registry_url": { 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "registry_url", 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-11T20:09:24.362 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: "registry_username": { 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "registry_username", 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-11T20:09:24.363 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "secure_monitoring_stack": { 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "secure_monitoring_stack", 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.364 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "service_discovery_port": { 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "service_discovery_port", 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8765", 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "cephadm service discovery port", 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.365 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: "ssh_config_file": { 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ssh_config_file", 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.366 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: "use_agent": { 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "use_agent", 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.367 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: "use_repo_digest": { 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "use_repo_digest", 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.368 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_on_failed_host_check": { 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_on_failed_host_check", 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "raise a health warning if the host check fails", 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.369 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_on_stray_daemons": { 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_on_stray_daemons", 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.370 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_on_stray_hosts": { 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_on_stray_hosts", 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.371 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crash", 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.372 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.373 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.374 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.375 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.376 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "retain_interval": { 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "retain_interval", 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "31536000", 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.377 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.378 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.378 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_recent_interval": { 2024-03-11T20:09:24.378 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_recent_interval", 2024-03-11T20:09:24.378 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.378 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.378 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.378 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "1209600", 2024-03-11T20:09:24.378 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.378 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.378 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.378 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-11T20:09:24.378 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "dashboard", 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.379 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "ALERTMANAGER_API_HOST": { 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.380 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.381 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: "AUDIT_API_ENABLED": { 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "AUDIT_API_ENABLED", 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.382 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.383 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "ENABLE_BROWSABLE_API": { 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.384 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.385 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.386 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.387 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.389 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.392 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_API_PASSWORD": { 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "admin", 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-11T20:09:24.393 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_API_URL": { 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_API_URL", 2024-03-11T20:09:24.394 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.395 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.395 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.395 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.395 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.395 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.395 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.395 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.395 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.395 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.395 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.395 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_API_USERNAME": { 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "admin", 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.396 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.397 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-11T20:09:24.398 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-11T20:09:24.398 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.398 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.398 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.398 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.398 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.398 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.398 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.398 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.398 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.398 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.398 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.399 INFO:teuthology.orchestra.run.smithi097.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "PROMETHEUS_API_HOST": { 2024-03-11T20:09:24.400 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-11T20:09:24.401 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.402 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.402 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.402 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.402 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.402 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.402 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.402 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.402 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.402 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.402 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.402 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.402 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-11T20:09:24.402 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-11T20:09:24.403 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-11T20:09:24.404 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.404 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.404 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.404 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.404 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.404 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.404 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.404 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.404 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.404 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.404 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.404 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-11T20:09:24.406 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-11T20:09:24.406 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.406 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.406 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.406 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.406 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.406 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.406 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.406 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.406 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.406 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.406 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.406 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.407 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-11T20:09:24.408 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-11T20:09:24.408 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.408 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.408 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.408 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.408 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.408 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.408 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.408 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.408 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.408 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.408 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.409 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_ENABLED": { 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.410 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-11T20:09:24.411 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.412 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8", 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.413 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "45", 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "RGW_API_ACCESS_KEY": { 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.414 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.415 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "admin", 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "RGW_API_SECRET_KEY": { 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.416 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "RGW_API_SSL_VERIFY": { 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.417 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.418 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.419 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5", 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.420 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "cross_origin_url": { 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cross_origin_url", 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: "crt_file": { 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crt_file", 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: "debug": { 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "debug", 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.423 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Enable/disable debug options", 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "jwt_token_ttl": { 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "jwt_token_ttl", 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "28800", 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.424 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: "key_file": { 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "key_file", 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.425 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.426 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.427 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.428 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.429 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.430 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: "motd": { 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "motd", 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "The message of the day", 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "redirect_resolve_ip_addr": { 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "redirect_resolve_ip_addr", 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.432 INFO:teuthology.orchestra.run.smithi097.stdout: "server_addr": { 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_addr", 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "::", 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "server_port": { 2024-03-11T20:09:24.433 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_port", 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8080", 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: "ssl": { 2024-03-11T20:09:24.434 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ssl", 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: "ssl_server_port": { 2024-03-11T20:09:24.435 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ssl_server_port", 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8443", 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "standby_behaviour": { 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "standby_behaviour", 2024-03-11T20:09:24.436 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "redirect", 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: "redirect" 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.437 INFO:teuthology.orchestra.run.smithi097.stdout: "standby_error_status_code": { 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "standby_error_status_code", 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "500", 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "400", 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "599", 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.438 INFO:teuthology.orchestra.run.smithi097.stdout: "url_prefix": { 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "url_prefix", 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "devicehealth", 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "enable_monitoring": { 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "enable_monitoring", 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.440 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "monitor device health metrics", 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.441 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.442 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.443 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.444 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.445 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: "mark_out_threshold": { 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mark_out_threshold", 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "2419200", 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_name": { 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "pool_name", 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "device_health_metrics", 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: "retention_period": { 2024-03-11T20:09:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "retention_period", 2024-03-11T20:09:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "15552000", 2024-03-11T20:09:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how long to retain device health metrics", 2024-03-11T20:09:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: "scrape_frequency": { 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "scrape_frequency", 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "86400", 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.450 INFO:teuthology.orchestra.run.smithi097.stdout: "self_heal": { 2024-03-11T20:09:24.450 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "self_heal", 2024-03-11T20:09:24.450 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.450 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.450 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.450 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.450 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.450 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.450 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.450 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-11T20:09:24.450 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.450 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to wake up and check device health", 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.451 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_threshold": { 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_threshold", 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "7257600", 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.452 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "diskprediction_local", 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.453 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.454 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.455 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.456 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.457 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.458 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.458 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.458 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.458 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.458 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.458 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.458 INFO:teuthology.orchestra.run.smithi097.stdout: "predict_interval": { 2024-03-11T20:09:24.458 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "predict_interval", 2024-03-11T20:09:24.458 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.458 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.458 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.458 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "86400", 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: "predictor_model": { 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "predictor_model", 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.459 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "prophetstor", 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.460 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "influx", 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": false, 2024-03-11T20:09:24.461 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "influxdb python module not found", 2024-03-11T20:09:24.462 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.462 INFO:teuthology.orchestra.run.smithi097.stdout: "batch_size": { 2024-03-11T20:09:24.462 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "batch_size", 2024-03-11T20:09:24.462 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.462 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.462 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.462 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5000", 2024-03-11T20:09:24.462 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.462 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.462 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.462 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-11T20:09:24.462 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.462 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: "database": { 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "database", 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "ceph", 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.463 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: "hostname": { 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "hostname", 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "InfluxDB server hostname", 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.464 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "30", 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "5", 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.465 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.466 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.467 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.468 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.469 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.469 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.469 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.469 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.469 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.469 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.469 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.469 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.469 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.469 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.469 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.469 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.470 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "password": { 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "password", 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "password of InfluxDB server user", 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.471 INFO:teuthology.orchestra.run.smithi097.stdout: "port": { 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "port", 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8086", 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "InfluxDB server port", 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.472 INFO:teuthology.orchestra.run.smithi097.stdout: "ssl": { 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ssl", 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "false", 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.473 INFO:teuthology.orchestra.run.smithi097.stdout: "threads": { 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "threads", 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5", 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1", 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "32", 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.474 INFO:teuthology.orchestra.run.smithi097.stdout: "username": { 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "username", 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "username of InfluxDB server user", 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "verify_ssl": { 2024-03-11T20:09:24.475 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "verify_ssl", 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "true", 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "insights", 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.477 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.478 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.479 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.481 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.482 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "iostat", 2024-03-11T20:09:24.482 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.482 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.482 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.482 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.482 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.482 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.482 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.482 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.482 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.482 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.483 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.484 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.485 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.486 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "k8sevents", 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_event_retention_days": { 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ceph_event_retention_days", 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.487 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "7", 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "config_check_secs": { 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "config_check_secs", 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.488 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "10", 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.489 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.491 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.492 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.493 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "localpool", 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: "failure_domain": { 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "failure_domain", 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.494 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "host", 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "failure domain for any created local pool", 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.495 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.496 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.497 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.498 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.499 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "min_size": { 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "min_size", 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.500 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "default min_size for any created local pool", 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "num_rep": { 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "num_rep", 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "3", 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.501 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.502 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "default replica count for any created local pool", 2024-03-11T20:09:24.502 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.502 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.502 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.502 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num": { 2024-03-11T20:09:24.502 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "pg_num", 2024-03-11T20:09:24.502 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.502 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.502 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.502 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "128", 2024-03-11T20:09:24.502 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.502 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "default pg_num for any created local pool", 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "prefix": { 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "prefix", 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.503 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "name prefix for any created local pool", 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "subtree": { 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "subtree", 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "rack", 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-11T20:09:24.504 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mds_autoscaler", 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.505 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.506 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.507 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.508 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.508 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.508 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.508 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.508 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.508 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.508 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.508 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.509 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.510 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mirroring", 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.511 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.512 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.513 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.515 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.515 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.515 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.515 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.515 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.515 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.515 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.515 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.515 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.515 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.515 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.516 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.516 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.516 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.516 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.516 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.516 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.516 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.516 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.516 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "nfs", 2024-03-11T20:09:24.516 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.516 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.517 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.518 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.519 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.520 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.522 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "orchestrator", 2024-03-11T20:09:24.522 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.522 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.522 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.522 INFO:teuthology.orchestra.run.smithi097.stdout: "fail_fs": { 2024-03-11T20:09:24.522 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "fail_fs", 2024-03-11T20:09:24.522 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.522 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.522 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.522 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.522 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.522 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.523 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.524 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.525 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.526 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.527 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.528 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.528 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.528 INFO:teuthology.orchestra.run.smithi097.stdout: "orchestrator": { 2024-03-11T20:09:24.528 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "orchestrator", 2024-03-11T20:09:24.528 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.528 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.528 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.528 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.528 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.528 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.528 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.528 INFO:teuthology.orchestra.run.smithi097.stdout: "cephadm", 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: "rook", 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: "test_orchestrator" 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Orchestrator backend", 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "osd_perf_query", 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.529 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.530 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.531 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.532 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.533 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "osd_support", 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.535 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.536 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.537 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.539 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "pg_autoscaler", 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.540 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.541 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.542 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.543 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.544 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.545 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: "threshold": { 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "threshold", 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "3.0", 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1.0", 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "scaling threshold", 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-11T20:09:24.546 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "progress", 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: "allow_pg_recovery_event": { 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "allow_pg_recovery_event", 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.547 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": { 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "enabled", 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.548 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.549 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.550 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.551 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.552 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.553 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: "max_completed_events": { 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_completed_events", 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "50", 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "number of past completed events to remember", 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5", 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how long the module is going to sleep", 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.555 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "prometheus", 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: "cache": { 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cache", 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.556 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: "exclude_perf_counters": { 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "exclude_perf_counters", 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.557 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-11T20:09:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-11T20:09:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.558 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.559 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.560 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.561 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.562 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_stats_pools": { 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rbd_stats_pools", 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.563 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "300", 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "scrape_interval": { 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "scrape_interval", 2024-03-11T20:09:24.564 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "15.0", 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "server_addr": { 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_addr", 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.565 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "::", 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: "server_port": { 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_port", 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.566 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "9283", 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "stale_cache_strategy": { 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "stale_cache_strategy", 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "log", 2024-03-11T20:09:24.567 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "standby_behaviour": { 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "standby_behaviour", 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "default", 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.568 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: "default", 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: "error" 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: "standby_error_status_code": { 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "standby_error_status_code", 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.569 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "500", 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "400", 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "599", 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rbd_support", 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.570 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.571 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.572 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.573 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.573 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.573 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.573 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.573 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.573 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.573 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.573 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.573 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.573 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.573 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.573 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.574 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: "max_concurrent_snap_create": { 2024-03-11T20:09:24.575 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_concurrent_snap_create", 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "mirror_snapshot_schedule": { 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mirror_snapshot_schedule", 2024-03-11T20:09:24.576 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "trash_purge_schedule": { 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "trash_purge_schedule", 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.577 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "restful", 2024-03-11T20:09:24.578 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "enable_auth": { 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "enable_auth", 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: "key_file": { 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "key_file", 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.580 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.581 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.582 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.583 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.584 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "server_addr": { 2024-03-11T20:09:24.585 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_addr", 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: "server_port": { 2024-03-11T20:09:24.586 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_port", 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rgw", 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.588 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.589 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.590 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rook", 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.593 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.594 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.595 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.596 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.597 INFO:teuthology.orchestra.run.smithi097.stdout: "storage_class": { 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "storage_class", 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "local", 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.598 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "selftest", 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.599 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.600 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.601 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.602 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.603 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "roption1": { 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "roption1", 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.604 INFO:teuthology.orchestra.run.smithi097.stdout: "roption2": { 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "roption2", 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "xyz", 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.605 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption1": { 2024-03-11T20:09:24.606 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption1", 2024-03-11T20:09:24.606 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.606 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.606 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.606 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.606 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.606 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.606 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.606 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.606 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.606 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.606 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption2": { 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption2", 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.607 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption3": { 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption3", 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.608 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption4": { 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption4", 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption5": { 2024-03-11T20:09:24.609 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption5", 2024-03-11T20:09:24.610 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.610 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.610 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.610 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.610 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.610 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.610 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.610 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.610 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.610 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.610 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.610 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption6": { 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption6", 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.611 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption7": { 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption7", 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1", 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "42", 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "testkey": { 2024-03-11T20:09:24.612 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "testkey", 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "testlkey": { 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "testlkey", 2024-03-11T20:09:24.613 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: "testnewline": { 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "testnewline", 2024-03-11T20:09:24.614 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "snap_schedule", 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "allow_m_granularity": { 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "allow_m_granularity", 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "allow minute scheduled snapshots", 2024-03-11T20:09:24.616 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "dump_on_update": { 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "dump_on_update", 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "dump database to debug log on update", 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.617 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.618 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.619 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.620 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.622 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "stats", 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.623 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.624 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.625 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.626 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.627 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "status", 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.628 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.629 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.631 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.632 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "telegraf", 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.633 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: "address": { 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "address", 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "15", 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.635 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.636 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.637 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.638 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.638 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.638 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.638 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.638 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.638 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.638 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.638 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.638 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.638 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.638 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.638 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.639 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.640 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "telemetry", 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_basic": { 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_basic", 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-11T20:09:24.641 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_crash": { 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_crash", 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-11T20:09:24.642 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_device": { 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_device", 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-11T20:09:24.643 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_ident": { 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_ident", 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-11T20:09:24.644 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_perf": { 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_perf", 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-11T20:09:24.645 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.646 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.646 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.646 INFO:teuthology.orchestra.run.smithi097.stdout: "contact": { 2024-03-11T20:09:24.646 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "contact", 2024-03-11T20:09:24.646 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.646 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.646 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.646 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.646 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.646 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.646 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.646 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "description": { 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "description", 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.647 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.648 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.648 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.648 INFO:teuthology.orchestra.run.smithi097.stdout: "device_url": { 2024-03-11T20:09:24.648 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "device_url", 2024-03-11T20:09:24.648 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.648 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.648 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.648 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-11T20:09:24.648 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.648 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.648 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.648 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": { 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "enabled", 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.649 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.650 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.650 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.650 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.650 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:24.650 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:24.650 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.650 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.650 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.650 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "24", 2024-03-11T20:09:24.650 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "8", 2024-03-11T20:09:24.650 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.650 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "last_opt_revision": { 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "last_opt_revision", 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "1", 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.651 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "leaderboard": { 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "leaderboard", 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.652 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: "leaderboard_description": { 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "leaderboard_description", 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.653 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.654 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.656 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.658 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.658 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.658 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.658 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.658 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.658 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.658 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.658 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.658 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.658 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.658 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.658 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.658 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: "organization": { 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "organization", 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "proxy": { 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "proxy", 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.660 INFO:teuthology.orchestra.run.smithi097.stdout: "url": { 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "url", 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.662 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "test_orchestrator", 2024-03-11T20:09:24.662 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.662 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.662 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.662 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.662 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.662 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.662 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.662 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.662 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.662 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.662 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.662 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.663 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.664 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.665 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.666 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "volumes", 2024-03-11T20:09:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.668 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.669 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.670 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.671 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "max_concurrent_clones": { 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_concurrent_clones", 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.672 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "4", 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: "periodic_async_work": { 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "periodic_async_work", 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.673 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.674 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.674 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.674 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.674 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Periodically check for async work", 2024-03-11T20:09:24.674 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.674 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.674 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.674 INFO:teuthology.orchestra.run.smithi097.stdout: "snapshot_clone_delay": { 2024-03-11T20:09:24.674 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "snapshot_clone_delay", 2024-03-11T20:09:24.674 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.674 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.674 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "zabbix", 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.675 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.676 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.676 INFO:teuthology.orchestra.run.smithi097.stdout: "discovery_interval": { 2024-03-11T20:09:24.676 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "discovery_interval", 2024-03-11T20:09:24.678 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:24.678 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.678 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.678 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "100", 2024-03-11T20:09:24.678 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.678 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.678 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.678 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.678 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "identifier": { 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "identifier", 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.679 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.680 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.680 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:24.680 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:24.680 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.680 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.680 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.680 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:24.680 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.680 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.680 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.681 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.682 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.683 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.684 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.684 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.684 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.684 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.684 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.684 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.684 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.684 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.684 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.684 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.684 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.684 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.684 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.685 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.685 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.685 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.685 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.685 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.685 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.685 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.685 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.686 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.686 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.686 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.686 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.686 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.686 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.686 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.686 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.686 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.686 INFO:teuthology.orchestra.run.smithi097.stdout: "zabbix_host": { 2024-03-11T20:09:24.686 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "zabbix_host", 2024-03-11T20:09:24.686 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: "zabbix_port": { 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "zabbix_port", 2024-03-11T20:09:24.687 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10051", 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "zabbix_sender": { 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "zabbix_sender", 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.688 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.689 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: "modules": [ 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: "cephadm", 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: "dashboard", 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: "iostat", 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: "nfs", 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: "restful", 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: "snap_schedule" 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: "available_modules": [ 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "alerts", 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.690 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "How frequently to reexamine health status", 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.692 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.693 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.694 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.695 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_destination": { 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_destination", 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.696 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.697 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.697 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.697 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.697 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.697 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Email address to send alerts to", 2024-03-11T20:09:24.697 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.697 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.697 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.697 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_from_name": { 2024-03-11T20:09:24.697 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_from_name", 2024-03-11T20:09:24.697 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.697 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "Ceph", 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Email From: name", 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_host": { 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_host", 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.698 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "SMTP server", 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_password": { 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_password", 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.699 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Password to authenticate with", 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_port": { 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_port", 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.700 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.701 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "465", 2024-03-11T20:09:24.701 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.701 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.701 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.701 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "SMTP port", 2024-03-11T20:09:24.701 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.701 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.701 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.701 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_sender": { 2024-03-11T20:09:24.701 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_sender", 2024-03-11T20:09:24.701 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.701 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "SMTP envelope sender", 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_ssl": { 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_ssl", 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.702 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "smtp_user": { 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "smtp_user", 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.703 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "User to authenticate as", 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "balancer", 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.704 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "active": { 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "active", 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "automatically balance PGs across cluster", 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.705 INFO:teuthology.orchestra.run.smithi097.stdout: "begin_time": { 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "begin_time", 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0000", 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "beginning time of day to automatically balance", 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "begin_weekday": { 2024-03-11T20:09:24.706 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "begin_weekday", 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "0", 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "6", 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_compat_max_iterations": { 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crush_compat_max_iterations", 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:24.707 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "25", 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1", 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "250", 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_compat_metrics": { 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crush_compat_metrics", 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.708 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "pgs,objects,bytes", 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "crush_compat_step": { 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crush_compat_step", 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.709 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0.5", 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "0.001", 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "0.999", 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "aggressiveness of optimization", 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "end_time": { 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "end_time", 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "2359", 2024-03-11T20:09:24.710 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "ending time of day to automatically balance", 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "end_weekday": { 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "end_weekday", 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "0", 2024-03-11T20:09:24.711 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "6", 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.712 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.713 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.714 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.715 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: "min_score": { 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "min_score", 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "mode": { 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mode", 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "upmap", 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "crush-compat", 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "none", 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "read", 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "upmap", 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: "upmap-read" 2024-03-11T20:09:24.718 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Balancer mode", 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_ids": { 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "pool_ids", 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.719 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-11T20:09:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "upmap_max_deviation": { 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "upmap_max_deviation", 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5", 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1", 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: "upmap_max_optimizations": { 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "upmap_max_optimizations", 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.722 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cephadm", 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: "agent_down_multiplier": { 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "agent_down_multiplier", 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "3.0", 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.723 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "agent_refresh_rate": { 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "agent_refresh_rate", 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "20", 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.724 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "agent_starting_port": { 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "agent_starting_port", 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "4721", 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.725 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "allow_ptrace": { 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "allow_ptrace", 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-11T20:09:24.726 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "autotune_interval": { 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "autotune_interval", 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.727 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "autotune_memory_target_ratio": { 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "autotune_memory_target_ratio", 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0.7", 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.728 INFO:teuthology.orchestra.run.smithi097.stdout: "cephadm_log_destination": { 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cephadm_log_destination", 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "file", 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "file,syslog", 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "syslog" 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.729 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.730 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.730 INFO:teuthology.orchestra.run.smithi097.stdout: "cgroups_split": { 2024-03-11T20:09:24.730 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cgroups_split", 2024-03-11T20:09:24.730 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.730 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.730 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.730 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.730 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.730 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.730 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.730 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-11T20:09:24.730 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: "config_checks_enabled": { 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "config_checks_enabled", 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-11T20:09:24.731 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: "config_dashboard": { 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "config_dashboard", 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-11T20:09:24.732 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_alertmanager": { 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_alertmanager", 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Prometheus container image", 2024-03-11T20:09:24.733 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.734 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.734 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.734 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_base": { 2024-03-11T20:09:24.734 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_base", 2024-03-11T20:09:24.734 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.734 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.734 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.734 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-11T20:09:24.734 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.734 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.734 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Container image name, without the tag", 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_elasticsearch": { 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_elasticsearch", 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.735 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "elasticsearch container image", 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_grafana": { 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_grafana", 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Prometheus container image", 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.736 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_haproxy": { 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_haproxy", 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "HAproxy container image", 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.737 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_jaeger_agent": { 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_jaeger_agent", 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Jaeger agent container image", 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_jaeger_collector": { 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_jaeger_collector", 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Jaeger collector container image", 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.739 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_jaeger_query": { 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_jaeger_query", 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Jaeger query container image", 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_keepalived": { 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_keepalived", 2024-03-11T20:09:24.740 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Keepalived container image", 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_loki": { 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_loki", 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Loki container image", 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_node_exporter": { 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_node_exporter", 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.742 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Prometheus container image", 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_nvmeof": { 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_nvmeof", 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.743 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-11T20:09:24.744 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.744 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.744 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.744 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Nvme-of container image", 2024-03-11T20:09:24.744 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.744 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.744 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.744 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_prometheus": { 2024-03-11T20:09:24.744 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_prometheus", 2024-03-11T20:09:24.744 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.744 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.744 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Prometheus container image", 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_promtail": { 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_promtail", 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.745 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Promtail container image", 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: "container_image_snmp_gateway": { 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_image_snmp_gateway", 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.746 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "SNMP Gateway container image", 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "container_init": { 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "container_init", 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Run podman/docker with `--init`", 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: "daemon_cache_timeout": { 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "daemon_cache_timeout", 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:24.748 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "default_cephadm_command_timeout": { 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "default_cephadm_command_timeout", 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "900", 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.749 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: "default_registry": { 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "default_registry", 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "docker.io", 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.750 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: "device_cache_timeout": { 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "device_cache_timeout", 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "1800", 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.751 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "seconds to cache device inventory", 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "device_enhanced_scan": { 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "device_enhanced_scan", 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.752 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "facts_cache_timeout": { 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "facts_cache_timeout", 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "seconds to cache host facts data", 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "host_check_interval": { 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "host_check_interval", 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to perform a host check", 2024-03-11T20:09:24.754 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.755 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.755 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.755 INFO:teuthology.orchestra.run.smithi097.stdout: "hw_monitoring": { 2024-03-11T20:09:24.755 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "hw_monitoring", 2024-03-11T20:09:24.755 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.755 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.755 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.755 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.755 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.755 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.755 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: "inventory_list_all": { 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "inventory_list_all", 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.756 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.757 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-11T20:09:24.757 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.757 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.757 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.757 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.757 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.757 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.757 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.758 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.758 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.758 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.758 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.758 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.758 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.758 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.758 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.759 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.759 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.759 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.759 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.759 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.759 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.759 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.759 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.759 INFO:teuthology.orchestra.run.smithi097.stdout: "log_refresh_metadata": { 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_refresh_metadata", 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.760 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.761 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.762 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.763 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.764 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "*", 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.765 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "max_count_per_host": { 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_count_per_host", 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.766 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "max number of daemons per service per host", 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "max_osd_draining_count": { 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_osd_draining_count", 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.767 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "migration_current": { 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "migration_current", 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "internal - do not modify", 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.768 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: "mode": { 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mode", 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "root", 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: "cephadm-package", 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: "root" 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "mode for remote execution of cephadm", 2024-03-11T20:09:24.769 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "oob_default_addr": { 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "oob_default_addr", 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "169.254.1.1", 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.770 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: "prometheus_alerts_path": { 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "prometheus_alerts_path", 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.771 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "registry_insecure": { 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "registry_insecure", 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.772 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: "registry_password": { 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "registry_password", 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: "registry_url": { 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "registry_url", 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "registry_username": { 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "registry_username", 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.775 INFO:teuthology.orchestra.run.smithi097.stdout: "secure_monitoring_stack": { 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "secure_monitoring_stack", 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "service_discovery_port": { 2024-03-11T20:09:24.776 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "service_discovery_port", 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8765", 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "cephadm service discovery port", 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "ssh_config_file": { 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ssh_config_file", 2024-03-11T20:09:24.777 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: "use_agent": { 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "use_agent", 2024-03-11T20:09:24.778 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "use_repo_digest": { 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "use_repo_digest", 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.779 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_on_failed_host_check": { 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_on_failed_host_check", 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.780 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "raise a health warning if the host check fails", 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_on_stray_daemons": { 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_on_stray_daemons", 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.781 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_on_stray_hosts": { 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_on_stray_hosts", 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.782 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crash", 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.783 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.784 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.785 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.786 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.787 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "retain_interval": { 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "retain_interval", 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "31536000", 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.788 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_recent_interval": { 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_recent_interval", 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "1209600", 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.789 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "dashboard", 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-11T20:09:24.790 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "ALERTMANAGER_API_HOST": { 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-11T20:09:24.791 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-11T20:09:24.792 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "AUDIT_API_ENABLED": { 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "AUDIT_API_ENABLED", 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.793 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.794 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: "ENABLE_BROWSABLE_API": { 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.795 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.796 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.797 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.798 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.799 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.800 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.801 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.802 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.803 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_API_PASSWORD": { 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "admin", 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.804 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.805 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_API_URL": { 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_API_URL", 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_API_USERNAME": { 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "admin", 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.808 INFO:teuthology.orchestra.run.smithi097.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-11T20:09:24.809 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.810 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.810 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.810 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.810 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.810 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.810 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.810 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.810 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.810 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.810 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.810 INFO:teuthology.orchestra.run.smithi097.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-11T20:09:24.810 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "PROMETHEUS_API_HOST": { 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.811 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.812 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.813 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.814 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.815 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.816 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.817 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.818 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.819 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_ENABLED": { 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.820 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.821 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.822 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.822 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.822 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.822 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-11T20:09:24.822 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-11T20:09:24.822 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.822 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.822 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.822 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.822 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.822 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.822 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.822 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8", 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.823 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "45", 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.824 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: "RGW_API_ACCESS_KEY": { 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.825 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.826 INFO:teuthology.orchestra.run.smithi097.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-11T20:09:24.826 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-11T20:09:24.826 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.826 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.826 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.826 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "admin", 2024-03-11T20:09:24.826 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.826 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.826 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.826 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.826 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.826 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "RGW_API_SECRET_KEY": { 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.827 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: "RGW_API_SSL_VERIFY": { 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.829 INFO:teuthology.orchestra.run.smithi097.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-11T20:09:24.830 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-11T20:09:24.830 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.830 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.830 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.830 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.830 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.830 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.830 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.830 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.830 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.830 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.830 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5", 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "cross_origin_url": { 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cross_origin_url", 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.832 INFO:teuthology.orchestra.run.smithi097.stdout: "crt_file": { 2024-03-11T20:09:24.833 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "crt_file", 2024-03-11T20:09:24.833 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.833 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.833 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.833 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.833 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.833 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.833 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.833 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.833 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.833 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.833 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: "debug": { 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "debug", 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Enable/disable debug options", 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.834 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "jwt_token_ttl": { 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "jwt_token_ttl", 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "28800", 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "key_file": { 2024-03-11T20:09:24.835 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "key_file", 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.836 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.837 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.838 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.839 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.840 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "motd": { 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "motd", 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.841 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "The message of the day", 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "redirect_resolve_ip_addr": { 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "redirect_resolve_ip_addr", 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.842 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "server_addr": { 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_addr", 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "::", 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.843 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "server_port": { 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_port", 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8080", 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.844 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "ssl": { 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ssl", 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.845 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: "ssl_server_port": { 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ssl_server_port", 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8443", 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: "standby_behaviour": { 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "standby_behaviour", 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "redirect", 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: "redirect" 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.847 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "standby_error_status_code": { 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "standby_error_status_code", 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "500", 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "400", 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "599", 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.848 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: "url_prefix": { 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "url_prefix", 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.849 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.850 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.850 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.850 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.850 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "devicehealth", 2024-03-11T20:09:24.850 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.850 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.850 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.850 INFO:teuthology.orchestra.run.smithi097.stdout: "enable_monitoring": { 2024-03-11T20:09:24.850 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "enable_monitoring", 2024-03-11T20:09:24.850 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.850 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.850 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.850 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "monitor device health metrics", 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.851 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.852 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.853 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.853 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.853 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.853 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.853 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.853 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.853 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.853 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.853 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.853 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.853 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.853 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.854 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.855 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "mark_out_threshold": { 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mark_out_threshold", 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "2419200", 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.856 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: "pool_name": { 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "pool_name", 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "device_health_metrics", 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.857 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.858 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-11T20:09:24.858 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.858 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.858 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.858 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.858 INFO:teuthology.orchestra.run.smithi097.stdout: "retention_period": { 2024-03-11T20:09:24.858 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "retention_period", 2024-03-11T20:09:24.858 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.858 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.858 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.858 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "15552000", 2024-03-11T20:09:24.858 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.858 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how long to retain device health metrics", 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "scrape_frequency": { 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "scrape_frequency", 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "86400", 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.859 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "self_heal": { 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "self_heal", 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-11T20:09:24.860 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how frequently to wake up and check device health", 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.861 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: "warn_threshold": { 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "warn_threshold", 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "7257600", 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.862 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "diskprediction_local", 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.863 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.864 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.865 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.866 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.867 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "predict_interval": { 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "predict_interval", 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "86400", 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.868 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "predictor_model": { 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "predictor_model", 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "prophetstor", 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.869 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "600", 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.870 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "influx", 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": false, 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "influxdb python module not found", 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: "batch_size": { 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "batch_size", 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5000", 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.871 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "database": { 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "database", 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "ceph", 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.872 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "hostname": { 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "hostname", 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "InfluxDB server hostname", 2024-03-11T20:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "30", 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "5", 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.874 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.875 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.876 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.877 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.878 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.878 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.878 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.878 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.878 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.878 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.878 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.878 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.878 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.878 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.878 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.879 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: "password": { 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "password", 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "password of InfluxDB server user", 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "port": { 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "port", 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "8086", 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "InfluxDB server port", 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "ssl": { 2024-03-11T20:09:24.881 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ssl", 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "false", 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: "threads": { 2024-03-11T20:09:24.882 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "threads", 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5", 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1", 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "32", 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: "username": { 2024-03-11T20:09:24.883 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "username", 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "username of InfluxDB server user", 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "verify_ssl": { 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "verify_ssl", 2024-03-11T20:09:24.884 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "true", 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.885 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "insights", 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.886 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.887 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.888 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.889 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.890 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "iostat", 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.891 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.892 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.893 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.894 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.894 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.894 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.894 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.894 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.894 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.894 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.894 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.894 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.894 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.894 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.894 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.895 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.896 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.896 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.896 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.896 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.896 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.896 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.896 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.896 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.896 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.896 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "k8sevents", 2024-03-11T20:09:24.896 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.896 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.896 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_event_retention_days": { 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "ceph_event_retention_days", 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "7", 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.897 INFO:teuthology.orchestra.run.smithi097.stdout: "config_check_secs": { 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "config_check_secs", 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "10", 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.898 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.899 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.900 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.901 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.902 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "localpool", 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.903 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "failure_domain": { 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "failure_domain", 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "host", 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "failure domain for any created local pool", 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.904 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.905 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.907 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.908 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "min_size": { 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "min_size", 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.909 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "default min_size for any created local pool", 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: "num_rep": { 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "num_rep", 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "3", 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.910 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "default replica count for any created local pool", 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_num": { 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "pg_num", 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "128", 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.911 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "default pg_num for any created local pool", 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "prefix": { 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "prefix", 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.912 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "name prefix for any created local pool", 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: "subtree": { 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "subtree", 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "rack", 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.913 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mds_autoscaler", 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.914 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.915 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.916 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.916 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.916 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.916 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.916 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.916 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.916 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.916 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.916 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.916 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.916 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.916 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.916 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.917 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.918 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.919 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mirroring", 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.920 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.921 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.922 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.923 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "nfs", 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.925 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.926 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.926 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.926 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.926 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.926 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.926 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.926 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.926 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.926 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.926 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.926 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.926 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.927 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.927 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.927 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.927 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.927 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.927 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.927 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.927 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.927 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.927 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.927 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.927 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.927 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.929 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.930 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.930 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.930 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.930 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.930 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.930 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.930 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.930 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.930 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.930 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "orchestrator", 2024-03-11T20:09:24.930 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "fail_fs": { 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "fail_fs", 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.931 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.932 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.933 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.934 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "orchestrator": { 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "orchestrator", 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "cephadm", 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "rook", 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "test_orchestrator" 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Orchestrator backend", 2024-03-11T20:09:24.937 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.938 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.938 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.938 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.938 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.938 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.938 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "osd_perf_query", 2024-03-11T20:09:24.938 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.938 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.938 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.938 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.938 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.938 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.939 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.939 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.939 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.939 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.939 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.939 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.939 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.939 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.939 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.939 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.939 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.939 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.939 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.940 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.941 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.942 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.943 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.943 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.943 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.943 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.943 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.943 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.943 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.943 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.943 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.943 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.943 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.943 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.944 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "osd_support", 2024-03-11T20:09:24.944 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.944 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.944 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.944 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.944 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.944 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.944 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.944 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.944 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.944 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.944 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.945 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.945 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.945 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.945 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.945 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.945 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.945 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.945 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.945 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.945 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.945 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.945 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.945 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.946 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.947 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.948 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "pg_autoscaler", 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.949 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.950 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.951 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.951 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.951 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.951 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.951 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.951 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.951 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.951 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.951 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.952 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.952 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.952 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.952 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.952 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.952 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.952 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.952 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.953 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.953 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.953 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.953 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.953 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.953 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.953 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.953 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.954 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.954 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.954 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.954 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.954 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.954 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.954 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.954 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.955 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.955 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.955 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.955 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.955 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.955 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.955 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.955 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.956 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.956 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.956 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.956 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.956 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.956 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.956 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.956 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.956 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.957 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.957 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.957 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.957 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.957 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.957 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:24.957 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.957 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.957 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.957 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.957 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "threshold": { 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "threshold", 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "3.0", 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1.0", 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "scaling threshold", 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-11T20:09:24.958 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "progress", 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: "allow_pg_recovery_event": { 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "allow_pg_recovery_event", 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.959 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": { 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "enabled", 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.960 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.961 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.962 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.963 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.964 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.965 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: "max_completed_events": { 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_completed_events", 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "50", 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "number of past completed events to remember", 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.966 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: "sleep_interval": { 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "sleep_interval", 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "5", 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "how long the module is going to sleep", 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.967 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.968 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.968 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.968 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.968 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.968 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "prometheus", 2024-03-11T20:09:24.968 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.968 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.968 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.968 INFO:teuthology.orchestra.run.smithi097.stdout: "cache": { 2024-03-11T20:09:24.968 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "cache", 2024-03-11T20:09:24.968 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "exclude_perf_counters": { 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "exclude_perf_counters", 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.969 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.970 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.970 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.970 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.970 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.970 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-11T20:09:24.970 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-11T20:09:24.970 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.970 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.970 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.970 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.970 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.970 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.970 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.971 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.972 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.973 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.974 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_stats_pools": { 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rbd_stats_pools", 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.975 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "300", 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.976 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "scrape_interval": { 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "scrape_interval", 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "15.0", 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.977 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.978 INFO:teuthology.orchestra.run.smithi097.stdout: "server_addr": { 2024-03-11T20:09:24.978 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_addr", 2024-03-11T20:09:24.978 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.978 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.978 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.978 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "::", 2024-03-11T20:09:24.978 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.978 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.978 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.978 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-11T20:09:24.978 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.978 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: "server_port": { 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_port", 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "9283", 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.979 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "stale_cache_strategy": { 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "stale_cache_strategy", 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "log", 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.980 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: "standby_behaviour": { 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "standby_behaviour", 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "default", 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: "default", 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: "error" 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.981 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: "standby_error_status_code": { 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "standby_error_status_code", 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "500", 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "400", 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "599", 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.982 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rbd_support", 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.983 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.985 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.986 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.987 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "max_concurrent_snap_create": { 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_concurrent_snap_create", 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10", 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.988 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "mirror_snapshot_schedule": { 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "mirror_snapshot_schedule", 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.989 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "trash_purge_schedule": { 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "trash_purge_schedule", 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.990 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "restful", 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: "enable_auth": { 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "enable_auth", 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.991 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "key_file": { 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "key_file", 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.992 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.993 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.994 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.995 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.996 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "server_addr": { 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_addr", 2024-03-11T20:09:24.997 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: "server_port": { 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "server_port", 2024-03-11T20:09:24.998 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:24.999 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rgw", 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.000 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.001 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.002 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.002 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.002 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.002 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.002 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.002 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.002 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.002 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.002 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:25.002 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:25.002 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.002 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.003 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.004 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.004 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:25.004 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:25.004 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.004 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.004 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.004 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.004 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.004 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.004 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.004 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.004 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rook", 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.005 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.006 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.006 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.006 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.006 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.006 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.006 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.006 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.006 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.006 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.006 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.006 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.006 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.006 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.008 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.009 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "storage_class": { 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "storage_class", 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.010 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "local", 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "selftest", 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:25.011 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:25.012 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:25.012 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.012 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.012 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.012 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.012 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.012 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.012 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.012 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.012 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.012 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.012 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.012 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.013 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.014 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:25.015 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "roption1": { 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "roption1", 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.016 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "roption2": { 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "roption2", 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.017 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "xyz", 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption1": { 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption1", 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.018 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption2": { 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption2", 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.019 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption3": { 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption3", 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "float", 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.020 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption4": { 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption4", 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.021 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption5": { 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption5", 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.022 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption6": { 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption6", 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.023 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "rwoption7": { 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rwoption7", 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "1", 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "42", 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.024 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "testkey": { 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "testkey", 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.025 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "testlkey": { 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "testlkey", 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "testnewline": { 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "testnewline", 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.027 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "snap_schedule", 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "allow_m_granularity": { 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "allow_m_granularity", 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.028 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "allow minute scheduled snapshots", 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "dump_on_update": { 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "dump_on_update", 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.029 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "dump database to debug log on update", 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.030 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.031 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.033 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.034 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "stats", 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.035 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:25.036 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.037 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.038 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.039 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "status", 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.040 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:25.041 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.042 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.043 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.044 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "telegraf", 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "address": { 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "address", 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.045 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "15", 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.046 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.047 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:25.048 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:25.049 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.050 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.051 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "telemetry", 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_basic": { 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_basic", 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.052 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_crash": { 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_crash", 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_device": { 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_device", 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.054 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "True", 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_ident": { 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_ident", 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.055 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "channel_perf": { 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "channel_perf", 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.056 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "contact": { 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "contact", 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.057 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "description": { 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "description", 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.058 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "device_url": { 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "device_url", 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.059 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": { 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "enabled", 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.060 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "24", 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "8", 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.061 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "last_opt_revision": { 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "last_opt_revision", 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "1", 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.062 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "leaderboard": { 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "leaderboard", 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.063 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "leaderboard_description": { 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "leaderboard_description", 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.065 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.066 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.067 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.068 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "organization": { 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "organization", 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.069 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "proxy": { 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "proxy", 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.070 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "url": { 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "url", 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-11T20:09:25.071 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "test_orchestrator", 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:25.072 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.073 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.074 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.075 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.076 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "volumes", 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:25.077 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.078 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.079 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.080 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:25.081 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.082 INFO:teuthology.orchestra.run.smithi097.stdout: "max_concurrent_clones": { 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "max_concurrent_clones", 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "4", 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "periodic_async_work": { 2024-03-11T20:09:25.083 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "periodic_async_work", 2024-03-11T20:09:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Periodically check for async work", 2024-03-11T20:09:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.084 INFO:teuthology.orchestra.run.smithi097.stdout: "snapshot_clone_delay": { 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "snapshot_clone_delay", 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "0", 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "zabbix", 2024-03-11T20:09:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "can_run": true, 2024-03-11T20:09:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "error_string": "", 2024-03-11T20:09:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "module_options": { 2024-03-11T20:09:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "discovery_interval": { 2024-03-11T20:09:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "discovery_interval", 2024-03-11T20:09:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "uint", 2024-03-11T20:09:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "100", 2024-03-11T20:09:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.086 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "identifier": { 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "identifier", 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.087 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "interval": { 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "interval", 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "secs", 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "60", 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.088 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "log_level": { 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_level", 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.089 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster": { 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster", 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.090 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_cluster_level": { 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_cluster_level", 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.091 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "info", 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [ 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "", 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "critical", 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "debug", 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "error", 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "info", 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "warning" 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.092 INFO:teuthology.orchestra.run.smithi097.stdout: "log_to_file": { 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "log_to_file", 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "bool", 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 1, 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "False", 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "zabbix_host": { 2024-03-11T20:09:25.093 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "zabbix_host", 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "", 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "zabbix_port": { 2024-03-11T20:09:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "zabbix_port", 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "int", 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "10051", 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "zabbix_sender": { 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "zabbix_sender", 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "str", 2024-03-11T20:09:25.095 INFO:teuthology.orchestra.run.smithi097.stdout: "level": "advanced", 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "flags": 0, 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "min": "", 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "max": "", 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "enum_allowed": [], 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "desc": "", 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "long_desc": "", 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "see_also": [] 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "services": { 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: "dashboard": "https://172.21.15.97:8443/" 2024-03-11T20:09:25.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "always_on_modules": { 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "octopus": [ 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "balancer", 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "crash", 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "devicehealth", 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "orchestrator", 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_autoscaler", 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "progress", 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_support", 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "status", 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "telemetry", 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "volumes" 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "pacific": [ 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "balancer", 2024-03-11T20:09:25.097 INFO:teuthology.orchestra.run.smithi097.stdout: "crash", 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "devicehealth", 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "orchestrator", 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_autoscaler", 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "progress", 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_support", 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "status", 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "telemetry", 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "volumes" 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "quincy": [ 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "balancer", 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "crash", 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "devicehealth", 2024-03-11T20:09:25.098 INFO:teuthology.orchestra.run.smithi097.stdout: "orchestrator", 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_autoscaler", 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "progress", 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_support", 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "status", 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "telemetry", 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "volumes" 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "reef": [ 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "balancer", 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "crash", 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "devicehealth", 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "orchestrator", 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_autoscaler", 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "progress", 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_support", 2024-03-11T20:09:25.099 INFO:teuthology.orchestra.run.smithi097.stdout: "status", 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "telemetry", 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "volumes" 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "squid": [ 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "balancer", 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "crash", 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "devicehealth", 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "orchestrator", 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "pg_autoscaler", 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "progress", 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "rbd_support", 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "status", 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "telemetry", 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "volumes" 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.100 INFO:teuthology.orchestra.run.smithi097.stdout: "last_failure_osd_epoch": 94, 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "active_clients": [ 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "devicehealth", 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:0", 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 2703876139 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "libcephsqlite", 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.101 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:0", 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 1504062275 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "rbd_support", 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:0", 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3514117918 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T20:09:25.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "snap_schedule", 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:0", 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 3121428943 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "snap_schedule", 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-03-11T20:09:25.103 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:0", 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 423483387 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "volumes", 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "addrvec": [ 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "type": "v2", 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "addr": "172.21.15.97:0", 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: "nonce": 2900475693 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T20:09:25.104 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T20:09:25.106 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:24 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2888796755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T20:09:25.106 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:24 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2664978075' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-11T20:09:25.106 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:24 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2888796755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T20:09:25.106 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:24 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2664978075' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-11T20:09:25.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.781+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2664978075 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe5dc0a1ba0 msgr2=0x7fe5dc0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:25.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.781+0000 7fe5fdc00640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe5dc0a1ba0 0x7fe5dc0a4060 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800bca0 tx=0x7fe5e800b410 comp rx=0 tx=0).stop 2024-03-11T20:09:25.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.781+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2664978075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f8108650 msgr2=0x7fe5f819fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:09:25.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.781+0000 7fe5fdc00640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f8108650 0x7fe5f819fb80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe5e400e9c0 tx=0x7fe5e400ee80 comp rx=0 tx=0).stop 2024-03-11T20:09:25.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.781+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2664978075 shutdown_connections 2024-03-11T20:09:25.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.781+0000 7fe5fdc00640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe5dc0a1ba0 0x7fe5dc0a4060 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:25.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.781+0000 7fe5fdc00640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe5f8199800 0x7fe5f8199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:25.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.781+0000 7fe5fdc00640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5f8108650 0x7fe5f819fb80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:25.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.781+0000 7fe5fdc00640 1 --2- 172.21.15.97:0/2664978075 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe5f8103580 0x7fe5f819f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:09:25.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.781+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2664978075 >> 172.21.15.97:0/2664978075 conn(0x7fe5f80fee40 msgr2=0x7fe5f8101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:09:25.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.781+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2664978075 shutdown_connections 2024-03-11T20:09:25.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:09:23.781+0000 7fe5fdc00640 1 -- 172.21.15.97:0/2664978075 wait complete. 2024-03-11T20:09:25.113 DEBUG:tasks.check_counter:Getting stats from a 2024-03-11T20:09:25.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-11T20:09:25.114 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:24 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2888796755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T20:09:25.114 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:24 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2664978075' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-11T20:09:25.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:25 smithi097 ceph-mon[31898]: pgmap v315: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 92 B/s wr, 11 op/s 2024-03-11T20:09:25.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:25 smithi114 ceph-mon[36376]: pgmap v315: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 92 B/s wr, 11 op/s 2024-03-11T20:09:25.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:25 smithi161 ceph-mon[36332]: pgmap v315: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 92 B/s wr, 11 op/s 2024-03-11T20:09:25.573 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:09:27.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:27 smithi097 ceph-mon[31898]: pgmap v316: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-11T20:09:27.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:27 smithi114 ceph-mon[36376]: pgmap v316: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-11T20:09:27.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:27 smithi161 ceph-mon[36332]: pgmap v316: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-11T20:09:28.069 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T20:09:28.069 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T20:09:28.069 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 98, 2024-03-11T20:09:28.069 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 256, 2024-03-11T20:09:28.069 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 32931, 2024-03-11T20:09:28.069 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 388966, 2024-03-11T20:09:28.069 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 17, 2024-03-11T20:09:28.070 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 6, 2024-03-11T20:09:28.070 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.081887276, 2024-03-11T20:09:28.070 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.046625474, 2024-03-11T20:09:28.070 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.025222989, 2024-03-11T20:09:28.070 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.001637281, 2024-03-11T20:09:28.070 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:28.070 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 252, 2024-03-11T20:09:28.070 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.064544260, 2024-03-11T20:09:28.070 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000256128 2024-03-11T20:09:28.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.070 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:28.071 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 156, 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000064413, 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000412 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 306080 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 733, 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 880, 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 1769741, 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 510152, 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 9, 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 6, 2024-03-11T20:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.228359640, 2024-03-11T20:09:28.073 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.089692490, 2024-03-11T20:09:28.073 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.147236389, 2024-03-11T20:09:28.073 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.017377443, 2024-03-11T20:09:28.073 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:28.073 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 882, 2024-03-11T20:09:28.073 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.104408203, 2024-03-11T20:09:28.073 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000118376 2024-03-11T20:09:28.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.073 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:28.073 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 1029, 2024-03-11T20:09:28.073 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000519462, 2024-03-11T20:09:28.073 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000504 2024-03-11T20:09:28.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.074 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 1637760, 2024-03-11T20:09:28.074 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 152624 2024-03-11T20:09:28.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.074 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T20:09:28.074 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 71, 2024-03-11T20:09:28.074 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 120, 2024-03-11T20:09:28.074 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 25900, 2024-03-11T20:09:28.074 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 184869, 2024-03-11T20:09:28.074 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 11, 2024-03-11T20:09:28.074 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 6, 2024-03-11T20:09:28.074 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.033376800, 2024-03-11T20:09:28.074 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.013666112, 2024-03-11T20:09:28.074 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.016265663, 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.000521841, 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 117, 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.034077619, 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000291261 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 116, 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000039658, 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000341 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 154304 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.075 INFO:teuthology.orchestra.run.smithi097.stdout: "cct": { 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: "total_workers": 1, 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: "unhealthy_workers": 0 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: "finisher-MDSRank": { 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: "queue_len": 0, 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: "complete_latency": { 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 64, 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.003399229, 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000053112 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: "finisher-PurgeQueue": { 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: "queue_len": 0, 2024-03-11T20:09:28.076 INFO:teuthology.orchestra.run.smithi097.stdout: "complete_latency": { 2024-03-11T20:09:28.077 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 2, 2024-03-11T20:09:28.077 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000003663, 2024-03-11T20:09:28.077 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000001831 2024-03-11T20:09:28.077 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:28.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.077 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": { 2024-03-11T20:09:28.077 INFO:teuthology.orchestra.run.smithi097.stdout: "request": 10, 2024-03-11T20:09:28.077 INFO:teuthology.orchestra.run.smithi097.stdout: "reply": 6, 2024-03-11T20:09:28.077 INFO:teuthology.orchestra.run.smithi097.stdout: "reply_latency": { 2024-03-11T20:09:28.077 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 6, 2024-03-11T20:09:28.077 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.004305679, 2024-03-11T20:09:28.077 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000717613 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: "slow_reply": 0, 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: "forward": 4, 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_fetch_complete": 0, 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_fetch_keys": 0, 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_commit": 11, 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_split": 0, 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_merge": 0, 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes": 14, 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_top": 0, 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_bottom": 14, 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_pin_tail": 0, 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_pinned": 14, 2024-03-11T20:09:28.078 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_expired": 0, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_with_caps": 2, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": 2, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "subtrees": 2, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse": 15, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_hit": 11, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_forward": 1, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_discover": 1, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_dir_fetch": 0, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_remote_ino": 0, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_lock": 0, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "load_cent": 3000, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "q": 3, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "exported": 0, 2024-03-11T20:09:28.079 INFO:teuthology.orchestra.run.smithi097.stdout: "exported_inodes": 0, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "imported": 0, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "imported_inodes": 0, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_dir_fetch": 0, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_backtrace_fetch": 0, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_peer_discover": 0, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rfiles": 4, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rbytes": 234, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rsnaps": 2, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_set_tag": 0, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_inotable_repaired": 0, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dir_inodes": 0, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T20:09:28.080 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_file_inodes": 0, 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_inode_file_caps": 0, 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_revoke": 2, 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_grant": 3, 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_caps": 2, 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_caps_dirty": 0, 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_cap_release": 0, 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "process_request_cap_release": 1 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache": { 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_update": 0, 2024-03-11T20:09:28.081 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_update_receipt": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_try_discover": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_send_discover": 2, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_handle_discover": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays_delayed": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays_enqueuing": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_created": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_enqueued": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_reintegrated": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_migrated": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_processing": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_enqueued": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_prioritized": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_started": 0, 2024-03-11T20:09:28.082 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_completed": 0, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_exportdir": 0, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_flush": 0, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_fragmentdir": 0, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_fragstats": 0, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_inodestats": 0 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log": { 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "evlrg": 0, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "evadd": 10, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "evex": 0, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "evtrm": 0, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "ev": 10, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "evexg": 0, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "evexd": 0, 2024-03-11T20:09:28.083 INFO:teuthology.orchestra.run.smithi097.stdout: "segadd": 2, 2024-03-11T20:09:28.084 INFO:teuthology.orchestra.run.smithi097.stdout: "segex": 0, 2024-03-11T20:09:28.084 INFO:teuthology.orchestra.run.smithi097.stdout: "segtrm": 0, 2024-03-11T20:09:28.084 INFO:teuthology.orchestra.run.smithi097.stdout: "seg": 2, 2024-03-11T20:09:28.084 INFO:teuthology.orchestra.run.smithi097.stdout: "segmjr": 2, 2024-03-11T20:09:28.084 INFO:teuthology.orchestra.run.smithi097.stdout: "segexg": 0, 2024-03-11T20:09:28.084 INFO:teuthology.orchestra.run.smithi097.stdout: "segexd": 0, 2024-03-11T20:09:28.084 INFO:teuthology.orchestra.run.smithi097.stdout: "expos": 4194304, 2024-03-11T20:09:28.084 INFO:teuthology.orchestra.run.smithi097.stdout: "wrpos": 4197189, 2024-03-11T20:09:28.084 INFO:teuthology.orchestra.run.smithi097.stdout: "rdpos": 0, 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "jlat": { 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 9, 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.018572103, 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.002063567 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "replayed": 0 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_mem": { 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ino": 17, 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ino+": 17, 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "ino-": 0, 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "dir": 15, 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "dir+": 16, 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "dir-": 1, 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "dn": 14, 2024-03-11T20:09:28.085 INFO:teuthology.orchestra.run.smithi097.stdout: "dn+": 14, 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: "dn-": 0, 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: "cap": 2, 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: "cap+": 4, 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: "cap-": 2, 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: "rss": 50944, 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: "heap": 207228 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_server": { 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: "dispatch_client_request": 14, 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: "dispatch_server_request": 0, 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_request": 10, 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_session": 29, 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_peer_request": 1, 2024-03-11T20:09:28.086 INFO:teuthology.orchestra.run.smithi097.stdout: "req_create_latency": { 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getattr_latency": { 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 5, 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.003933407, 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000786681 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getfilelock_latency": { 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: "req_link_latency": { 2024-03-11T20:09:28.087 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookup_latency": { 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 1, 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000372272, 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000372272 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookuphash_latency": { 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.088 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupino_latency": { 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupname_latency": { 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupparent_latency": { 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupsnap_latency": { 2024-03-11T20:09:28.089 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.090 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.090 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.090 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.090 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lssnap_latency": { 2024-03-11T20:09:28.090 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.090 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.090 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.090 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.090 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mkdir_latency": { 2024-03-11T20:09:28.090 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.090 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.091 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.091 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mknod_latency": { 2024-03-11T20:09:28.091 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.091 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.091 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.091 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mksnap_latency": { 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: "req_open_latency": { 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: "req_readdir_latency": { 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.092 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rename_latency": { 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: "req_renamesnap_latency": { 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: "req_snapdiff_latency": { 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmdir_latency": { 2024-03-11T20:09:28.093 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmsnap_latency": { 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmxattr_latency": { 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setattr_latency": { 2024-03-11T20:09:28.094 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setdirlayout_latency": { 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setfilelock_latency": { 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setlayout_latency": { 2024-03-11T20:09:28.095 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setxattr_latency": { 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: "req_symlink_latency": { 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: "req_unlink_latency": { 2024-03-11T20:09:28.096 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: "cap_revoke_eviction": 0, 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: "cap_acquisition_throttle": 0, 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getvxattr_latency": { 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_sessions": { 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: "session_count": 2, 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: "session_add": 5, 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: "session_remove": 3, 2024-03-11T20:09:28.097 INFO:teuthology.orchestra.run.smithi097.stdout: "sessions_open": 2, 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: "sessions_stale": 0, 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: "total_load": 4, 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: "average_load": 1, 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: "avg_session_uptime": 593, 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: "mdthresh_evicted": 0 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: "mempool": { 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_bytes": 0, 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_items": 0, 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_items": 0, 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_items": 0, 2024-03-11T20:09:28.098 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T20:09:28.099 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T20:09:28.155 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T20:09:28.155 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T20:09:28.155 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T20:09:28.155 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_items": 0, 2024-03-11T20:09:28.155 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T20:09:28.155 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T20:09:28.155 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_bytes": 0, 2024-03-11T20:09:28.155 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_items": 0, 2024-03-11T20:09:28.155 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_bytes": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_items": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_items": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_bytes": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_items": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_bytes": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_items": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_bytes": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_items": 0, 2024-03-11T20:09:28.156 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T20:09:28.157 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_items": 0, 2024-03-11T20:09:28.157 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T20:09:28.157 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_items": 0, 2024-03-11T20:09:28.157 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_bytes": 273409, 2024-03-11T20:09:28.157 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_items": 105, 2024-03-11T20:09:28.157 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_bytes": 88, 2024-03-11T20:09:28.157 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_items": 1, 2024-03-11T20:09:28.157 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bytes": 0, 2024-03-11T20:09:28.157 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_items": 0, 2024-03-11T20:09:28.157 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_bytes": 0, 2024-03-11T20:09:28.157 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_items": 0, 2024-03-11T20:09:28.157 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_bytes": 0, 2024-03-11T20:09:28.157 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_items": 0, 2024-03-11T20:09:28.158 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_bytes": 9616, 2024-03-11T20:09:28.158 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_items": 185, 2024-03-11T20:09:28.158 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T20:09:28.158 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_items": 0, 2024-03-11T20:09:28.158 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_bytes": 0, 2024-03-11T20:09:28.158 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_items": 0, 2024-03-11T20:09:28.158 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_bytes": 67373, 2024-03-11T20:09:28.158 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_items": 297, 2024-03-11T20:09:28.158 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_bytes": 0, 2024-03-11T20:09:28.158 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_items": 0, 2024-03-11T20:09:28.158 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_bytes": 0, 2024-03-11T20:09:28.158 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_items": 0 2024-03-11T20:09:28.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.159 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter": { 2024-03-11T20:09:28.159 INFO:teuthology.orchestra.run.smithi097.stdout: "op_active": 0, 2024-03-11T20:09:28.159 INFO:teuthology.orchestra.run.smithi097.stdout: "op_laggy": 0, 2024-03-11T20:09:28.159 INFO:teuthology.orchestra.run.smithi097.stdout: "op_send": 51, 2024-03-11T20:09:28.159 INFO:teuthology.orchestra.run.smithi097.stdout: "op_send_bytes": 13956, 2024-03-11T20:09:28.159 INFO:teuthology.orchestra.run.smithi097.stdout: "op_resend": 28, 2024-03-11T20:09:28.159 INFO:teuthology.orchestra.run.smithi097.stdout: "op_reply": 51, 2024-03-11T20:09:28.159 INFO:teuthology.orchestra.run.smithi097.stdout: "op_latency": { 2024-03-11T20:09:28.159 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 51, 2024-03-11T20:09:28.159 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.325994524, 2024-03-11T20:09:28.160 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.006392049 2024-03-11T20:09:28.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.160 INFO:teuthology.orchestra.run.smithi097.stdout: "op_inflight": 0, 2024-03-11T20:09:28.160 INFO:teuthology.orchestra.run.smithi097.stdout: "oplen_avg": { 2024-03-11T20:09:28.160 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 51, 2024-03-11T20:09:28.160 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 77 2024-03-11T20:09:28.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.160 INFO:teuthology.orchestra.run.smithi097.stdout: "op": 51, 2024-03-11T20:09:28.160 INFO:teuthology.orchestra.run.smithi097.stdout: "op_r": 0, 2024-03-11T20:09:28.160 INFO:teuthology.orchestra.run.smithi097.stdout: "op_w": 51, 2024-03-11T20:09:28.160 INFO:teuthology.orchestra.run.smithi097.stdout: "op_rmw": 0, 2024-03-11T20:09:28.160 INFO:teuthology.orchestra.run.smithi097.stdout: "op_pg": 0, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_stat": 0, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_create": 10, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_read": 0, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_write": 9, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_writefull": 9, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_writesame": 0, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_append": 0, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_zero": 0, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_truncate": 0, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_delete": 6, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_mapext": 0, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_sparse_read": 0, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_clonerange": 0, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_getxattr": 0, 2024-03-11T20:09:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_setxattr": 20, 2024-03-11T20:09:28.162 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_cmpxattr": 0, 2024-03-11T20:09:28.162 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_rmxattr": 0, 2024-03-11T20:09:28.162 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_resetxattrs": 0, 2024-03-11T20:09:28.162 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_call": 0, 2024-03-11T20:09:28.162 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_watch": 0, 2024-03-11T20:09:28.162 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_notify": 0, 2024-03-11T20:09:28.162 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T20:09:28.162 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_pgls": 0, 2024-03-11T20:09:28.162 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_pgls_filter": 0, 2024-03-11T20:09:28.162 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_other": 0, 2024-03-11T20:09:28.162 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_active": 0, 2024-03-11T20:09:28.163 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_send": 0, 2024-03-11T20:09:28.163 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_resend": 0, 2024-03-11T20:09:28.163 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_ping": 0, 2024-03-11T20:09:28.163 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_active": 0, 2024-03-11T20:09:28.163 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_send": 0, 2024-03-11T20:09:28.163 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_resend": 0, 2024-03-11T20:09:28.163 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_active": 0, 2024-03-11T20:09:28.163 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_send": 0, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_resend": 0, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_active": 0, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_send": 0, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_resend": 0, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "command_active": 0, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "command_send": 0, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "command_resend": 0, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "map_epoch": 102, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "map_full": 0, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "map_inc": 13, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_sessions": 9, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_session_open": 9, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_session_close": 0, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_laggy": 0, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_wr": 22, 2024-03-11T20:09:28.164 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_rd": 0, 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_del": 1 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: "oft": { 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_objs": 1, 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_kv_pairs": 0, 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_updates": 0, 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_removes": 0 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: "osd-slow-ops": { 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: "slow_ops_count": 0 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: "purge_queue": { 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_ops": 0, 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing": 0, 2024-03-11T20:09:28.165 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_high_water": 0, 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executed_ops": 0, 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executed": 0, 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_item_in_journal": 0 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 104857600, 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 898, 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 1729091, 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 898, 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-03-11T20:09:28.166 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 898, 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 1729091, 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-objecter_bytes": { 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 104857600, 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:28.167 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 51, 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 13956, 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 51, 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 13956, 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-objecter_ops": { 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 1024, 2024-03-11T20:09:28.168 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 51, 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 51, 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 51, 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 51, 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-write_buf_throttle": { 2024-03-11T20:09:28.169 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 3758096384, 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 0, 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 0, 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.170 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-write_buf_throttle-0x5592b12e66e0": { 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 3758096384, 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 10, 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 2885, 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 10, 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 9, 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 2885, 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-03-11T20:09:28.171 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:28.172 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:28.172 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:28.172 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:28.172 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:28.172 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T20:09:29.109 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-11T20:09:29.109 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-11T20:09:29.109 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=2 2024-03-11T20:09:29.109 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-03-11T20:09:29.109 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T20:09:29.109 DEBUG:tasks.check_counter:Getting stats from d 2024-03-11T20:09:29.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-11T20:09:29.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:29 smithi097 ceph-mon[31898]: pgmap v317: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 15 op/s 2024-03-11T20:09:29.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:29 smithi114 ceph-mon[36376]: pgmap v317: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 15 op/s 2024-03-11T20:09:29.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:29 smithi161 ceph-mon[36332]: pgmap v317: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 15 op/s 2024-03-11T20:09:29.540 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:09:31.385 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:31 smithi097 ceph-mon[31898]: pgmap v318: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:09:31.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:31 smithi114 ceph-mon[36376]: pgmap v318: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:09:31.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:31 smithi161 ceph-mon[36332]: pgmap v318: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:09:32.010 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T20:09:32.010 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T20:09:32.010 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 1590, 2024-03-11T20:09:32.010 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 1592, 2024-03-11T20:09:32.010 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 476948, 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 494491, 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 6, 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 2, 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.311721345, 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.113402192, 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.078678955, 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.085330848, 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 1590, 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.123023131, 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000077373 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.011 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 715, 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 886, 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 178378, 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 674327, 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 7, 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 5, 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.191024826, 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.077297618, 2024-03-11T20:09:32.012 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.041695399, 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.051017678, 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 884, 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.091956242, 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000104022 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 453760 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.013 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 263, 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 248, 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 1617868, 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 141319, 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 3, 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 2, 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.105676704, 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.025375093, 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.102017264, 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.010430775, 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 247, 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.027130824, 2024-03-11T20:09:32.014 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000109841 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 1617600, 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 141008 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: "cct": { 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: "total_workers": 1, 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: "unhealthy_workers": 0 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: "finisher-MDSRank": { 2024-03-11T20:09:32.015 INFO:teuthology.orchestra.run.smithi097.stdout: "queue_len": 0, 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: "complete_latency": { 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 3537, 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.198783703, 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000056201 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: "finisher-PurgeQueue": { 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: "queue_len": 0, 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: "complete_latency": { 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.016 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": { 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "request": 0, 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "reply": 0, 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "reply_latency": { 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "slow_reply": 0, 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "forward": 0, 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_fetch_complete": 0, 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_fetch_keys": 0, 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_commit": 0, 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_split": 0, 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_merge": 0, 2024-03-11T20:09:32.017 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes": 0, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_top": 0, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_bottom": 0, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_pin_tail": 0, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_pinned": 0, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_expired": 0, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "inodes_with_caps": 0, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "caps": 0, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "subtrees": 1, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse": 0, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_hit": 0, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_forward": 0, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_discover": 0, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_dir_fetch": 0, 2024-03-11T20:09:32.018 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_remote_ino": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "traverse_lock": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "load_cent": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "q": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "exported": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "exported_inodes": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "imported": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "imported_inodes": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_dir_fetch": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_backtrace_fetch": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "openino_peer_discover": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rfiles": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rbytes": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "root_rsnaps": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T20:09:32.019 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_set_tag": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_inotable_repaired": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dir_inodes": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "scrub_file_inodes": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_inode_file_caps": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_grant": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_caps": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_caps_dirty": 0, 2024-03-11T20:09:32.020 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_cap_release": 0, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "process_request_cap_release": 0 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_cache": { 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_update": 0, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_update_receipt": 0, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_try_discover": 0, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_send_discover": 0, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "dir_handle_discover": 0, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays": 0, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays_delayed": 0, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "num_strays_enqueuing": 0, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_created": 0, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_enqueued": 0, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_reintegrated": 0, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "strays_migrated": 0, 2024-03-11T20:09:32.021 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_processing": 0, 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_enqueued": 0, 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "num_recovering_prioritized": 0, 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_started": 0, 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "recovery_completed": 0, 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_exportdir": 0, 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_flush": 0, 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_fragmentdir": 0, 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_fragstats": 0, 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "ireq_inodestats": 0 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_log": { 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "evlrg": 0, 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "evadd": 0, 2024-03-11T20:09:32.022 INFO:teuthology.orchestra.run.smithi097.stdout: "evex": 0, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "evtrm": 0, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "ev": 6, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "evexg": 0, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "evexd": 0, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "segadd": 0, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "segex": 0, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "segtrm": 0, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "seg": 2, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "segmjr": 2, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "segexg": 0, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "segexd": 0, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "expos": 4194304, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "wrpos": 4196125, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "rdpos": 4195734, 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "jlat": { 2024-03-11T20:09:32.023 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: "replayed": 6 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_mem": { 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: "ino": 3, 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: "ino+": 3, 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: "ino-": 0, 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: "dir": 1, 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: "dir+": 1, 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: "dir-": 0, 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: "dn": 0, 2024-03-11T20:09:32.024 INFO:teuthology.orchestra.run.smithi097.stdout: "dn+": 0, 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "dn-": 0, 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "cap": 0, 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "cap+": 0, 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "cap-": 0, 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "rss": 48512, 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "heap": 215420 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_server": { 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "dispatch_client_request": 0, 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "dispatch_server_request": 0, 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_request": 0, 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_client_session": 0, 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "handle_peer_request": 0, 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "req_create_latency": { 2024-03-11T20:09:32.025 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.026 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.026 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.026 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getattr_latency": { 2024-03-11T20:09:32.026 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.026 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.026 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.026 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getfilelock_latency": { 2024-03-11T20:09:32.026 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.026 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.026 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: "req_link_latency": { 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookup_latency": { 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookuphash_latency": { 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupino_latency": { 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupname_latency": { 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupparent_latency": { 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lookupsnap_latency": { 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: "req_lssnap_latency": { 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mkdir_latency": { 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.030 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mknod_latency": { 2024-03-11T20:09:32.030 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.030 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.030 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: "req_mksnap_latency": { 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: "req_open_latency": { 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: "req_readdir_latency": { 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rename_latency": { 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: "req_renamesnap_latency": { 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: "req_snapdiff_latency": { 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.032 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.033 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmdir_latency": { 2024-03-11T20:09:32.033 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.033 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.033 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.033 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmsnap_latency": { 2024-03-11T20:09:32.033 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.033 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: "req_rmxattr_latency": { 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setattr_latency": { 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setdirlayout_latency": { 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.034 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setfilelock_latency": { 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setlayout_latency": { 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: "req_setxattr_latency": { 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.035 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: "req_symlink_latency": { 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: "req_unlink_latency": { 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: "cap_revoke_eviction": 0, 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: "cap_acquisition_throttle": 0, 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: "req_getvxattr_latency": { 2024-03-11T20:09:32.036 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.037 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.037 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.037 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:32.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.037 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_sessions": { 2024-03-11T20:09:32.037 INFO:teuthology.orchestra.run.smithi097.stdout: "session_count": 2, 2024-03-11T20:09:32.037 INFO:teuthology.orchestra.run.smithi097.stdout: "session_add": 3, 2024-03-11T20:09:32.037 INFO:teuthology.orchestra.run.smithi097.stdout: "session_remove": 1, 2024-03-11T20:09:32.037 INFO:teuthology.orchestra.run.smithi097.stdout: "sessions_open": 2, 2024-03-11T20:09:32.037 INFO:teuthology.orchestra.run.smithi097.stdout: "sessions_stale": 0, 2024-03-11T20:09:32.037 INFO:teuthology.orchestra.run.smithi097.stdout: "total_load": 0, 2024-03-11T20:09:32.037 INFO:teuthology.orchestra.run.smithi097.stdout: "average_load": 0, 2024-03-11T20:09:32.038 INFO:teuthology.orchestra.run.smithi097.stdout: "avg_session_uptime": 2047, 2024-03-11T20:09:32.038 INFO:teuthology.orchestra.run.smithi097.stdout: "mdthresh_evicted": 0 2024-03-11T20:09:32.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.038 INFO:teuthology.orchestra.run.smithi097.stdout: "mempool": { 2024-03-11T20:09:32.038 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_bytes": 0, 2024-03-11T20:09:32.038 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_items": 0, 2024-03-11T20:09:32.038 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T20:09:32.038 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_items": 0, 2024-03-11T20:09:32.038 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T20:09:32.038 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_items": 0, 2024-03-11T20:09:32.039 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T20:09:32.039 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T20:09:32.039 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T20:09:32.039 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T20:09:32.039 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T20:09:32.039 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_items": 0, 2024-03-11T20:09:32.039 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T20:09:32.039 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T20:09:32.039 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_bytes": 0, 2024-03-11T20:09:32.039 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_items": 0, 2024-03-11T20:09:32.039 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_bytes": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_items": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_items": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_bytes": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_items": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_bytes": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_items": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_bytes": 0, 2024-03-11T20:09:32.040 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_items": 0, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_items": 0, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_items": 0, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_bytes": 191192, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_items": 53, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_bytes": 0, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_items": 0, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bytes": 0, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_items": 0, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_bytes": 0, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_items": 0, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_bytes": 0, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_items": 0, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_bytes": 9616, 2024-03-11T20:09:32.041 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_items": 185, 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_items": 0, 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_bytes": 0, 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_items": 0, 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_bytes": 8320, 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_items": 13, 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_bytes": 0, 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_items": 0, 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_bytes": 0, 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_items": 0 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "objecter": { 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "op_active": 0, 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "op_laggy": 0, 2024-03-11T20:09:32.042 INFO:teuthology.orchestra.run.smithi097.stdout: "op_send": 2300, 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "op_send_bytes": 21, 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "op_resend": 3, 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "op_reply": 2300, 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "op_latency": { 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 2300, 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 1.863968310, 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000810421 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "op_inflight": 0, 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "oplen_avg": { 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 2300, 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 2301 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "op": 2300, 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "op_r": 2300, 2024-03-11T20:09:32.043 INFO:teuthology.orchestra.run.smithi097.stdout: "op_w": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "op_rmw": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "op_pg": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_stat": 1498, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_create": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_read": 800, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_write": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_writefull": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_writesame": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_append": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_zero": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_truncate": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_delete": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_mapext": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_sparse_read": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_clonerange": 0, 2024-03-11T20:09:32.044 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_getxattr": 1, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_setxattr": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_cmpxattr": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_rmxattr": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_resetxattrs": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_call": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_watch": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_notify": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_pgls": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_pgls_filter": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "osdop_other": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_active": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_send": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_resend": 0, 2024-03-11T20:09:32.045 INFO:teuthology.orchestra.run.smithi097.stdout: "linger_ping": 0, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_active": 0, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_send": 0, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "poolop_resend": 0, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_active": 0, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_send": 0, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "poolstat_resend": 0, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_active": 0, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_send": 0, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "statfs_resend": 0, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "command_active": 0, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "command_send": 0, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "command_resend": 0, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "map_epoch": 102, 2024-03-11T20:09:32.046 INFO:teuthology.orchestra.run.smithi097.stdout: "map_full": 0, 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "map_inc": 13, 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_sessions": 5, 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_session_open": 5, 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_session_close": 0, 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_laggy": 0, 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_wr": 0, 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_rd": 2, 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_del": 0 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "oft": { 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_objs": 0, 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_kv_pairs": 0, 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_updates": 0, 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "omap_total_removes": 0 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.047 INFO:teuthology.orchestra.run.smithi097.stdout: "osd-slow-ops": { 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: "slow_ops_count": 0 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: "purge_queue": { 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_ops": 0, 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing": 0, 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executing_high_water": 0, 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executed_ops": 0, 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_executed": 0, 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: "pq_item_in_journal": 0 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 104857600, 2024-03-11T20:09:32.048 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 2564, 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 2026038, 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 2564, 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 2564, 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 2026038, 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.049 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-objecter_bytes": { 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 104857600, 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 2300, 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 1826, 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 2300, 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 1826, 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.050 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-objecter_ops": { 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 1024, 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 2300, 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 2300, 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 2300, 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 2300, 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-03-11T20:09:32.051 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-write_buf_throttle": { 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 3758096384, 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 0, 2024-03-11T20:09:32.052 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 0, 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-write_buf_throttle-0x55c0385a6a00": { 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 3758096384, 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 0, 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 0, 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-03-11T20:09:32.053 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-03-11T20:09:32.054 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 0, 2024-03-11T20:09:32.054 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 0, 2024-03-11T20:09:32.054 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-03-11T20:09:32.054 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:32.054 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:32.054 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:32.054 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:32.054 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:32.054 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T20:09:33.100 INFO:tasks.check_counter:Daemon mds.d mds.exported=0 2024-03-11T20:09:33.100 INFO:tasks.check_counter:Daemon mds.d mds.imported=0 2024-03-11T20:09:33.100 INFO:tasks.check_counter:Daemon mds.d mds.root_rsnaps=0 2024-03-11T20:09:33.100 INFO:tasks.check_counter:Daemon mds.d mds_server.req_mksnap_latency.avgcount=0 2024-03-11T20:09:33.100 INFO:tasks.check_counter:Daemon mds.d mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T20:09:33.100 DEBUG:tasks.check_counter:Getting stats from g 2024-03-11T20:09:33.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-11T20:09:33.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:33 smithi097 ceph-mon[31898]: pgmap v319: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-11T20:09:33.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:33 smithi114 ceph-mon[36376]: pgmap v319: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-11T20:09:33.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:33 smithi161 ceph-mon[36332]: pgmap v319: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-11T20:09:33.569 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:09:35.327 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:35 smithi161 ceph-mon[36332]: pgmap v320: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:35.395 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:35 smithi097 ceph-mon[31898]: pgmap v320: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:35.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:35 smithi114 ceph-mon[36376]: pgmap v320: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:36.014 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T20:09:36.014 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T20:09:36.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 0, 2024-03-11T20:09:36.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 2, 2024-03-11T20:09:36.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 0, 2024-03-11T20:09:36.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 0, 2024-03-11T20:09:36.014 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 4, 2024-03-11T20:09:36.015 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 0, 2024-03-11T20:09:36.015 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.004205384, 2024-03-11T20:09:36.015 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.000000000, 2024-03-11T20:09:36.015 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.001012011, 2024-03-11T20:09:36.015 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T20:09:36.015 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:36.015 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:36.015 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:36.015 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:36.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:36.015 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:36.016 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:36.016 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:36.016 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:36.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:36.016 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T20:09:36.016 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T20:09:36.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:36.016 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T20:09:36.016 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 237, 2024-03-11T20:09:36.016 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 226, 2024-03-11T20:09:36.016 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 1567680, 2024-03-11T20:09:36.017 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 136016, 2024-03-11T20:09:36.017 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 2, 2024-03-11T20:09:36.017 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 1, 2024-03-11T20:09:36.017 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.102407934, 2024-03-11T20:09:36.017 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.035362090, 2024-03-11T20:09:36.017 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.079901556, 2024-03-11T20:09:36.017 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.010386583, 2024-03-11T20:09:36.017 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:36.017 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 225, 2024-03-11T20:09:36.017 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.021180898, 2024-03-11T20:09:36.018 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000094137 2024-03-11T20:09:36.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:36.018 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:36.018 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:36.018 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:36.018 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:36.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:36.018 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 1567680, 2024-03-11T20:09:36.018 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 136016 2024-03-11T20:09:36.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:36.018 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 6, 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 175, 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 3312, 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 185600, 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 5, 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 3, 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.035321950, 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.025451446, 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.004085259, 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 173, 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.027678521, 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000159991 2024-03-11T20:09:36.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 185600 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: "cct": { 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: "total_workers": 0, 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: "unhealthy_workers": 0 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: "mempool": { 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_bytes": 0, 2024-03-11T20:09:36.020 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_items": 0, 2024-03-11T20:09:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T20:09:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_items": 0, 2024-03-11T20:09:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T20:09:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_items": 0, 2024-03-11T20:09:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T20:09:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T20:09:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T20:09:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T20:09:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T20:09:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_items": 0, 2024-03-11T20:09:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T20:09:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T20:09:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_bytes": 0, 2024-03-11T20:09:36.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_items": 0, 2024-03-11T20:09:36.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_bytes": 0, 2024-03-11T20:09:36.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_items": 0, 2024-03-11T20:09:36.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T20:09:36.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T20:09:36.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T20:09:36.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T20:09:36.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T20:09:36.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_items": 0, 2024-03-11T20:09:36.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_bytes": 0, 2024-03-11T20:09:36.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_items": 0, 2024-03-11T20:09:36.022 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_bytes": 0, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_items": 0, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_bytes": 0, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_items": 0, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_items": 0, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_items": 0, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_bytes": 148714, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_items": 30, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_bytes": 0, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_items": 0, 2024-03-11T20:09:36.023 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bytes": 0, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_items": 0, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_bytes": 0, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_items": 0, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_bytes": 0, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_items": 0, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_bytes": 0, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_items": 0, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_items": 0, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_bytes": 0, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_items": 0, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_bytes": 928, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_items": 2, 2024-03-11T20:09:36.024 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_bytes": 0, 2024-03-11T20:09:36.025 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_items": 0, 2024-03-11T20:09:36.025 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_bytes": 0, 2024-03-11T20:09:36.025 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_items": 0 2024-03-11T20:09:36.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:36.025 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T20:09:36.025 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-03-11T20:09:36.025 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 104857600, 2024-03-11T20:09:36.025 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-03-11T20:09:36.025 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 239, 2024-03-11T20:09:36.025 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 1534258, 2024-03-11T20:09:36.025 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:36.025 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 239, 2024-03-11T20:09:36.025 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-03-11T20:09:36.026 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-03-11T20:09:36.026 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 239, 2024-03-11T20:09:36.026 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 1534258, 2024-03-11T20:09:36.026 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-03-11T20:09:36.026 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:36.026 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:36.026 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:36.026 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:36.026 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:36.026 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T20:09:37.072 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-11T20:09:37.072 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-11T20:09:37.072 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-11T20:09:37.072 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-11T20:09:37.072 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-11T20:09:37.072 DEBUG:tasks.check_counter:Getting stats from j 2024-03-11T20:09:37.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-11T20:09:37.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:37 smithi097 ceph-mon[31898]: pgmap v321: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:37.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:37 smithi114 ceph-mon[36376]: pgmap v321: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:37.507 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:09:37.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:37 smithi161 ceph-mon[36332]: pgmap v321: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:39.318 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:39 smithi097 ceph-mon[31898]: pgmap v322: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:39.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:39 smithi114 ceph-mon[36376]: pgmap v322: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:39.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:39 smithi161 ceph-mon[36332]: pgmap v322: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:39.955 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T20:09:39.955 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T20:09:39.955 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 0, 2024-03-11T20:09:39.955 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 2, 2024-03-11T20:09:39.955 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 0, 2024-03-11T20:09:39.955 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 0, 2024-03-11T20:09:39.955 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 4, 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 0, 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.003802052, 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.000000000, 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.000880496, 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:39.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 243, 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 232, 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 1569312, 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 139760, 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 2, 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 1, 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.106876041, 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.035907254, 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.086505651, 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.010391712, 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:39.957 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 231, 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.022339518, 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000096707 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 1569312, 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 139760 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_messages": 6, 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages": 180, 2024-03-11T20:09:39.958 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_bytes": 3312, 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_bytes": 186480, 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_created_connections": 5, 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_active_connections": 3, 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_total_time": 0.035311782, 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_send_time": 0.024800580, 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_recv_time": 0.004672501, 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 178, 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.030787418, 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000172963 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:39.959 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: "msgr_send_encrypted_bytes": 186480 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: "cct": { 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: "total_workers": 0, 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: "unhealthy_workers": 0 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: "mempool": { 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_bytes": 0, 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: "bloom_filter_items": 0, 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_alloc_items": 0, 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T20:09:39.960 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_data_items": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_other_items": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_bytes": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_extent_items": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_bytes": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_blob_items": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T20:09:39.961 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_fsck_items": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_bytes": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_txc_items": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_bytes": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluestore_writing_items": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_bytes": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_items": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_reader_items": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T20:09:39.962 INFO:teuthology.orchestra.run.smithi097.stdout: "bluefs_file_writer_items": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_bytes": 149745, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_anon_items": 31, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_bytes": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "buffer_meta_items": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_bytes": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_items": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_bytes": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_mapbl_items": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_bytes": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "osd_pglog_items": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_bytes": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_items": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "osdmap_mapping_items": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_bytes": 0, 2024-03-11T20:09:39.963 INFO:teuthology.orchestra.run.smithi097.stdout: "pgmap_items": 0, 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_bytes": 928, 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "mds_co_items": 2, 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_bytes": 0, 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_1_items": 0, 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_bytes": 0, 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "unittest_2_items": 0 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 0, 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "max": 104857600, 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "get_started": 0, 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "get": 245, 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "get_sum": 1535038, 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout: "get_or_fail_success": 245, 2024-03-11T20:09:39.965 INFO:teuthology.orchestra.run.smithi097.stdout: "take": 0, 2024-03-11T20:09:39.965 INFO:teuthology.orchestra.run.smithi097.stdout: "take_sum": 0, 2024-03-11T20:09:39.965 INFO:teuthology.orchestra.run.smithi097.stdout: "put": 245, 2024-03-11T20:09:39.965 INFO:teuthology.orchestra.run.smithi097.stdout: "put_sum": 1535038, 2024-03-11T20:09:39.965 INFO:teuthology.orchestra.run.smithi097.stdout: "wait": { 2024-03-11T20:09:39.965 INFO:teuthology.orchestra.run.smithi097.stdout: "avgcount": 0, 2024-03-11T20:09:39.965 INFO:teuthology.orchestra.run.smithi097.stdout: "sum": 0.000000000, 2024-03-11T20:09:39.965 INFO:teuthology.orchestra.run.smithi097.stdout: "avgtime": 0.000000000 2024-03-11T20:09:39.965 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:39.965 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T20:09:39.965 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T20:09:40.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:40 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:09:40.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:40 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:40 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:40 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:40 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:40 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.472 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:40 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:09:40.473 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:40 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.473 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:40 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.473 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:40 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.473 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:40 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.473 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:40 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:40 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:09:40.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:40 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:40 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:40 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:40 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:40 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:40.932 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-11T20:09:40.932 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-11T20:09:40.932 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-11T20:09:40.932 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-11T20:09:40.932 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-11T20:09:40.932 DEBUG:tasks.check_counter:Getting stats from b 2024-03-11T20:09:40.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-11T20:09:41.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:41 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:41.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:41 smithi097 ceph-mon[31898]: pgmap v323: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:41.199 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:09:41.228 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:41 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:41.228 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:41 smithi114 ceph-mon[36376]: pgmap v323: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:41.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:41 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:41.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:41 smithi161 ceph-mon[36332]: pgmap v323: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:42.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:42 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:42.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:42 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:42.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:42 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:42 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:42 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:42 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:42 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:42 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:42 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:42 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.775 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:42 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:42.775 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:42 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:42.775 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:42 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.775 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:42 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.775 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:42 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.775 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:42 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.775 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:42 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.775 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:42 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.775 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:42 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.775 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:42 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:42 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:42.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:42 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:42.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:42 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:42 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:42 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:42 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:42 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:42 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:42 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:42 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:42.926 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-11T20:09:42.926 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T20:09:42.926 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 1070, 2024-03-11T20:09:42.926 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 1005, 2024-03-11T20:09:42.926 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 231158, 2024-03-11T20:09:42.926 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 1686252, 2024-03-11T20:09:42.926 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 13, 2024-03-11T20:09:42.927 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 6, 2024-03-11T20:09:42.927 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.104219754, 2024-03-11T20:09:42.927 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.035950428, 2024-03-11T20:09:42.927 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.032612009, 2024-03-11T20:09:42.927 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.026419616, 2024-03-11T20:09:42.927 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:42.927 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1001, 2024-03-11T20:09:42.927 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.539393544, 2024-03-11T20:09:42.927 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000538854 2024-03-11T20:09:42.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.927 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:42.927 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 791, 2024-03-11T20:09:42.928 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000272711, 2024-03-11T20:09:42.928 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000344 2024-03-11T20:09:42.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.928 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-11T20:09:42.928 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 312688 2024-03-11T20:09:42.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.928 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T20:09:42.928 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 2041, 2024-03-11T20:09:42.928 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 1061, 2024-03-11T20:09:42.928 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 921658, 2024-03-11T20:09:42.928 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 605361, 2024-03-11T20:09:42.928 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 18, 2024-03-11T20:09:42.928 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 6, 2024-03-11T20:09:42.929 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.238584329, 2024-03-11T20:09:42.929 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.070497023, 2024-03-11T20:09:42.929 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.087819241, 2024-03-11T20:09:42.929 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.076872465, 2024-03-11T20:09:42.929 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:42.929 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1060, 2024-03-11T20:09:42.929 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.246833580, 2024-03-11T20:09:42.929 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000232861 2024-03-11T20:09:42.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.929 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:42.929 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2526, 2024-03-11T20:09:42.929 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000460979, 2024-03-11T20:09:42.930 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000182 2024-03-11T20:09:42.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.930 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 347776, 2024-03-11T20:09:42.930 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 245504 2024-03-11T20:09:42.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.930 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T20:09:42.930 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 2116, 2024-03-11T20:09:42.930 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 1206, 2024-03-11T20:09:42.930 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 2471899, 2024-03-11T20:09:42.930 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 675426, 2024-03-11T20:09:42.930 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 31, 2024-03-11T20:09:42.930 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 6, 2024-03-11T20:09:42.931 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.273174470, 2024-03-11T20:09:42.931 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.073389311, 2024-03-11T20:09:42.931 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.123102430, 2024-03-11T20:09:42.931 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.083704442, 2024-03-11T20:09:42.931 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:42.931 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1206, 2024-03-11T20:09:42.931 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.328891468, 2024-03-11T20:09:42.931 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000272712 2024-03-11T20:09:42.931 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.931 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:42.931 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2137, 2024-03-11T20:09:42.931 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000462415, 2024-03-11T20:09:42.931 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000216 2024-03-11T20:09:42.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.932 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 1748416, 2024-03-11T20:09:42.932 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 179280 2024-03-11T20:09:42.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.932 INFO:teuthology.orchestra.run.smithi114.stdout: "cct": { 2024-03-11T20:09:42.932 INFO:teuthology.orchestra.run.smithi114.stdout: "total_workers": 1, 2024-03-11T20:09:42.932 INFO:teuthology.orchestra.run.smithi114.stdout: "unhealthy_workers": 0 2024-03-11T20:09:42.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.932 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-MDSRank": { 2024-03-11T20:09:42.932 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-03-11T20:09:42.932 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-03-11T20:09:42.932 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 614, 2024-03-11T20:09:42.932 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.125756020, 2024-03-11T20:09:42.933 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000204814 2024-03-11T20:09:42.933 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:42.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.933 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-PurgeQueue": { 2024-03-11T20:09:42.933 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-03-11T20:09:42.933 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-03-11T20:09:42.933 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 142, 2024-03-11T20:09:42.933 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.005093872, 2024-03-11T20:09:42.933 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000035872 2024-03-11T20:09:42.933 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:42.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.933 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-03-11T20:09:42.934 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 169, 2024-03-11T20:09:42.934 INFO:teuthology.orchestra.run.smithi114.stdout: "reply": 169, 2024-03-11T20:09:42.934 INFO:teuthology.orchestra.run.smithi114.stdout: "reply_latency": { 2024-03-11T20:09:42.934 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 169, 2024-03-11T20:09:42.934 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.173176273, 2024-03-11T20:09:42.934 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.001024711 2024-03-11T20:09:42.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.934 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_reply": 0, 2024-03-11T20:09:42.934 INFO:teuthology.orchestra.run.smithi114.stdout: "forward": 0, 2024-03-11T20:09:42.934 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_complete": 0, 2024-03-11T20:09:42.934 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_keys": 0, 2024-03-11T20:09:42.934 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_commit": 62, 2024-03-11T20:09:42.934 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_split": 0, 2024-03-11T20:09:42.935 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_merge": 0, 2024-03-11T20:09:42.935 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes": 38, 2024-03-11T20:09:42.935 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_top": 9, 2024-03-11T20:09:42.935 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_bottom": 29, 2024-03-11T20:09:42.935 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pin_tail": 0, 2024-03-11T20:09:42.935 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pinned": 25, 2024-03-11T20:09:42.935 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_expired": 0, 2024-03-11T20:09:42.935 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_with_caps": 3, 2024-03-11T20:09:42.935 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 3, 2024-03-11T20:09:42.935 INFO:teuthology.orchestra.run.smithi114.stdout: "subtrees": 2, 2024-03-11T20:09:42.935 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse": 214, 2024-03-11T20:09:42.935 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_hit": 205, 2024-03-11T20:09:42.935 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_forward": 0, 2024-03-11T20:09:42.936 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_discover": 0, 2024-03-11T20:09:42.936 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_dir_fetch": 0, 2024-03-11T20:09:42.936 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_remote_ino": 0, 2024-03-11T20:09:42.936 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_lock": 0, 2024-03-11T20:09:42.936 INFO:teuthology.orchestra.run.smithi114.stdout: "load_cent": 0, 2024-03-11T20:09:42.936 INFO:teuthology.orchestra.run.smithi114.stdout: "q": 0, 2024-03-11T20:09:42.936 INFO:teuthology.orchestra.run.smithi114.stdout: "exported": 0, 2024-03-11T20:09:42.936 INFO:teuthology.orchestra.run.smithi114.stdout: "exported_inodes": 0, 2024-03-11T20:09:42.936 INFO:teuthology.orchestra.run.smithi114.stdout: "imported": 0, 2024-03-11T20:09:42.936 INFO:teuthology.orchestra.run.smithi114.stdout: "imported_inodes": 0, 2024-03-11T20:09:42.936 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_dir_fetch": 0, 2024-03-11T20:09:42.936 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_backtrace_fetch": 0, 2024-03-11T20:09:42.937 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_peer_discover": 0, 2024-03-11T20:09:42.937 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rfiles": 4, 2024-03-11T20:09:42.937 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rbytes": 234, 2024-03-11T20:09:42.937 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rsnaps": 6, 2024-03-11T20:09:42.937 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T20:09:42.937 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_set_tag": 0, 2024-03-11T20:09:42.937 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T20:09:42.937 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_inotable_repaired": 0, 2024-03-11T20:09:42.937 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_inodes": 0, 2024-03-11T20:09:42.937 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T20:09:42.937 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T20:09:42.937 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_file_inodes": 0, 2024-03-11T20:09:42.937 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_inode_file_caps": 1, 2024-03-11T20:09:42.938 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_revoke": 18, 2024-03-11T20:09:42.938 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_grant": 33, 2024-03-11T20:09:42.938 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T20:09:42.938 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T20:09:42.938 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T20:09:42.938 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps": 111, 2024-03-11T20:09:42.938 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps_dirty": 89, 2024-03-11T20:09:42.938 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_cap_release": 1, 2024-03-11T20:09:42.938 INFO:teuthology.orchestra.run.smithi114.stdout: "process_request_cap_release": 27 2024-03-11T20:09:42.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.938 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache": { 2024-03-11T20:09:42.938 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update": 0, 2024-03-11T20:09:42.938 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update_receipt": 0, 2024-03-11T20:09:42.939 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_try_discover": 0, 2024-03-11T20:09:42.939 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_send_discover": 0, 2024-03-11T20:09:42.939 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_handle_discover": 7, 2024-03-11T20:09:42.939 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays": 2, 2024-03-11T20:09:42.939 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_delayed": 0, 2024-03-11T20:09:42.939 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_enqueuing": 0, 2024-03-11T20:09:42.939 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_created": 6, 2024-03-11T20:09:42.939 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_enqueued": 4, 2024-03-11T20:09:42.939 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_reintegrated": 0, 2024-03-11T20:09:42.939 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_migrated": 0, 2024-03-11T20:09:42.939 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_processing": 0, 2024-03-11T20:09:42.939 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_enqueued": 0, 2024-03-11T20:09:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_prioritized": 0, 2024-03-11T20:09:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_started": 0, 2024-03-11T20:09:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_completed": 0, 2024-03-11T20:09:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T20:09:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_exportdir": 0, 2024-03-11T20:09:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_flush": 0, 2024-03-11T20:09:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragmentdir": 0, 2024-03-11T20:09:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragstats": 0, 2024-03-11T20:09:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_inodestats": 0 2024-03-11T20:09:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log": { 2024-03-11T20:09:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: "evlrg": 0, 2024-03-11T20:09:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: "evadd": 258, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "evex": 170, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "evtrm": 170, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "ev": 88, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "evexg": 0, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "evexd": 23, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "segadd": 12, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "segex": 11, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "segtrm": 11, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "seg": 1, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "segmjr": 1, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "segexg": 0, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "segexd": 1, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "expos": 5075625, 2024-03-11T20:09:42.941 INFO:teuthology.orchestra.run.smithi114.stdout: "wrpos": 5264552, 2024-03-11T20:09:42.942 INFO:teuthology.orchestra.run.smithi114.stdout: "rdpos": 0, 2024-03-11T20:09:42.942 INFO:teuthology.orchestra.run.smithi114.stdout: "jlat": { 2024-03-11T20:09:42.942 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 213, 2024-03-11T20:09:42.942 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.430262889, 2024-03-11T20:09:42.942 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.002020013 2024-03-11T20:09:42.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.942 INFO:teuthology.orchestra.run.smithi114.stdout: "replayed": 0 2024-03-11T20:09:42.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.942 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_mem": { 2024-03-11T20:09:42.942 INFO:teuthology.orchestra.run.smithi114.stdout: "ino": 29, 2024-03-11T20:09:42.942 INFO:teuthology.orchestra.run.smithi114.stdout: "ino+": 33, 2024-03-11T20:09:42.942 INFO:teuthology.orchestra.run.smithi114.stdout: "ino-": 4, 2024-03-11T20:09:42.943 INFO:teuthology.orchestra.run.smithi114.stdout: "dir": 20, 2024-03-11T20:09:42.943 INFO:teuthology.orchestra.run.smithi114.stdout: "dir+": 20, 2024-03-11T20:09:42.943 INFO:teuthology.orchestra.run.smithi114.stdout: "dir-": 0, 2024-03-11T20:09:42.943 INFO:teuthology.orchestra.run.smithi114.stdout: "dn": 38, 2024-03-11T20:09:42.943 INFO:teuthology.orchestra.run.smithi114.stdout: "dn+": 42, 2024-03-11T20:09:42.943 INFO:teuthology.orchestra.run.smithi114.stdout: "dn-": 4, 2024-03-11T20:09:42.943 INFO:teuthology.orchestra.run.smithi114.stdout: "cap": 3, 2024-03-11T20:09:42.943 INFO:teuthology.orchestra.run.smithi114.stdout: "cap+": 21, 2024-03-11T20:09:42.943 INFO:teuthology.orchestra.run.smithi114.stdout: "cap-": 18, 2024-03-11T20:09:42.943 INFO:teuthology.orchestra.run.smithi114.stdout: "rss": 52688, 2024-03-11T20:09:42.943 INFO:teuthology.orchestra.run.smithi114.stdout: "heap": 215420 2024-03-11T20:09:42.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.943 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_server": { 2024-03-11T20:09:42.944 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_client_request": 216, 2024-03-11T20:09:42.944 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_server_request": 1, 2024-03-11T20:09:42.944 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_request": 169, 2024-03-11T20:09:42.944 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_session": 36, 2024-03-11T20:09:42.944 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_peer_request": 2, 2024-03-11T20:09:42.944 INFO:teuthology.orchestra.run.smithi114.stdout: "req_create_latency": { 2024-03-11T20:09:42.944 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 8, 2024-03-11T20:09:42.944 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.007128071, 2024-03-11T20:09:42.944 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000891008 2024-03-11T20:09:42.944 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.944 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getattr_latency": { 2024-03-11T20:09:42.944 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 107, 2024-03-11T20:09:42.944 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.024759903, 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000231400 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getfilelock_latency": { 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: "req_link_latency": { 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookup_latency": { 2024-03-11T20:09:42.945 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1, 2024-03-11T20:09:42.946 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000332196, 2024-03-11T20:09:42.946 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000332196 2024-03-11T20:09:42.946 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.946 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookuphash_latency": { 2024-03-11T20:09:42.946 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.946 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.946 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.946 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.946 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupino_latency": { 2024-03-11T20:09:42.946 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1, 2024-03-11T20:09:42.946 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000108662, 2024-03-11T20:09:42.946 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000108662 2024-03-11T20:09:42.947 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.947 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupname_latency": { 2024-03-11T20:09:42.947 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.947 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.947 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.947 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.947 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupparent_latency": { 2024-03-11T20:09:42.947 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.947 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.947 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.947 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.947 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupsnap_latency": { 2024-03-11T20:09:42.947 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.948 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.948 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.948 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lssnap_latency": { 2024-03-11T20:09:42.948 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 10, 2024-03-11T20:09:42.948 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.002101322, 2024-03-11T20:09:42.948 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000210132 2024-03-11T20:09:42.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.948 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mkdir_latency": { 2024-03-11T20:09:42.948 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 8, 2024-03-11T20:09:42.949 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.014073455, 2024-03-11T20:09:42.949 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.001759181 2024-03-11T20:09:42.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.949 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mknod_latency": { 2024-03-11T20:09:42.949 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.949 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.949 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.950 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mksnap_latency": { 2024-03-11T20:09:42.950 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 10, 2024-03-11T20:09:42.950 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.055311239, 2024-03-11T20:09:42.950 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.005531123 2024-03-11T20:09:42.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.950 INFO:teuthology.orchestra.run.smithi114.stdout: "req_open_latency": { 2024-03-11T20:09:42.950 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.950 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.950 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.950 INFO:teuthology.orchestra.run.smithi114.stdout: "req_readdir_latency": { 2024-03-11T20:09:42.951 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-03-11T20:09:42.951 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.003427360, 2024-03-11T20:09:42.951 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.001713680 2024-03-11T20:09:42.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.951 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rename_latency": { 2024-03-11T20:09:42.951 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-03-11T20:09:42.951 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.002511159, 2024-03-11T20:09:42.951 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.001255579 2024-03-11T20:09:42.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.951 INFO:teuthology.orchestra.run.smithi114.stdout: "req_renamesnap_latency": { 2024-03-11T20:09:42.951 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.951 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.951 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.952 INFO:teuthology.orchestra.run.smithi114.stdout: "req_snapdiff_latency": { 2024-03-11T20:09:42.952 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.952 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.952 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.952 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmdir_latency": { 2024-03-11T20:09:42.952 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-03-11T20:09:42.952 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.009849544, 2024-03-11T20:09:42.952 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.004924772 2024-03-11T20:09:42.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.952 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmsnap_latency": { 2024-03-11T20:09:42.953 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 4, 2024-03-11T20:09:42.953 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.021033306, 2024-03-11T20:09:42.953 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.005258326 2024-03-11T20:09:42.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.953 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmxattr_latency": { 2024-03-11T20:09:42.953 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.953 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.953 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.953 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setattr_latency": { 2024-03-11T20:09:42.953 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-03-11T20:09:42.953 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.001483067, 2024-03-11T20:09:42.953 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000741533 2024-03-11T20:09:42.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.954 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setdirlayout_latency": { 2024-03-11T20:09:42.954 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.954 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.954 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.954 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setfilelock_latency": { 2024-03-11T20:09:42.954 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.954 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.954 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.954 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setlayout_latency": { 2024-03-11T20:09:42.955 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.955 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.955 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.955 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setxattr_latency": { 2024-03-11T20:09:42.955 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 8, 2024-03-11T20:09:42.955 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.023833441, 2024-03-11T20:09:42.955 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.002979180 2024-03-11T20:09:42.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.955 INFO:teuthology.orchestra.run.smithi114.stdout: "req_symlink_latency": { 2024-03-11T20:09:42.955 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.955 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.955 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.956 INFO:teuthology.orchestra.run.smithi114.stdout: "req_unlink_latency": { 2024-03-11T20:09:42.956 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 4, 2024-03-11T20:09:42.956 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.007223548, 2024-03-11T20:09:42.956 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.001805887 2024-03-11T20:09:42.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.956 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_revoke_eviction": 0, 2024-03-11T20:09:42.956 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_acquisition_throttle": 0, 2024-03-11T20:09:42.956 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getvxattr_latency": { 2024-03-11T20:09:42.956 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.956 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.956 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.956 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:42.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.957 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_sessions": { 2024-03-11T20:09:42.957 INFO:teuthology.orchestra.run.smithi114.stdout: "session_count": 2, 2024-03-11T20:09:42.957 INFO:teuthology.orchestra.run.smithi114.stdout: "session_add": 6, 2024-03-11T20:09:42.957 INFO:teuthology.orchestra.run.smithi114.stdout: "session_remove": 4, 2024-03-11T20:09:42.957 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_open": 2, 2024-03-11T20:09:42.957 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_stale": 0, 2024-03-11T20:09:42.957 INFO:teuthology.orchestra.run.smithi114.stdout: "total_load": 27, 2024-03-11T20:09:42.957 INFO:teuthology.orchestra.run.smithi114.stdout: "average_load": 13, 2024-03-11T20:09:42.957 INFO:teuthology.orchestra.run.smithi114.stdout: "avg_session_uptime": 606, 2024-03-11T20:09:42.957 INFO:teuthology.orchestra.run.smithi114.stdout: "mdthresh_evicted": 0 2024-03-11T20:09:42.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.957 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool": { 2024-03-11T20:09:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_bytes": 0, 2024-03-11T20:09:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_items": 0, 2024-03-11T20:09:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T20:09:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_items": 0, 2024-03-11T20:09:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T20:09:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_items": 0, 2024-03-11T20:09:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T20:09:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T20:09:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T20:09:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T20:09:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T20:09:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_items": 0, 2024-03-11T20:09:42.958 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T20:09:42.959 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T20:09:42.959 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_bytes": 0, 2024-03-11T20:09:42.959 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_items": 0, 2024-03-11T20:09:42.959 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_bytes": 0, 2024-03-11T20:09:42.959 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_items": 0, 2024-03-11T20:09:42.959 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T20:09:42.959 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T20:09:42.959 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T20:09:42.959 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T20:09:42.959 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T20:09:42.959 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_items": 0, 2024-03-11T20:09:42.959 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_bytes": 0, 2024-03-11T20:09:42.959 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_items": 0, 2024-03-11T20:09:42.960 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T20:09:42.960 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T20:09:42.960 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_bytes": 0, 2024-03-11T20:09:42.960 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_items": 0, 2024-03-11T20:09:42.960 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_bytes": 0, 2024-03-11T20:09:42.960 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_items": 0, 2024-03-11T20:09:42.960 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T20:09:42.960 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_items": 0, 2024-03-11T20:09:42.960 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T20:09:42.960 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_items": 0, 2024-03-11T20:09:42.960 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_bytes": 364664, 2024-03-11T20:09:42.960 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_items": 156, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_bytes": 704, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_items": 8, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bytes": 0, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_items": 0, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_bytes": 0, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_items": 0, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_bytes": 0, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_items": 0, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_bytes": 9616, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_items": 185, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_items": 0, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_bytes": 0, 2024-03-11T20:09:42.961 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_items": 0, 2024-03-11T20:09:42.962 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_bytes": 131165, 2024-03-11T20:09:42.962 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_items": 1167, 2024-03-11T20:09:42.962 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_bytes": 0, 2024-03-11T20:09:42.962 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_items": 0, 2024-03-11T20:09:42.962 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_bytes": 0, 2024-03-11T20:09:42.962 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_items": 0 2024-03-11T20:09:42.962 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.962 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter": { 2024-03-11T20:09:42.962 INFO:teuthology.orchestra.run.smithi114.stdout: "op_active": 0, 2024-03-11T20:09:42.962 INFO:teuthology.orchestra.run.smithi114.stdout: "op_laggy": 0, 2024-03-11T20:09:42.962 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send": 1057, 2024-03-11T20:09:42.962 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send_bytes": 1181301, 2024-03-11T20:09:42.963 INFO:teuthology.orchestra.run.smithi114.stdout: "op_resend": 37, 2024-03-11T20:09:42.963 INFO:teuthology.orchestra.run.smithi114.stdout: "op_reply": 1057, 2024-03-11T20:09:42.963 INFO:teuthology.orchestra.run.smithi114.stdout: "op_latency": { 2024-03-11T20:09:42.963 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1057, 2024-03-11T20:09:42.963 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 24.134481964, 2024-03-11T20:09:42.963 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.022833000 2024-03-11T20:09:42.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.963 INFO:teuthology.orchestra.run.smithi114.stdout: "op_inflight": 0, 2024-03-11T20:09:42.963 INFO:teuthology.orchestra.run.smithi114.stdout: "oplen_avg": { 2024-03-11T20:09:42.963 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1057, 2024-03-11T20:09:42.963 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 1238 2024-03-11T20:09:42.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.963 INFO:teuthology.orchestra.run.smithi114.stdout: "op": 1057, 2024-03-11T20:09:42.964 INFO:teuthology.orchestra.run.smithi114.stdout: "op_r": 1, 2024-03-11T20:09:42.964 INFO:teuthology.orchestra.run.smithi114.stdout: "op_w": 1056, 2024-03-11T20:09:42.964 INFO:teuthology.orchestra.run.smithi114.stdout: "op_rmw": 0, 2024-03-11T20:09:42.964 INFO:teuthology.orchestra.run.smithi114.stdout: "op_pg": 0, 2024-03-11T20:09:42.964 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_stat": 42, 2024-03-11T20:09:42.964 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_create": 36, 2024-03-11T20:09:42.964 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_read": 1, 2024-03-11T20:09:42.964 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_write": 214, 2024-03-11T20:09:42.964 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writefull": 57, 2024-03-11T20:09:42.964 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writesame": 0, 2024-03-11T20:09:42.964 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_append": 0, 2024-03-11T20:09:42.964 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_zero": 0, 2024-03-11T20:09:42.964 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_truncate": 0, 2024-03-11T20:09:42.965 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_delete": 648, 2024-03-11T20:09:42.965 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_mapext": 0, 2024-03-11T20:09:42.965 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_sparse_read": 0, 2024-03-11T20:09:42.965 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_clonerange": 0, 2024-03-11T20:09:42.965 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_getxattr": 0, 2024-03-11T20:09:42.965 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_setxattr": 64, 2024-03-11T20:09:42.965 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_cmpxattr": 0, 2024-03-11T20:09:42.965 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_rmxattr": 0, 2024-03-11T20:09:42.965 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_resetxattrs": 0, 2024-03-11T20:09:42.965 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_call": 0, 2024-03-11T20:09:42.965 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_watch": 0, 2024-03-11T20:09:42.965 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_notify": 0, 2024-03-11T20:09:42.965 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T20:09:42.966 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls": 0, 2024-03-11T20:09:42.966 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls_filter": 0, 2024-03-11T20:09:42.966 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_other": 0, 2024-03-11T20:09:42.966 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_active": 0, 2024-03-11T20:09:42.966 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_send": 0, 2024-03-11T20:09:42.966 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_resend": 0, 2024-03-11T20:09:42.966 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_ping": 0, 2024-03-11T20:09:42.966 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_active": 0, 2024-03-11T20:09:42.966 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_send": 0, 2024-03-11T20:09:42.966 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_resend": 0, 2024-03-11T20:09:42.966 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_active": 0, 2024-03-11T20:09:42.966 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_send": 0, 2024-03-11T20:09:42.966 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_resend": 0, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_active": 0, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_send": 0, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_resend": 0, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "command_active": 0, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "command_send": 0, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "command_resend": 0, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "map_epoch": 102, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "map_full": 0, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "map_inc": 18, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_sessions": 12, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_open": 12, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_close": 0, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_laggy": 0, 2024-03-11T20:09:42.967 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_wr": 165, 2024-03-11T20:09:42.968 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_rd": 0, 2024-03-11T20:09:42.968 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_del": 11 2024-03-11T20:09:42.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.968 INFO:teuthology.orchestra.run.smithi114.stdout: "oft": { 2024-03-11T20:09:42.968 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_objs": 1, 2024-03-11T20:09:42.968 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_kv_pairs": 0, 2024-03-11T20:09:42.968 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_updates": 0, 2024-03-11T20:09:42.968 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_removes": 0 2024-03-11T20:09:42.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.968 INFO:teuthology.orchestra.run.smithi114.stdout: "osd-slow-ops": { 2024-03-11T20:09:42.968 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_ops_count": 0 2024-03-11T20:09:42.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: "purge_queue": { 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops": 0, 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops_high_water": 136, 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing": 0, 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_high_water": 4, 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed_ops": 136, 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed": 4, 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_item_in_journal": 0 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:42.969 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 5223, 2024-03-11T20:09:42.970 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 3089107, 2024-03-11T20:09:42.970 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:42.970 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 5223, 2024-03-11T20:09:42.970 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-03-11T20:09:42.970 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-03-11T20:09:42.970 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 5223, 2024-03-11T20:09:42.970 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 3089107, 2024-03-11T20:09:42.970 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:42.970 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.970 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.970 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.970 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:42.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.971 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_bytes": { 2024-03-11T20:09:42.971 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:42.971 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-03-11T20:09:42.971 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:42.971 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-03-11T20:09:42.971 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-03-11T20:09:42.971 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:42.971 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:42.971 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 1057, 2024-03-11T20:09:42.971 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 1181769, 2024-03-11T20:09:42.971 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 1057, 2024-03-11T20:09:42.971 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1181769, 2024-03-11T20:09:42.971 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:42.972 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.972 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.972 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.972 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:42.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.972 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_ops": { 2024-03-11T20:09:42.972 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:42.972 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 1024, 2024-03-11T20:09:42.972 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:42.972 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-03-11T20:09:42.972 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-03-11T20:09:42.972 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:42.972 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:42.973 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 1057, 2024-03-11T20:09:42.973 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 1057, 2024-03-11T20:09:42.973 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 1057, 2024-03-11T20:09:42.973 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1057, 2024-03-11T20:09:42.973 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:42.973 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.973 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.973 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.973 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:42.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.973 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle": { 2024-03-11T20:09:42.973 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:42.973 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-03-11T20:09:42.974 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:42.974 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 4, 2024-03-11T20:09:42.974 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 468, 2024-03-11T20:09:42.974 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:42.974 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 4, 2024-03-11T20:09:42.974 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-03-11T20:09:42.974 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-03-11T20:09:42.974 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 1, 2024-03-11T20:09:42.974 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 468, 2024-03-11T20:09:42.974 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:42.974 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.974 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.974 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle-0x563383ebe6e0": { 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 258, 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 1070248, 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 258, 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 213, 2024-03-11T20:09:42.975 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1070248, 2024-03-11T20:09:42.976 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:42.976 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:42.976 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:42.976 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:42.976 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:42.976 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:42.976 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-11T20:09:43.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:43 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi161 to 174.1M 2024-03-11T20:09:43.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:43 smithi114 ceph-mon[36376]: Unable to set osd_memory_target on smithi161 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-11T20:09:43.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:43 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:43.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:43 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:43.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:43 smithi114 ceph-mon[36376]: pgmap v324: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:43.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:43 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:43.808 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:43 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:43.810 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-11T20:09:43.810 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-11T20:09:43.810 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-11T20:09:43.810 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-03-11T20:09:43.810 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-03-11T20:09:43.810 DEBUG:tasks.check_counter:Getting stats from e 2024-03-11T20:09:43.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-11T20:09:44.076 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:09:44.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:43 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi161 to 174.1M 2024-03-11T20:09:44.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:43 smithi097 ceph-mon[31898]: Unable to set osd_memory_target on smithi161 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-11T20:09:44.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:43 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:44.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:43 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:44.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:43 smithi097 ceph-mon[31898]: pgmap v324: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:44.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:43 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:44.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:43 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:44.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:43 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi161 to 174.1M 2024-03-11T20:09:44.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:43 smithi161 ceph-mon[36332]: Unable to set osd_memory_target on smithi161 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-11T20:09:44.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:43 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:44.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:43 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:44.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:43 smithi161 ceph-mon[36332]: pgmap v324: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:44.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:43 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:44.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:43 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:45.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.158 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:09:45.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:09:45.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:44 smithi097 ceph-mon[31898]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:45.188 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.189 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.190 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.190 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:09:45.190 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:09:45.190 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:44 smithi114 ceph-mon[36376]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:45.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:09:45.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:09:45.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:09:45.276 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:44 smithi161 ceph-mon[36332]: from='mgr.15396 ' entity='mgr.x' 2024-03-11T20:09:45.688 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-11T20:09:45.689 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T20:09:45.689 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 266, 2024-03-11T20:09:45.689 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 251, 2024-03-11T20:09:45.689 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 1619250, 2024-03-11T20:09:45.689 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 143197, 2024-03-11T20:09:45.689 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 5, 2024-03-11T20:09:45.689 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 1, 2024-03-11T20:09:45.689 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.053449311, 2024-03-11T20:09:45.689 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.016639271, 2024-03-11T20:09:45.689 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.038669050, 2024-03-11T20:09:45.690 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.006078998, 2024-03-11T20:09:45.690 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:45.690 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 250, 2024-03-11T20:09:45.690 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.013889169, 2024-03-11T20:09:45.690 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000055556 2024-03-11T20:09:45.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.690 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:45.690 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.690 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.690 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.690 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 1618416, 2024-03-11T20:09:45.691 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 142880 2024-03-11T20:09:45.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.691 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T20:09:45.691 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 5, 2024-03-11T20:09:45.691 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 182, 2024-03-11T20:09:45.691 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 994, 2024-03-11T20:09:45.691 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 538659, 2024-03-11T20:09:45.691 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 7, 2024-03-11T20:09:45.691 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 4, 2024-03-11T20:09:45.691 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.011355714, 2024-03-11T20:09:45.691 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.007270701, 2024-03-11T20:09:45.692 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.001888149, 2024-03-11T20:09:45.692 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000056569, 2024-03-11T20:09:45.692 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:45.692 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 178, 2024-03-11T20:09:45.692 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.020015618, 2024-03-11T20:09:45.692 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000112447 2024-03-11T20:09:45.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.692 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:45.692 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.692 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.692 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.693 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-11T20:09:45.693 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 538032 2024-03-11T20:09:45.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.693 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T20:09:45.694 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 1619, 2024-03-11T20:09:45.694 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 1619, 2024-03-11T20:09:45.694 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 486405, 2024-03-11T20:09:45.694 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 502771, 2024-03-11T20:09:45.694 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 4, 2024-03-11T20:09:45.694 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 3, 2024-03-11T20:09:45.695 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.165242645, 2024-03-11T20:09:45.695 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.057393716, 2024-03-11T20:09:45.695 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.041449383, 2024-03-11T20:09:45.695 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.050936547, 2024-03-11T20:09:45.695 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:45.695 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1618, 2024-03-11T20:09:45.695 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.069944055, 2024-03-11T20:09:45.695 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000043228 2024-03-11T20:09:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-11T20:09:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 448 2024-03-11T20:09:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "cct": { 2024-03-11T20:09:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "total_workers": 1, 2024-03-11T20:09:45.696 INFO:teuthology.orchestra.run.smithi114.stdout: "unhealthy_workers": 0 2024-03-11T20:09:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-MDSRank": { 2024-03-11T20:09:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-03-11T20:09:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-03-11T20:09:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 3229, 2024-03-11T20:09:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.095523253, 2024-03-11T20:09:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000029582 2024-03-11T20:09:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-PurgeQueue": { 2024-03-11T20:09:45.697 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-03-11T20:09:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-03-11T20:09:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-03-11T20:09:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 0, 2024-03-11T20:09:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "reply": 0, 2024-03-11T20:09:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "reply_latency": { 2024-03-11T20:09:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.698 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_reply": 0, 2024-03-11T20:09:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "forward": 0, 2024-03-11T20:09:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_complete": 0, 2024-03-11T20:09:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_keys": 0, 2024-03-11T20:09:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_commit": 0, 2024-03-11T20:09:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_split": 0, 2024-03-11T20:09:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_merge": 0, 2024-03-11T20:09:45.699 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes": 0, 2024-03-11T20:09:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_top": 0, 2024-03-11T20:09:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_bottom": 0, 2024-03-11T20:09:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pin_tail": 0, 2024-03-11T20:09:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pinned": 0, 2024-03-11T20:09:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_expired": 0, 2024-03-11T20:09:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_with_caps": 0, 2024-03-11T20:09:45.700 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-03-11T20:09:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "subtrees": 1, 2024-03-11T20:09:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse": 0, 2024-03-11T20:09:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_hit": 0, 2024-03-11T20:09:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_forward": 0, 2024-03-11T20:09:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_discover": 0, 2024-03-11T20:09:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_dir_fetch": 0, 2024-03-11T20:09:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_remote_ino": 0, 2024-03-11T20:09:45.701 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_lock": 0, 2024-03-11T20:09:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "load_cent": 0, 2024-03-11T20:09:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "q": 0, 2024-03-11T20:09:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "exported": 0, 2024-03-11T20:09:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "exported_inodes": 0, 2024-03-11T20:09:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "imported": 0, 2024-03-11T20:09:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "imported_inodes": 0, 2024-03-11T20:09:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_dir_fetch": 0, 2024-03-11T20:09:45.702 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_backtrace_fetch": 0, 2024-03-11T20:09:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_peer_discover": 0, 2024-03-11T20:09:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rfiles": 0, 2024-03-11T20:09:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rbytes": 0, 2024-03-11T20:09:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rsnaps": 0, 2024-03-11T20:09:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T20:09:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_set_tag": 0, 2024-03-11T20:09:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T20:09:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_inotable_repaired": 0, 2024-03-11T20:09:45.703 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_inodes": 0, 2024-03-11T20:09:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T20:09:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T20:09:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_file_inodes": 0, 2024-03-11T20:09:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_inode_file_caps": 0, 2024-03-11T20:09:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T20:09:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_grant": 0, 2024-03-11T20:09:45.704 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T20:09:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T20:09:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T20:09:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps": 0, 2024-03-11T20:09:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps_dirty": 0, 2024-03-11T20:09:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_cap_release": 0, 2024-03-11T20:09:45.705 INFO:teuthology.orchestra.run.smithi114.stdout: "process_request_cap_release": 0 2024-03-11T20:09:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache": { 2024-03-11T20:09:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update": 0, 2024-03-11T20:09:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update_receipt": 0, 2024-03-11T20:09:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_try_discover": 0, 2024-03-11T20:09:45.706 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_send_discover": 0, 2024-03-11T20:09:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_handle_discover": 0, 2024-03-11T20:09:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays": 0, 2024-03-11T20:09:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_delayed": 0, 2024-03-11T20:09:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_enqueuing": 0, 2024-03-11T20:09:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_created": 0, 2024-03-11T20:09:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_enqueued": 0, 2024-03-11T20:09:45.707 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_reintegrated": 0, 2024-03-11T20:09:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_migrated": 0, 2024-03-11T20:09:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_processing": 0, 2024-03-11T20:09:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_enqueued": 0, 2024-03-11T20:09:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_prioritized": 0, 2024-03-11T20:09:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_started": 0, 2024-03-11T20:09:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_completed": 0, 2024-03-11T20:09:45.708 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T20:09:45.709 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_exportdir": 0, 2024-03-11T20:09:45.709 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_flush": 0, 2024-03-11T20:09:45.709 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragmentdir": 0, 2024-03-11T20:09:45.709 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragstats": 0, 2024-03-11T20:09:45.709 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_inodestats": 0 2024-03-11T20:09:45.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.709 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log": { 2024-03-11T20:09:45.709 INFO:teuthology.orchestra.run.smithi114.stdout: "evlrg": 0, 2024-03-11T20:09:45.710 INFO:teuthology.orchestra.run.smithi114.stdout: "evadd": 0, 2024-03-11T20:09:45.710 INFO:teuthology.orchestra.run.smithi114.stdout: "evex": 0, 2024-03-11T20:09:45.710 INFO:teuthology.orchestra.run.smithi114.stdout: "evtrm": 0, 2024-03-11T20:09:45.710 INFO:teuthology.orchestra.run.smithi114.stdout: "ev": 2, 2024-03-11T20:09:45.710 INFO:teuthology.orchestra.run.smithi114.stdout: "evexg": 0, 2024-03-11T20:09:45.710 INFO:teuthology.orchestra.run.smithi114.stdout: "evexd": 0, 2024-03-11T20:09:45.710 INFO:teuthology.orchestra.run.smithi114.stdout: "segadd": 0, 2024-03-11T20:09:45.710 INFO:teuthology.orchestra.run.smithi114.stdout: "segex": 0, 2024-03-11T20:09:45.711 INFO:teuthology.orchestra.run.smithi114.stdout: "segtrm": 0, 2024-03-11T20:09:45.711 INFO:teuthology.orchestra.run.smithi114.stdout: "seg": 2, 2024-03-11T20:09:45.711 INFO:teuthology.orchestra.run.smithi114.stdout: "segmjr": 2, 2024-03-11T20:09:45.711 INFO:teuthology.orchestra.run.smithi114.stdout: "segexg": 0, 2024-03-11T20:09:45.711 INFO:teuthology.orchestra.run.smithi114.stdout: "segexd": 0, 2024-03-11T20:09:45.711 INFO:teuthology.orchestra.run.smithi114.stdout: "expos": 4194304, 2024-03-11T20:09:45.711 INFO:teuthology.orchestra.run.smithi114.stdout: "wrpos": 4194892, 2024-03-11T20:09:45.712 INFO:teuthology.orchestra.run.smithi114.stdout: "rdpos": 4194352, 2024-03-11T20:09:45.712 INFO:teuthology.orchestra.run.smithi114.stdout: "jlat": { 2024-03-11T20:09:45.712 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.712 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.712 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.712 INFO:teuthology.orchestra.run.smithi114.stdout: "replayed": 2 2024-03-11T20:09:45.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.713 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_mem": { 2024-03-11T20:09:45.713 INFO:teuthology.orchestra.run.smithi114.stdout: "ino": 3, 2024-03-11T20:09:45.713 INFO:teuthology.orchestra.run.smithi114.stdout: "ino+": 3, 2024-03-11T20:09:45.713 INFO:teuthology.orchestra.run.smithi114.stdout: "ino-": 0, 2024-03-11T20:09:45.713 INFO:teuthology.orchestra.run.smithi114.stdout: "dir": 1, 2024-03-11T20:09:45.713 INFO:teuthology.orchestra.run.smithi114.stdout: "dir+": 1, 2024-03-11T20:09:45.714 INFO:teuthology.orchestra.run.smithi114.stdout: "dir-": 0, 2024-03-11T20:09:45.714 INFO:teuthology.orchestra.run.smithi114.stdout: "dn": 0, 2024-03-11T20:09:45.714 INFO:teuthology.orchestra.run.smithi114.stdout: "dn+": 0, 2024-03-11T20:09:45.714 INFO:teuthology.orchestra.run.smithi114.stdout: "dn-": 0, 2024-03-11T20:09:45.714 INFO:teuthology.orchestra.run.smithi114.stdout: "cap": 0, 2024-03-11T20:09:45.714 INFO:teuthology.orchestra.run.smithi114.stdout: "cap+": 0, 2024-03-11T20:09:45.714 INFO:teuthology.orchestra.run.smithi114.stdout: "cap-": 0, 2024-03-11T20:09:45.715 INFO:teuthology.orchestra.run.smithi114.stdout: "rss": 46428, 2024-03-11T20:09:45.715 INFO:teuthology.orchestra.run.smithi114.stdout: "heap": 215420 2024-03-11T20:09:45.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.715 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_server": { 2024-03-11T20:09:45.715 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_client_request": 0, 2024-03-11T20:09:45.715 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_server_request": 0, 2024-03-11T20:09:45.715 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_request": 0, 2024-03-11T20:09:45.715 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_session": 0, 2024-03-11T20:09:45.715 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_peer_request": 0, 2024-03-11T20:09:45.715 INFO:teuthology.orchestra.run.smithi114.stdout: "req_create_latency": { 2024-03-11T20:09:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getattr_latency": { 2024-03-11T20:09:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getfilelock_latency": { 2024-03-11T20:09:45.716 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: "req_link_latency": { 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookup_latency": { 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookuphash_latency": { 2024-03-11T20:09:45.717 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupino_latency": { 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupname_latency": { 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.718 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupparent_latency": { 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupsnap_latency": { 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lssnap_latency": { 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.719 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mkdir_latency": { 2024-03-11T20:09:45.720 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.720 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.720 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.720 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mknod_latency": { 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mksnap_latency": { 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: "req_open_latency": { 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.721 INFO:teuthology.orchestra.run.smithi114.stdout: "req_readdir_latency": { 2024-03-11T20:09:45.722 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.722 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.722 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.722 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.722 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rename_latency": { 2024-03-11T20:09:45.722 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.722 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.722 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.722 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.722 INFO:teuthology.orchestra.run.smithi114.stdout: "req_renamesnap_latency": { 2024-03-11T20:09:45.722 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.722 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.723 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.723 INFO:teuthology.orchestra.run.smithi114.stdout: "req_snapdiff_latency": { 2024-03-11T20:09:45.723 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.723 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.723 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.723 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmdir_latency": { 2024-03-11T20:09:45.723 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.723 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmsnap_latency": { 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmxattr_latency": { 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setattr_latency": { 2024-03-11T20:09:45.724 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.725 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.725 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.725 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setdirlayout_latency": { 2024-03-11T20:09:45.725 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.725 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.725 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.725 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setfilelock_latency": { 2024-03-11T20:09:45.726 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.726 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.726 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.726 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setlayout_latency": { 2024-03-11T20:09:45.726 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.726 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.726 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.726 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setxattr_latency": { 2024-03-11T20:09:45.726 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.726 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.726 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: "req_symlink_latency": { 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: "req_unlink_latency": { 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_revoke_eviction": 0, 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_acquisition_throttle": 0, 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getvxattr_latency": { 2024-03-11T20:09:45.727 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_sessions": { 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: "session_count": 0, 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: "session_add": 0, 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: "session_remove": 0, 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_open": 0, 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_stale": 0, 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: "total_load": 0, 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: "average_load": 0, 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: "avg_session_uptime": 0, 2024-03-11T20:09:45.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mdthresh_evicted": 0 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool": { 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_bytes": 0, 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_items": 0, 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_items": 0, 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_items": 0, 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T20:09:45.729 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_items": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_bytes": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_items": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_bytes": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_items": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_items": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_bytes": 0, 2024-03-11T20:09:45.730 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_items": 0, 2024-03-11T20:09:45.731 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T20:09:45.731 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T20:09:45.731 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_bytes": 0, 2024-03-11T20:09:45.731 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_items": 0, 2024-03-11T20:09:45.731 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_bytes": 0, 2024-03-11T20:09:45.731 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_items": 0, 2024-03-11T20:09:45.731 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T20:09:45.731 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_items": 0, 2024-03-11T20:09:45.731 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T20:09:45.731 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_items": 0, 2024-03-11T20:09:45.731 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_bytes": 182099, 2024-03-11T20:09:45.731 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_items": 49, 2024-03-11T20:09:45.731 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_bytes": 0, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_items": 0, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bytes": 0, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_items": 0, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_bytes": 0, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_items": 0, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_bytes": 0, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_items": 0, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_bytes": 9616, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_items": 185, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_items": 0, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_bytes": 0, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_items": 0, 2024-03-11T20:09:45.732 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_bytes": 8320, 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_items": 13, 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_bytes": 0, 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_items": 0, 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_bytes": 0, 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_items": 0 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter": { 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "op_active": 0, 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "op_laggy": 0, 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send": 1618, 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send_bytes": 21, 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "op_resend": 3, 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "op_reply": 1618, 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "op_latency": { 2024-03-11T20:09:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1618, 2024-03-11T20:09:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 1.185974557, 2024-03-11T20:09:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000732987 2024-03-11T20:09:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: "op_inflight": 0, 2024-03-11T20:09:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: "oplen_avg": { 2024-03-11T20:09:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1618, 2024-03-11T20:09:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 1619 2024-03-11T20:09:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: "op": 1618, 2024-03-11T20:09:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: "op_r": 1618, 2024-03-11T20:09:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: "op_w": 0, 2024-03-11T20:09:45.735 INFO:teuthology.orchestra.run.smithi114.stdout: "op_rmw": 0, 2024-03-11T20:09:45.735 INFO:teuthology.orchestra.run.smithi114.stdout: "op_pg": 0, 2024-03-11T20:09:45.735 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_stat": 806, 2024-03-11T20:09:45.735 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_create": 0, 2024-03-11T20:09:45.735 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_read": 810, 2024-03-11T20:09:45.735 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_write": 0, 2024-03-11T20:09:45.735 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writefull": 0, 2024-03-11T20:09:45.735 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writesame": 0, 2024-03-11T20:09:45.735 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_append": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_zero": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_truncate": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_delete": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_mapext": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_sparse_read": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_clonerange": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_getxattr": 1, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_setxattr": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_cmpxattr": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_rmxattr": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_resetxattrs": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_call": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_watch": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_notify": 0, 2024-03-11T20:09:45.736 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T20:09:45.737 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls": 0, 2024-03-11T20:09:45.737 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls_filter": 0, 2024-03-11T20:09:45.737 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_other": 0, 2024-03-11T20:09:45.737 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_active": 0, 2024-03-11T20:09:45.737 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_send": 0, 2024-03-11T20:09:45.737 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_resend": 0, 2024-03-11T20:09:45.737 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_ping": 0, 2024-03-11T20:09:45.737 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_active": 0, 2024-03-11T20:09:45.737 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_send": 0, 2024-03-11T20:09:45.737 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_resend": 0, 2024-03-11T20:09:45.737 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_active": 0, 2024-03-11T20:09:45.737 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_send": 0, 2024-03-11T20:09:45.738 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_resend": 0, 2024-03-11T20:09:45.738 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_active": 0, 2024-03-11T20:09:45.738 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_send": 0, 2024-03-11T20:09:45.738 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_resend": 0, 2024-03-11T20:09:45.738 INFO:teuthology.orchestra.run.smithi114.stdout: "command_active": 0, 2024-03-11T20:09:45.738 INFO:teuthology.orchestra.run.smithi114.stdout: "command_send": 0, 2024-03-11T20:09:45.738 INFO:teuthology.orchestra.run.smithi114.stdout: "command_resend": 0, 2024-03-11T20:09:45.738 INFO:teuthology.orchestra.run.smithi114.stdout: "map_epoch": 102, 2024-03-11T20:09:45.738 INFO:teuthology.orchestra.run.smithi114.stdout: "map_full": 0, 2024-03-11T20:09:45.738 INFO:teuthology.orchestra.run.smithi114.stdout: "map_inc": 13, 2024-03-11T20:09:45.738 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_sessions": 4, 2024-03-11T20:09:45.738 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_open": 4, 2024-03-11T20:09:45.738 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_close": 0, 2024-03-11T20:09:45.739 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_laggy": 0, 2024-03-11T20:09:45.739 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_wr": 0, 2024-03-11T20:09:45.739 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_rd": 2, 2024-03-11T20:09:45.739 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_del": 0 2024-03-11T20:09:45.739 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.739 INFO:teuthology.orchestra.run.smithi114.stdout: "oft": { 2024-03-11T20:09:45.739 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_objs": 0, 2024-03-11T20:09:45.739 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_kv_pairs": 0, 2024-03-11T20:09:45.739 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_updates": 0, 2024-03-11T20:09:45.739 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_removes": 0 2024-03-11T20:09:45.739 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.739 INFO:teuthology.orchestra.run.smithi114.stdout: "osd-slow-ops": { 2024-03-11T20:09:45.740 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_ops_count": 0 2024-03-11T20:09:45.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.740 INFO:teuthology.orchestra.run.smithi114.stdout: "purge_queue": { 2024-03-11T20:09:45.740 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops": 0, 2024-03-11T20:09:45.740 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T20:09:45.740 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing": 0, 2024-03-11T20:09:45.740 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_high_water": 0, 2024-03-11T20:09:45.740 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed_ops": 0, 2024-03-11T20:09:45.740 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed": 0, 2024-03-11T20:09:45.740 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_item_in_journal": 0 2024-03-11T20:09:45.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.740 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T20:09:45.740 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:45.741 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-03-11T20:09:45.741 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:45.741 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 1886, 2024-03-11T20:09:45.741 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 1920314, 2024-03-11T20:09:45.741 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:45.741 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 1886, 2024-03-11T20:09:45.741 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-03-11T20:09:45.741 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-03-11T20:09:45.741 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 1886, 2024-03-11T20:09:45.741 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1920314, 2024-03-11T20:09:45.741 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_bytes": { 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 1618, 2024-03-11T20:09:45.742 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 593, 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 1618, 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 593, 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_ops": { 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 1024, 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-03-11T20:09:45.743 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 1618, 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 1618, 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 1618, 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1618, 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle": { 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-03-11T20:09:45.744 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 0, 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 0, 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle-0x55fc6dcb8960": { 2024-03-11T20:09:45.745 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 0, 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 0, 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:45.746 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:45.747 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:45.747 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-11T20:09:46.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:45 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi114 to 174.1M 2024-03-11T20:09:46.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:45 smithi097 ceph-mon[31898]: Unable to set osd_memory_target on smithi114 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-11T20:09:46.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:45 smithi097 ceph-mon[31898]: Adjusting osd_memory_target on smithi097 to 176.7M 2024-03-11T20:09:46.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:45 smithi097 ceph-mon[31898]: Unable to set osd_memory_target on smithi097 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-11T20:09:46.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:45 smithi097 ceph-mon[31898]: pgmap v325: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:46.164 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:45 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi114 to 174.1M 2024-03-11T20:09:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:45 smithi114 ceph-mon[36376]: Unable to set osd_memory_target on smithi114 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-11T20:09:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:45 smithi114 ceph-mon[36376]: Adjusting osd_memory_target on smithi097 to 176.7M 2024-03-11T20:09:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:45 smithi114 ceph-mon[36376]: Unable to set osd_memory_target on smithi097 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-11T20:09:46.165 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:45 smithi114 ceph-mon[36376]: pgmap v325: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:46.274 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:45 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi114 to 174.1M 2024-03-11T20:09:46.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:45 smithi161 ceph-mon[36332]: Unable to set osd_memory_target on smithi114 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-11T20:09:46.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:45 smithi161 ceph-mon[36332]: Adjusting osd_memory_target on smithi097 to 176.7M 2024-03-11T20:09:46.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:45 smithi161 ceph-mon[36332]: Unable to set osd_memory_target on smithi097 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-11T20:09:46.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:45 smithi161 ceph-mon[36332]: pgmap v325: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:46.492 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-11T20:09:46.492 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-11T20:09:46.492 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-03-11T20:09:46.492 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-11T20:09:46.492 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T20:09:46.492 DEBUG:tasks.check_counter:Getting stats from h 2024-03-11T20:09:46.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-11T20:09:46.731 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:09:47.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:47 smithi114 ceph-mon[36376]: pgmap v326: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:47.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:47 smithi097 ceph-mon[31898]: pgmap v326: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:47.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:47 smithi161 ceph-mon[36332]: pgmap v326: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:48.502 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-11T20:09:48.502 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T20:09:48.502 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 628, 2024-03-11T20:09:48.502 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 831, 2024-03-11T20:09:48.502 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 1727843, 2024-03-11T20:09:48.502 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 467864, 2024-03-11T20:09:48.502 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 9, 2024-03-11T20:09:48.502 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 2, 2024-03-11T20:09:48.502 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.121127018, 2024-03-11T20:09:48.502 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.048232326, 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.102585665, 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.010701168, 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 833, 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.071304100, 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000085599 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 828, 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000241756, 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000291 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 1628960, 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 147504 2024-03-11T20:09:48.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 72, 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 262, 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 25476, 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 570332, 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 12, 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 6, 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.018181001, 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.009368920, 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.007011317, 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000287718, 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 258, 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.034577201, 2024-03-11T20:09:48.504 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000134020 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 120, 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000028010, 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000233 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 2048, 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 544608 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 123, 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 127, 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 46526, 2024-03-11T20:09:48.505 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 93310, 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 13, 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 6, 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.014616921, 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.004529840, 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.010601236, 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000353394, 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 123, 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.029528557, 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000240069 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 184, 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000129479, 2024-03-11T20:09:48.506 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000703 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 1936, 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 1584 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: "cct": { 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: "total_workers": 1, 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: "unhealthy_workers": 0 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-MDSRank": { 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 56, 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.002129781, 2024-03-11T20:09:48.507 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000038031 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-PurgeQueue": { 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000007483, 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000003741 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 4, 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: "reply": 2, 2024-03-11T20:09:48.508 INFO:teuthology.orchestra.run.smithi114.stdout: "reply_latency": { 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000333590, 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000166795 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_reply": 0, 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "forward": 2, 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_complete": 0, 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_keys": 0, 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_commit": 11, 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_split": 0, 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_merge": 0, 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes": 10, 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_top": 0, 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_bottom": 10, 2024-03-11T20:09:48.509 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pin_tail": 0, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pinned": 10, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_expired": 0, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_with_caps": 1, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "subtrees": 1, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse": 4, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_hit": 2, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_forward": 0, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_discover": 0, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_dir_fetch": 0, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_remote_ino": 0, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_lock": 0, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "load_cent": 0, 2024-03-11T20:09:48.510 INFO:teuthology.orchestra.run.smithi114.stdout: "q": 0, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "exported": 0, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "exported_inodes": 0, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "imported": 0, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "imported_inodes": 0, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_dir_fetch": 0, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_backtrace_fetch": 0, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_peer_discover": 0, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rfiles": 4, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rbytes": 234, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rsnaps": 2, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_set_tag": 0, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_inotable_repaired": 0, 2024-03-11T20:09:48.511 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_inodes": 0, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_file_inodes": 0, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_inode_file_caps": 0, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_grant": 0, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps": 0, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps_dirty": 0, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_cap_release": 0, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "process_request_cap_release": 2 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.512 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache": { 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update": 0, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update_receipt": 0, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_try_discover": 0, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_send_discover": 1, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_handle_discover": 0, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays": 0, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_delayed": 0, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_enqueuing": 0, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_created": 0, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_enqueued": 0, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_reintegrated": 0, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_migrated": 0, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_processing": 0, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_enqueued": 0, 2024-03-11T20:09:48.513 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_prioritized": 0, 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_started": 0, 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_completed": 0, 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_exportdir": 0, 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_flush": 0, 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragmentdir": 0, 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragstats": 0, 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_inodestats": 0 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log": { 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "evlrg": 0, 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "evadd": 6, 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "evex": 0, 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "evtrm": 0, 2024-03-11T20:09:48.514 INFO:teuthology.orchestra.run.smithi114.stdout: "ev": 6, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "evexg": 0, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "evexd": 0, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "segadd": 2, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "segex": 0, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "segtrm": 0, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "seg": 2, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "segmjr": 2, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "segexg": 0, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "segexd": 0, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "expos": 4194304, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "wrpos": 4196125, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "rdpos": 0, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "jlat": { 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 5, 2024-03-11T20:09:48.515 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.024213671, 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.004842734 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: "replayed": 0 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_mem": { 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: "ino": 13, 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: "ino+": 13, 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: "ino-": 0, 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: "dir": 11, 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: "dir+": 12, 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: "dir-": 1, 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: "dn": 10, 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: "dn+": 10, 2024-03-11T20:09:48.516 INFO:teuthology.orchestra.run.smithi114.stdout: "dn-": 0, 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "cap": 1, 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "cap+": 2, 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "cap-": 1, 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "rss": 52336, 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "heap": 207228 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_server": { 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_client_request": 4, 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_server_request": 0, 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_request": 4, 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_session": 21, 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_peer_request": 0, 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "req_create_latency": { 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.517 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getattr_latency": { 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1, 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000248895, 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000248895 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getfilelock_latency": { 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: "req_link_latency": { 2024-03-11T20:09:48.518 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookup_latency": { 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookuphash_latency": { 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupino_latency": { 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1, 2024-03-11T20:09:48.519 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000084695, 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000084695 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupname_latency": { 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupparent_latency": { 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupsnap_latency": { 2024-03-11T20:09:48.520 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.521 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.521 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.521 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.521 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lssnap_latency": { 2024-03-11T20:09:48.521 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.521 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.521 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.521 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.521 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mkdir_latency": { 2024-03-11T20:09:48.521 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.521 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.521 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.522 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.522 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mknod_latency": { 2024-03-11T20:09:48.522 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.522 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.522 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.522 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.522 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mksnap_latency": { 2024-03-11T20:09:48.522 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: "req_open_latency": { 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: "req_readdir_latency": { 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rename_latency": { 2024-03-11T20:09:48.523 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: "req_renamesnap_latency": { 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: "req_snapdiff_latency": { 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.524 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmdir_latency": { 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmsnap_latency": { 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmxattr_latency": { 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.525 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setattr_latency": { 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setdirlayout_latency": { 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setfilelock_latency": { 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.526 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setlayout_latency": { 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setxattr_latency": { 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: "req_symlink_latency": { 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: "req_unlink_latency": { 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_revoke_eviction": 0, 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_acquisition_throttle": 0, 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getvxattr_latency": { 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_sessions": { 2024-03-11T20:09:48.528 INFO:teuthology.orchestra.run.smithi114.stdout: "session_count": 2, 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: "session_add": 3, 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: "session_remove": 1, 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_open": 2, 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_stale": 0, 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: "total_load": 1, 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: "average_load": 1, 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: "avg_session_uptime": 605, 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: "mdthresh_evicted": 0 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool": { 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_bytes": 0, 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_items": 0, 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T20:09:48.529 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_items": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_items": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_items": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_bytes": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_items": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_bytes": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_items": 0, 2024-03-11T20:09:48.530 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T20:09:48.531 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T20:09:48.531 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T20:09:48.531 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T20:09:48.531 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T20:09:48.531 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_items": 0, 2024-03-11T20:09:48.531 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_bytes": 0, 2024-03-11T20:09:48.531 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_items": 0, 2024-03-11T20:09:48.531 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T20:09:48.531 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T20:09:48.531 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_bytes": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_items": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_bytes": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_items": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_items": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_items": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_bytes": 264065, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_items": 94, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_bytes": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_items": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bytes": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_items": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_bytes": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_items": 0, 2024-03-11T20:09:48.532 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_bytes": 0, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_items": 0, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_bytes": 9616, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_items": 185, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_items": 0, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_bytes": 0, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_items": 0, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_bytes": 50416, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_items": 201, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_bytes": 0, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_items": 0, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_bytes": 0, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_items": 0 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.533 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter": { 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "op_active": 0, 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "op_laggy": 0, 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send": 42, 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send_bytes": 12213, 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "op_resend": 33, 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "op_reply": 42, 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "op_latency": { 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 42, 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.293993672, 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.006999849 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "op_inflight": 0, 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "oplen_avg": { 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 42, 2024-03-11T20:09:48.534 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 64 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "op": 42, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "op_r": 0, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "op_w": 42, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "op_rmw": 0, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "op_pg": 0, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_stat": 0, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_create": 10, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_read": 0, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_write": 5, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writefull": 8, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writesame": 0, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_append": 0, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_zero": 0, 2024-03-11T20:09:48.535 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_truncate": 0, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_delete": 6, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_mapext": 0, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_sparse_read": 0, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_clonerange": 0, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_getxattr": 0, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_setxattr": 20, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_cmpxattr": 0, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_rmxattr": 0, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_resetxattrs": 0, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_call": 0, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_watch": 0, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_notify": 0, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls": 0, 2024-03-11T20:09:48.536 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls_filter": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_other": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_active": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_send": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_resend": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_ping": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_active": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_send": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_resend": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_active": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_send": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_resend": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_active": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_send": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_resend": 0, 2024-03-11T20:09:48.537 INFO:teuthology.orchestra.run.smithi114.stdout: "command_active": 0, 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "command_send": 0, 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "command_resend": 0, 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "map_epoch": 102, 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "map_full": 0, 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "map_inc": 13, 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_sessions": 8, 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_open": 8, 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_close": 0, 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_laggy": 0, 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_wr": 14, 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_rd": 0, 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_del": 1 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "oft": { 2024-03-11T20:09:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_objs": 1, 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_kv_pairs": 0, 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_updates": 0, 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_removes": 0 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: "osd-slow-ops": { 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_ops_count": 0 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: "purge_queue": { 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops": 0, 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing": 0, 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_high_water": 0, 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed_ops": 0, 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed": 0, 2024-03-11T20:09:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_item_in_journal": 0 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 819, 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 1707527, 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 819, 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 819, 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1707527, 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_bytes": { 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 42, 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 12213, 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 42, 2024-03-11T20:09:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 12213, 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_ops": { 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 1024, 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:48.542 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 42, 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 42, 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 42, 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 42, 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle": { 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-03-11T20:09:48.543 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 0, 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 0, 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle-0x55c96ff606e0": { 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-03-11T20:09:48.544 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 6, 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 1821, 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 6, 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 5, 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1821, 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:48.545 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-11T20:09:49.228 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:49 smithi114 ceph-mon[36376]: pgmap v327: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:49.269 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-11T20:09:49.269 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-11T20:09:49.270 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=2 2024-03-11T20:09:49.270 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-11T20:09:49.270 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T20:09:49.270 DEBUG:tasks.check_counter:Getting stats from k 2024-03-11T20:09:49.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-11T20:09:49.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:49 smithi097 ceph-mon[31898]: pgmap v327: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:09:49.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:49 smithi161 ceph-mon[36332]: pgmap v327: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:51.188 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:51 smithi114 ceph-mon[36376]: pgmap v328: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:51.188 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-03-11T20:09:51.188 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T20:09:51.188 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 391, 2024-03-11T20:09:51.188 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 596, 2024-03-11T20:09:51.188 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 118049, 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 237748, 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 10, 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 5, 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.068154564, 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.031834127, 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.027999708, 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.004219077, 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 594, 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.041499523, 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000069864 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:51.189 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 942, 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000227868, 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000241 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 72, 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 126, 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 25802, 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 222703, 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 16, 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 3, 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.015230060, 2024-03-11T20:09:51.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.005319233, 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.008604117, 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000695136, 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 123, 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.031494957, 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000256056 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 114, 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000022589, 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000198 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 2000, 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 152800 2024-03-11T20:09:51.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 368, 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 479, 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 1646714, 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 491093, 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 7, 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 6, 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.064868664, 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.024242168, 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.044764623, 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.006380681, 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 479, 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.052122851, 2024-03-11T20:09:51.192 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000108815 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 93, 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000020139, 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000216 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 1622496, 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 463696 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: "cct": { 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: "total_workers": 1, 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: "unhealthy_workers": 0 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-MDSRank": { 2024-03-11T20:09:51.193 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 48, 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.001922791, 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000040058 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-PurgeQueue": { 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000007040, 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000003520 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.194 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 4, 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "reply": 3, 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "reply_latency": { 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 3, 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.001275861, 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000425287 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_reply": 0, 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "forward": 1, 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_complete": 0, 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_keys": 0, 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_commit": 11, 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_split": 0, 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_merge": 0, 2024-03-11T20:09:51.195 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes": 11, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_top": 0, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_bottom": 11, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pin_tail": 0, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pinned": 10, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_expired": 1, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_with_caps": 0, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "subtrees": 2, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse": 8, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_hit": 4, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_forward": 0, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_discover": 2, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_dir_fetch": 0, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_remote_ino": 0, 2024-03-11T20:09:51.196 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_lock": 0, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "load_cent": 0, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "q": 0, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "exported": 0, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "exported_inodes": 0, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "imported": 0, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "imported_inodes": 0, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_dir_fetch": 0, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_backtrace_fetch": 0, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_peer_discover": 1, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rfiles": 4, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rbytes": 234, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rsnaps": 2, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_set_tag": 0, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T20:09:51.197 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_inotable_repaired": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_inodes": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_file_inodes": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_inode_file_caps": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_grant": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps_dirty": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_cap_release": 0, 2024-03-11T20:09:51.198 INFO:teuthology.orchestra.run.smithi114.stdout: "process_request_cap_release": 0 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache": { 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update": 0, 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update_receipt": 0, 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_try_discover": 0, 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_send_discover": 3, 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_handle_discover": 0, 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays": 0, 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_delayed": 0, 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_enqueuing": 0, 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_created": 0, 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_enqueued": 0, 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_reintegrated": 0, 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_migrated": 0, 2024-03-11T20:09:51.199 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_processing": 0, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_enqueued": 0, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_prioritized": 0, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_started": 0, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_completed": 0, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_exportdir": 0, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_flush": 0, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragmentdir": 0, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragstats": 0, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_inodestats": 0 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log": { 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "evlrg": 0, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "evadd": 5, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "evex": 0, 2024-03-11T20:09:51.200 INFO:teuthology.orchestra.run.smithi114.stdout: "evtrm": 0, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "ev": 5, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "evexg": 0, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "evexd": 0, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "segadd": 2, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "segex": 0, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "segtrm": 0, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "seg": 2, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "segmjr": 2, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "segexg": 0, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "segexd": 0, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "expos": 4194304, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "wrpos": 4195734, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "rdpos": 0, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "jlat": { 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 4, 2024-03-11T20:09:51.201 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.010839828, 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.002709957 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: "replayed": 0 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_mem": { 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: "ino": 14, 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: "ino+": 14, 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: "ino-": 0, 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: "dir": 13, 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: "dir+": 14, 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: "dir-": 1, 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: "dn": 11, 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: "dn+": 12, 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: "dn-": 1, 2024-03-11T20:09:51.202 INFO:teuthology.orchestra.run.smithi114.stdout: "cap": 0, 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "cap+": 2, 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "cap-": 2, 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "rss": 49220, 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "heap": 207228 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_server": { 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_client_request": 6, 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_server_request": 0, 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_request": 4, 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_session": 13, 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_peer_request": 0, 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "req_create_latency": { 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.203 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getattr_latency": { 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1, 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000209930, 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000209930 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getfilelock_latency": { 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: "req_link_latency": { 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.205 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookup_latency": { 2024-03-11T20:09:51.205 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-03-11T20:09:51.205 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.001065931, 2024-03-11T20:09:51.205 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000532965 2024-03-11T20:09:51.205 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.205 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookuphash_latency": { 2024-03-11T20:09:51.205 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.205 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.205 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.205 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupino_latency": { 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupname_latency": { 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupparent_latency": { 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.206 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupsnap_latency": { 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lssnap_latency": { 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mkdir_latency": { 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.207 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.208 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.208 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mknod_latency": { 2024-03-11T20:09:51.208 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.208 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.208 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.208 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.208 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mksnap_latency": { 2024-03-11T20:09:51.208 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.208 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: "req_open_latency": { 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: "req_readdir_latency": { 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rename_latency": { 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.209 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: "req_renamesnap_latency": { 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: "req_snapdiff_latency": { 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmdir_latency": { 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.210 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmsnap_latency": { 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmxattr_latency": { 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setattr_latency": { 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setdirlayout_latency": { 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setfilelock_latency": { 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setlayout_latency": { 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.212 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setxattr_latency": { 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: "req_symlink_latency": { 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: "req_unlink_latency": { 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.213 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_revoke_eviction": 0, 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_acquisition_throttle": 0, 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getvxattr_latency": { 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_sessions": { 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: "session_count": 1, 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: "session_add": 2, 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: "session_remove": 1, 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_open": 1, 2024-03-11T20:09:51.214 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_stale": 0, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "total_load": 2, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "average_load": 2, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "avg_session_uptime": 630, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "mdthresh_evicted": 0 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool": { 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_bytes": 0, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_items": 0, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_items": 0, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_items": 0, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T20:09:51.215 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_items": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_bytes": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_items": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_bytes": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_items": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_items": 0, 2024-03-11T20:09:51.216 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_bytes": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_items": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_bytes": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_items": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_bytes": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_items": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_items": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_items": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_bytes": 272135, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_items": 101, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_bytes": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_items": 0, 2024-03-11T20:09:51.217 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bytes": 0, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_items": 0, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_bytes": 0, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_items": 0, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_bytes": 0, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_items": 0, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_bytes": 9616, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_items": 185, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_items": 0, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_bytes": 0, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_items": 0, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_bytes": 55648, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_items": 224, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_bytes": 0, 2024-03-11T20:09:51.218 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_items": 0, 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_bytes": 0, 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_items": 0 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter": { 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "op_active": 0, 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "op_laggy": 0, 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send": 42, 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send_bytes": 11877, 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "op_resend": 33, 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "op_reply": 42, 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "op_latency": { 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 42, 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.287993799, 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.006856995 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.219 INFO:teuthology.orchestra.run.smithi114.stdout: "op_inflight": 0, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "oplen_avg": { 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 42, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 66 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "op": 42, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "op_r": 0, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "op_w": 42, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "op_rmw": 0, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "op_pg": 0, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_stat": 0, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_create": 10, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_read": 0, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_write": 4, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writefull": 7, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writesame": 0, 2024-03-11T20:09:51.220 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_append": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_zero": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_truncate": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_delete": 6, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_mapext": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_sparse_read": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_clonerange": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_getxattr": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_setxattr": 20, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_cmpxattr": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_rmxattr": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_resetxattrs": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_call": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_watch": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_notify": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T20:09:51.221 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls_filter": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_other": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_active": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_send": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_resend": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_ping": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_active": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_send": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_resend": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_active": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_send": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_resend": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_active": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_send": 0, 2024-03-11T20:09:51.222 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_resend": 0, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "command_active": 0, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "command_send": 0, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "command_resend": 0, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "map_epoch": 102, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "map_full": 0, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "map_inc": 13, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_sessions": 10, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_open": 10, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_close": 0, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_laggy": 0, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_wr": 17, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_rd": 0, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_del": 2 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "oft": { 2024-03-11T20:09:51.223 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_objs": 1, 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_kv_pairs": 0, 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_updates": 0, 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_removes": 0 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: "osd-slow-ops": { 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_ops_count": 0 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: "purge_queue": { 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops": 0, 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing": 0, 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_high_water": 0, 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed_ops": 0, 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed": 0, 2024-03-11T20:09:51.224 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_item_in_journal": 0 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 827, 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 1697691, 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 827, 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 827, 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1697691, 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:51.225 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_bytes": { 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 42, 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 11877, 2024-03-11T20:09:51.226 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 42, 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 11877, 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_ops": { 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 1024, 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:51.227 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 42, 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 42, 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 42, 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 42, 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle": { 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-03-11T20:09:51.228 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 0, 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 0, 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle-0x5561855846e0": { 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-03-11T20:09:51.229 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 5, 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 1430, 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 5, 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 4, 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1430, 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-03-11T20:09:51.230 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-03-11T20:09:51.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:51 smithi097 ceph-mon[31898]: pgmap v328: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:51.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:51 smithi161 ceph-mon[36332]: pgmap v328: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:51.954 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-11T20:09:51.954 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-11T20:09:51.954 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=2 2024-03-11T20:09:51.954 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-03-11T20:09:51.954 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T20:09:51.954 DEBUG:tasks.check_counter:Getting stats from c 2024-03-11T20:09:51.954 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-11T20:09:52.215 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T20:09:53.379 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:53 smithi161 ceph-mon[36332]: pgmap v329: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:53.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:53 smithi097 ceph-mon[31898]: pgmap v329: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:53.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:53 smithi114 ceph-mon[36376]: pgmap v329: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:54.097 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-11T20:09:54.097 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T20:09:54.097 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 1749, 2024-03-11T20:09:54.097 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 1751, 2024-03-11T20:09:54.097 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 1587513, 2024-03-11T20:09:54.097 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 543949, 2024-03-11T20:09:54.097 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 6, 2024-03-11T20:09:54.097 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 2, 2024-03-11T20:09:54.097 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.170171252, 2024-03-11T20:09:54.097 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.054067586, 2024-03-11T20:09:54.097 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.045381681, 2024-03-11T20:09:54.097 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.054445022, 2024-03-11T20:09:54.097 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1749, 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.066800106, 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000038193 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T20:09:54.098 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 11, 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 186, 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 5128, 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 543166, 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 8, 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 5, 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.012051199, 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.007117617, 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.002877167, 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.000131673, 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 184, 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.024146407, 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000131230 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.099 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 541856 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 1082, 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 1067, 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 1819383, 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 396817, 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 3, 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 2, 2024-03-11T20:09:54.100 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.127845311, 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.043738991, 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.054158171, 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.032541353, 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1066, 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.041484053, 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000038915 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 1618576, 2024-03-11T20:09:54.101 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 143040 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: "cct": { 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: "total_workers": 1, 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: "unhealthy_workers": 0 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-MDSRank": { 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 4247, 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.121712224, 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000028658 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.102 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-PurgeQueue": { 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": { 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: "request": 0, 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: "reply": 0, 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: "reply_latency": { 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.104 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_reply": 0, 2024-03-11T20:09:54.104 INFO:teuthology.orchestra.run.smithi161.stdout: "forward": 0, 2024-03-11T20:09:54.104 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_complete": 0, 2024-03-11T20:09:54.104 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_keys": 0, 2024-03-11T20:09:54.104 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_commit": 0, 2024-03-11T20:09:54.104 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_split": 0, 2024-03-11T20:09:54.104 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_merge": 0, 2024-03-11T20:09:54.104 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes": 31, 2024-03-11T20:09:54.104 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_top": 8, 2024-03-11T20:09:54.104 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_bottom": 23, 2024-03-11T20:09:54.104 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pin_tail": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pinned": 20, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_expired": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_with_caps": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "subtrees": 2, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_hit": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_forward": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_discover": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_dir_fetch": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_remote_ino": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_lock": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "load_cent": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "q": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "exported": 0, 2024-03-11T20:09:54.105 INFO:teuthology.orchestra.run.smithi161.stdout: "exported_inodes": 0, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "imported": 0, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "imported_inodes": 0, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_dir_fetch": 0, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_backtrace_fetch": 0, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_peer_discover": 0, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rfiles": 4, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rbytes": 234, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rsnaps": 6, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_set_tag": 0, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_inotable_repaired": 0, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_inodes": 0, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T20:09:54.106 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_file_inodes": 0, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_inode_file_caps": 0, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_grant": 0, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps": 0, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps_dirty": 0, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_cap_release": 0, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "process_request_cap_release": 0 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_cache": { 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update": 0, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update_receipt": 0, 2024-03-11T20:09:54.107 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_try_discover": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_send_discover": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_handle_discover": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_delayed": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_enqueuing": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_created": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_enqueued": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_reintegrated": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_migrated": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_processing": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_enqueued": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_prioritized": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_started": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_completed": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T20:09:54.108 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_exportdir": 0, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_flush": 0, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragmentdir": 0, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragstats": 0, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_inodestats": 0 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_log": { 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "evlrg": 0, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "evadd": 0, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "evex": 0, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "evtrm": 0, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "ev": 258, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "evexg": 0, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "evexd": 0, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "segadd": 0, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "segex": 0, 2024-03-11T20:09:54.109 INFO:teuthology.orchestra.run.smithi161.stdout: "segtrm": 0, 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: "seg": 2, 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: "segmjr": 12, 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: "segexg": 0, 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: "segexd": 0, 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: "expos": 5075625, 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: "wrpos": 5264552, 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: "rdpos": 5264416, 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: "jlat": { 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: "replayed": 258 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.110 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_mem": { 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "ino": 20, 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "ino+": 24, 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "ino-": 4, 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "dir": 11, 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "dir+": 11, 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "dir-": 0, 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "dn": 31, 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "dn+": 31, 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "dn-": 0, 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "cap": 0, 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "cap+": 0, 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "cap-": 0, 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "rss": 49972, 2024-03-11T20:09:54.111 INFO:teuthology.orchestra.run.smithi161.stdout: "heap": 215420 2024-03-11T20:09:54.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.112 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_server": { 2024-03-11T20:09:54.112 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_client_request": 0, 2024-03-11T20:09:54.112 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_server_request": 0, 2024-03-11T20:09:54.112 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_request": 0, 2024-03-11T20:09:54.112 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_session": 0, 2024-03-11T20:09:54.112 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_peer_request": 0, 2024-03-11T20:09:54.112 INFO:teuthology.orchestra.run.smithi161.stdout: "req_create_latency": { 2024-03-11T20:09:54.112 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.112 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.112 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.112 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getattr_latency": { 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getfilelock_latency": { 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: "req_link_latency": { 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookup_latency": { 2024-03-11T20:09:54.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookuphash_latency": { 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupino_latency": { 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupname_latency": { 2024-03-11T20:09:54.114 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupparent_latency": { 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupsnap_latency": { 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lssnap_latency": { 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.115 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.116 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.116 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mkdir_latency": { 2024-03-11T20:09:54.116 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.116 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.116 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mknod_latency": { 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mksnap_latency": { 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: "req_open_latency": { 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.117 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: "req_readdir_latency": { 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rename_latency": { 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: "req_renamesnap_latency": { 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.118 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: "req_snapdiff_latency": { 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmdir_latency": { 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmsnap_latency": { 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.119 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmxattr_latency": { 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setattr_latency": { 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setdirlayout_latency": { 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.120 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setfilelock_latency": { 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setlayout_latency": { 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setxattr_latency": { 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.121 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: "req_symlink_latency": { 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: "req_unlink_latency": { 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_revoke_eviction": 0, 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_acquisition_throttle": 0, 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getvxattr_latency": { 2024-03-11T20:09:54.122 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_sessions": { 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: "session_count": 2, 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: "session_add": 6, 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: "session_remove": 4, 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_open": 4, 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_stale": 0, 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: "total_load": 0, 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: "average_load": 0, 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: "avg_session_uptime": 4012, 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: "mdthresh_evicted": 0 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.123 INFO:teuthology.orchestra.run.smithi161.stdout: "mempool": { 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_bytes": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_items": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_items": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_items": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_items": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T20:09:54.124 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_bytes": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_items": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_bytes": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_items": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_items": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_bytes": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_items": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_bytes": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_items": 0, 2024-03-11T20:09:54.125 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_bytes": 0, 2024-03-11T20:09:54.126 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_items": 0, 2024-03-11T20:09:54.126 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T20:09:54.126 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_items": 0, 2024-03-11T20:09:54.126 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T20:09:54.126 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_items": 0, 2024-03-11T20:09:54.126 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_bytes": 190472, 2024-03-11T20:09:54.126 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_items": 61, 2024-03-11T20:09:54.126 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_bytes": 0, 2024-03-11T20:09:54.126 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_items": 0, 2024-03-11T20:09:54.126 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_bytes": 0, 2024-03-11T20:09:54.126 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_items": 0, 2024-03-11T20:09:54.127 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_bytes": 0, 2024-03-11T20:09:54.127 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_items": 0, 2024-03-11T20:09:54.127 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_bytes": 0, 2024-03-11T20:09:54.127 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_items": 0, 2024-03-11T20:09:54.127 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_bytes": 9616, 2024-03-11T20:09:54.127 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_items": 185, 2024-03-11T20:09:54.127 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T20:09:54.127 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_items": 0, 2024-03-11T20:09:54.127 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_bytes": 0, 2024-03-11T20:09:54.127 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_items": 0, 2024-03-11T20:09:54.127 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_bytes": 91205, 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_items": 759, 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_bytes": 0, 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_items": 0, 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_bytes": 0, 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_items": 0 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "objecter": { 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "op_active": 0, 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "op_laggy": 0, 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send": 2569, 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send_bytes": 26, 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "op_resend": 4, 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "op_reply": 2569, 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "op_latency": { 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2569, 2024-03-11T20:09:54.128 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 1.245974324, 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000485003 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "op_inflight": 0, 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "oplen_avg": { 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2569, 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 2570 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "op": 2569, 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "op_r": 2569, 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "op_w": 0, 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "op_rmw": 0, 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "op_pg": 0, 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_stat": 1630, 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_create": 0, 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_read": 936, 2024-03-11T20:09:54.129 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_write": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writefull": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writesame": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_append": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_zero": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_truncate": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_delete": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_mapext": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_sparse_read": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_clonerange": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_getxattr": 2, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_setxattr": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_cmpxattr": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_rmxattr": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_resetxattrs": 0, 2024-03-11T20:09:54.130 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_call": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_watch": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_notify": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls_filter": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_other": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_active": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_send": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_resend": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_ping": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_active": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_send": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_resend": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_active": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_send": 0, 2024-03-11T20:09:54.131 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_resend": 0, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_active": 0, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_send": 0, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_resend": 0, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "command_active": 0, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "command_send": 0, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "command_resend": 0, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "map_epoch": 102, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "map_full": 0, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "map_inc": 13, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_sessions": 5, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_open": 5, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_close": 0, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_laggy": 0, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_wr": 0, 2024-03-11T20:09:54.132 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_rd": 2, 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_del": 0 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: "oft": { 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_objs": 0, 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_kv_pairs": 0, 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_updates": 0, 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_removes": 0 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: "osd-slow-ops": { 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_ops_count": 0 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: "purge_queue": { 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops": 0, 2024-03-11T20:09:54.133 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing": 0, 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_high_water": 0, 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed_ops": 0, 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed": 0, 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_item_in_journal": 0 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 2838, 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 3139963, 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 2838, 2024-03-11T20:09:54.134 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-03-11T20:09:54.135 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-03-11T20:09:54.135 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2838, 2024-03-11T20:09:54.135 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 3139963, 2024-03-11T20:09:54.135 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:54.135 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.135 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.135 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.135 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:54.135 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.135 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_bytes": { 2024-03-11T20:09:54.135 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:54.135 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-03-11T20:09:54.135 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 2569, 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 1070258, 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2569, 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 1070258, 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.136 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_ops": { 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 1024, 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 2569, 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 2569, 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2569, 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 2569, 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.137 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle": { 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:54.138 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle-0x56429b8e8960": { 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-03-11T20:09:54.139 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-03-11T20:09:54.140 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:54.140 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:54.140 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:54.140 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:54.140 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:54.140 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:54.140 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-11T20:09:54.796 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-11T20:09:54.796 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-11T20:09:54.797 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-03-11T20:09:54.797 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-11T20:09:54.797 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T20:09:54.797 DEBUG:tasks.check_counter:Getting stats from f 2024-03-11T20:09:54.797 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-11T20:09:55.043 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T20:09:55.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:55 smithi097 ceph-mon[31898]: pgmap v330: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:55.283 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:55 smithi161 ceph-mon[36332]: pgmap v330: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:55.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:55 smithi114 ceph-mon[36376]: pgmap v330: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:56.645 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-11T20:09:56.645 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T20:09:56.645 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 1088, 2024-03-11T20:09:56.645 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 1071, 2024-03-11T20:09:56.645 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 1896464, 2024-03-11T20:09:56.645 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 398663, 2024-03-11T20:09:56.645 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 5, 2024-03-11T20:09:56.645 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 1, 2024-03-11T20:09:56.645 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.157828594, 2024-03-11T20:09:56.646 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.047072677, 2024-03-11T20:09:56.646 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.068906519, 2024-03-11T20:09:56.646 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.041491588, 2024-03-11T20:09:56.646 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:56.646 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1071, 2024-03-11T20:09:56.646 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.057097505, 2024-03-11T20:09:56.646 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000053312 2024-03-11T20:09:56.646 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.647 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:56.647 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.647 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.647 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.647 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 1621952, 2024-03-11T20:09:56.647 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 144576 2024-03-11T20:09:56.647 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.647 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T20:09:56.647 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 4, 2024-03-11T20:09:56.647 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 181, 2024-03-11T20:09:56.647 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 876, 2024-03-11T20:09:56.647 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 464300, 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 6, 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 4, 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.010970012, 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.007180164, 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.001736529, 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.000082109, 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 178, 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.019024110, 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000106877 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.648 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 463616 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 1557, 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 1559, 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 399624, 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 484234, 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 4, 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 3, 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.113434212, 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.037362147, 2024-03-11T20:09:56.649 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.028151854, 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.039800306, 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1557, 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.061217387, 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000039317 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.650 INFO:teuthology.orchestra.run.smithi161.stdout: "cct": { 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: "total_workers": 1, 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: "unhealthy_workers": 0 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-MDSRank": { 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 4014, 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.117495998, 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000029271 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-PurgeQueue": { 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-03-11T20:09:56.651 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": { 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: "request": 0, 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: "reply": 0, 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: "reply_latency": { 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_reply": 0, 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: "forward": 0, 2024-03-11T20:09:56.652 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_complete": 0, 2024-03-11T20:09:56.653 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_keys": 0, 2024-03-11T20:09:56.653 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_commit": 0, 2024-03-11T20:09:56.653 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_split": 0, 2024-03-11T20:09:56.653 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_merge": 0, 2024-03-11T20:09:56.653 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes": 0, 2024-03-11T20:09:56.653 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_top": 0, 2024-03-11T20:09:56.653 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_bottom": 0, 2024-03-11T20:09:56.653 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pin_tail": 0, 2024-03-11T20:09:56.654 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pinned": 0, 2024-03-11T20:09:56.654 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_expired": 0, 2024-03-11T20:09:56.654 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_with_caps": 0, 2024-03-11T20:09:56.654 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": 0, 2024-03-11T20:09:56.654 INFO:teuthology.orchestra.run.smithi161.stdout: "subtrees": 1, 2024-03-11T20:09:56.654 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse": 0, 2024-03-11T20:09:56.654 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_hit": 0, 2024-03-11T20:09:56.654 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_forward": 0, 2024-03-11T20:09:56.654 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_discover": 0, 2024-03-11T20:09:56.654 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_dir_fetch": 0, 2024-03-11T20:09:56.654 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_remote_ino": 0, 2024-03-11T20:09:56.654 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_lock": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "load_cent": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "q": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "exported": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "exported_inodes": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "imported": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "imported_inodes": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_dir_fetch": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_backtrace_fetch": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_peer_discover": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rfiles": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rbytes": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rsnaps": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_set_tag": 0, 2024-03-11T20:09:56.655 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T20:09:56.656 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_inotable_repaired": 0, 2024-03-11T20:09:56.656 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_inodes": 0, 2024-03-11T20:09:56.656 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T20:09:56.656 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T20:09:56.656 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_file_inodes": 0, 2024-03-11T20:09:56.656 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_inode_file_caps": 0, 2024-03-11T20:09:56.656 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T20:09:56.656 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_grant": 0, 2024-03-11T20:09:56.656 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T20:09:56.656 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T20:09:56.656 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T20:09:56.657 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps": 0, 2024-03-11T20:09:56.657 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps_dirty": 0, 2024-03-11T20:09:56.657 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_cap_release": 0, 2024-03-11T20:09:56.657 INFO:teuthology.orchestra.run.smithi161.stdout: "process_request_cap_release": 0 2024-03-11T20:09:56.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.657 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_cache": { 2024-03-11T20:09:56.657 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update": 0, 2024-03-11T20:09:56.657 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update_receipt": 0, 2024-03-11T20:09:56.657 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_try_discover": 0, 2024-03-11T20:09:56.657 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_send_discover": 0, 2024-03-11T20:09:56.657 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_handle_discover": 0, 2024-03-11T20:09:56.657 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_delayed": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_enqueuing": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_created": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_enqueued": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_reintegrated": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_migrated": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_processing": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_enqueued": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_prioritized": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_started": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_completed": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_exportdir": 0, 2024-03-11T20:09:56.658 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_flush": 0, 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragmentdir": 0, 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragstats": 0, 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_inodestats": 0 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_log": { 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "evlrg": 0, 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "evadd": 0, 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "evex": 0, 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "evtrm": 0, 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "ev": 10, 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "evexg": 0, 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "evexd": 0, 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "segadd": 0, 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "segex": 0, 2024-03-11T20:09:56.659 INFO:teuthology.orchestra.run.smithi161.stdout: "segtrm": 0, 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: "seg": 2, 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: "segmjr": 2, 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: "segexg": 0, 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: "segexd": 0, 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: "expos": 4194304, 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: "wrpos": 4197189, 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: "rdpos": 4197048, 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: "jlat": { 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: "replayed": 10 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.660 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_mem": { 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "ino": 3, 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "ino+": 3, 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "ino-": 0, 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "dir": 1, 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "dir+": 1, 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "dir-": 0, 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "dn": 0, 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "dn+": 0, 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "dn-": 0, 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "cap": 0, 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "cap+": 0, 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "cap-": 0, 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "rss": 50476, 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: "heap": 215420 2024-03-11T20:09:56.661 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_server": { 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_client_request": 0, 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_server_request": 0, 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_request": 0, 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_session": 0, 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_peer_request": 0, 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "req_create_latency": { 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getattr_latency": { 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.662 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getfilelock_latency": { 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: "req_link_latency": { 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookup_latency": { 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.663 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookuphash_latency": { 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupino_latency": { 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupname_latency": { 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.664 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupparent_latency": { 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupsnap_latency": { 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lssnap_latency": { 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.665 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mkdir_latency": { 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mknod_latency": { 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mksnap_latency": { 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.666 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.667 INFO:teuthology.orchestra.run.smithi161.stdout: "req_open_latency": { 2024-03-11T20:09:56.667 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.667 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.667 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.667 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.667 INFO:teuthology.orchestra.run.smithi161.stdout: "req_readdir_latency": { 2024-03-11T20:09:56.667 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.667 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rename_latency": { 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: "req_renamesnap_latency": { 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: "req_snapdiff_latency": { 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.668 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmdir_latency": { 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmsnap_latency": { 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmxattr_latency": { 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.669 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setattr_latency": { 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setdirlayout_latency": { 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setfilelock_latency": { 2024-03-11T20:09:56.670 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setlayout_latency": { 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setxattr_latency": { 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: "req_symlink_latency": { 2024-03-11T20:09:56.671 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: "req_unlink_latency": { 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_revoke_eviction": 0, 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_acquisition_throttle": 0, 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getvxattr_latency": { 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.672 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_sessions": { 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: "session_count": 2, 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: "session_add": 5, 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: "session_remove": 3, 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_open": 4, 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_stale": 0, 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: "total_load": 0, 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: "average_load": 0, 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: "avg_session_uptime": 4014, 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: "mdthresh_evicted": 0 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: "mempool": { 2024-03-11T20:09:56.673 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_bytes": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_items": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_items": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_items": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_items": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T20:09:56.674 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_bytes": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_items": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_bytes": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_items": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_items": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_bytes": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_items": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T20:09:56.675 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_bytes": 0, 2024-03-11T20:09:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_items": 0, 2024-03-11T20:09:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_bytes": 0, 2024-03-11T20:09:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_items": 0, 2024-03-11T20:09:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T20:09:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_items": 0, 2024-03-11T20:09:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T20:09:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_items": 0, 2024-03-11T20:09:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_bytes": 182076, 2024-03-11T20:09:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_items": 49, 2024-03-11T20:09:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_bytes": 0, 2024-03-11T20:09:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_items": 0, 2024-03-11T20:09:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_bytes": 0, 2024-03-11T20:09:56.676 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_items": 0, 2024-03-11T20:09:56.677 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_bytes": 0, 2024-03-11T20:09:56.677 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_items": 0, 2024-03-11T20:09:56.677 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_bytes": 0, 2024-03-11T20:09:56.677 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_items": 0, 2024-03-11T20:09:56.677 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_bytes": 9616, 2024-03-11T20:09:56.677 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_items": 185, 2024-03-11T20:09:56.677 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T20:09:56.677 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_items": 0, 2024-03-11T20:09:56.677 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_bytes": 0, 2024-03-11T20:09:56.677 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_items": 0, 2024-03-11T20:09:56.677 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_bytes": 8320, 2024-03-11T20:09:56.677 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_items": 13, 2024-03-11T20:09:56.677 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_bytes": 0, 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_items": 0, 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_bytes": 0, 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_items": 0 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "objecter": { 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "op_active": 0, 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "op_laggy": 0, 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send": 2376, 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send_bytes": 21, 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "op_resend": 3, 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "op_reply": 2376, 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "op_latency": { 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2376, 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 1.249974214, 2024-03-11T20:09:56.678 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000526083 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: "op_inflight": 0, 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: "oplen_avg": { 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2376, 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 2377 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: "op": 2376, 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: "op_r": 2376, 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: "op_w": 0, 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: "op_rmw": 0, 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: "op_pg": 0, 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_stat": 1548, 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_create": 0, 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_read": 826, 2024-03-11T20:09:56.679 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_write": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writefull": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writesame": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_append": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_zero": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_truncate": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_delete": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_mapext": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_sparse_read": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_clonerange": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_getxattr": 1, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_setxattr": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_cmpxattr": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_rmxattr": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_resetxattrs": 0, 2024-03-11T20:09:56.680 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_call": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_watch": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_notify": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls_filter": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_other": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_active": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_send": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_resend": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_ping": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_active": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_send": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_resend": 0, 2024-03-11T20:09:56.681 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_active": 0, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_send": 0, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_resend": 0, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_active": 0, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_send": 0, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_resend": 0, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "command_active": 0, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "command_send": 0, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "command_resend": 0, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "map_epoch": 102, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "map_full": 0, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "map_inc": 13, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_sessions": 4, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_open": 4, 2024-03-11T20:09:56.682 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_close": 0, 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_laggy": 0, 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_wr": 0, 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_rd": 2, 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_del": 0 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: "oft": { 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_objs": 0, 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_kv_pairs": 0, 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_updates": 0, 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_removes": 0 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: "osd-slow-ops": { 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_ops_count": 0 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.683 INFO:teuthology.orchestra.run.smithi161.stdout: "purge_queue": { 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops": 0, 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing": 0, 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_high_water": 0, 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed_ops": 0, 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed": 0, 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_item_in_journal": 0 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 2645, 2024-03-11T20:09:56.684 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 2042258, 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 2645, 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2645, 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 2042258, 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_bytes": { 2024-03-11T20:09:56.685 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:56.686 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-03-11T20:09:56.686 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:56.686 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:09:56.686 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:09:56.686 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:56.686 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:56.686 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 2376, 2024-03-11T20:09:56.686 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 2890, 2024-03-11T20:09:56.686 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2376, 2024-03-11T20:09:56.686 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 2890, 2024-03-11T20:09:56.686 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:56.686 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.686 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.687 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.687 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:56.687 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.687 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_ops": { 2024-03-11T20:09:56.687 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:56.687 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 1024, 2024-03-11T20:09:56.688 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:56.688 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:09:56.688 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:09:56.688 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:56.688 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:56.688 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 2376, 2024-03-11T20:09:56.688 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 2376, 2024-03-11T20:09:56.688 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2376, 2024-03-11T20:09:56.688 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 2376, 2024-03-11T20:09:56.688 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:56.688 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.689 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.689 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.689 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:56.689 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.689 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle": { 2024-03-11T20:09:56.689 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:56.689 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-03-11T20:09:56.689 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:56.689 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:09:56.689 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:09:56.690 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:56.690 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:56.690 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-03-11T20:09:56.690 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-03-11T20:09:56.690 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-03-11T20:09:56.690 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-03-11T20:09:56.690 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:56.690 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.690 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.690 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.690 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:56.691 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:56.691 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle-0x55a9b7b42960": { 2024-03-11T20:09:56.691 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:56.691 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-03-11T20:09:56.691 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:56.691 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:09:56.691 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:09:56.691 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:56.691 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:56.691 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-03-11T20:09:56.691 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-03-11T20:09:56.691 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-03-11T20:09:56.692 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-03-11T20:09:56.692 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:56.692 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:56.692 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:56.692 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:56.692 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:56.692 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:56.692 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-11T20:09:57.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:57 smithi097 ceph-mon[31898]: pgmap v331: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:57.443 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:57 smithi161 ceph-mon[36332]: pgmap v331: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:57.468 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-11T20:09:57.468 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-11T20:09:57.468 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-11T20:09:57.468 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-11T20:09:57.468 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T20:09:57.469 DEBUG:tasks.check_counter:Getting stats from i 2024-03-11T20:09:57.469 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-11T20:09:57.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:57 smithi114 ceph-mon[36376]: pgmap v331: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:09:57.708 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T20:09:59.187 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:09:59 smithi161 ceph-mon[36332]: pgmap v332: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:59.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:09:59 smithi097 ceph-mon[31898]: pgmap v332: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:59.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:09:59 smithi114 ceph-mon[36376]: pgmap v332: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-11T20:09:59.479 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-11T20:09:59.479 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T20:09:59.479 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 368, 2024-03-11T20:09:59.479 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 348, 2024-03-11T20:09:59.479 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 1658359, 2024-03-11T20:09:59.479 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 213792, 2024-03-11T20:09:59.479 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 12, 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 5, 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.058096153, 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.018501612, 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.042332721, 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.006200079, 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 348, 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.028519581, 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000081952 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 95, 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000015193, 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000159 2024-03-11T20:09:59.480 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 1634400, 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 148752 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 67, 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 244, 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 24695, 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 575263, 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 11, 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 7, 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.016273585, 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.008358041, 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.005946640, 2024-03-11T20:09:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.000358436, 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 240, 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.041803172, 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000174179 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 95, 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000019321, 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000203 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 480, 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 542832 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T20:09:59.482 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 365, 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 567, 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 110011, 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 112873, 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 7, 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 5, 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.055039503, 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.022603211, 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.025039457, 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.004890007, 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 565, 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.044832102, 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000079348 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:09:59.483 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 901, 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000272807, 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000302 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: "cct": { 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: "total_workers": 1, 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: "unhealthy_workers": 0 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-MDSRank": { 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 44, 2024-03-11T20:09:59.484 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.001322970, 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000030067 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-PurgeQueue": { 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2, 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000008660, 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000004330 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": { 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: "request": 0, 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: "reply": 0, 2024-03-11T20:09:59.485 INFO:teuthology.orchestra.run.smithi161.stdout: "reply_latency": { 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_reply": 0, 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "forward": 0, 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_complete": 0, 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_keys": 0, 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_commit": 11, 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_split": 0, 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_merge": 0, 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes": 10, 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_top": 0, 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_bottom": 10, 2024-03-11T20:09:59.486 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pin_tail": 0, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pinned": 10, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_expired": 0, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_with_caps": 0, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": 0, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "subtrees": 1, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse": 0, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_hit": 0, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_forward": 0, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_discover": 0, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_dir_fetch": 0, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_remote_ino": 0, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_lock": 0, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "load_cent": 0, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "q": 0, 2024-03-11T20:09:59.487 INFO:teuthology.orchestra.run.smithi161.stdout: "exported": 0, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "exported_inodes": 0, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "imported": 0, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "imported_inodes": 0, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_dir_fetch": 0, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_backtrace_fetch": 0, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_peer_discover": 0, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rfiles": 4, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rbytes": 234, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rsnaps": 2, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_set_tag": 0, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_inotable_repaired": 0, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_inodes": 0, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T20:09:59.488 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_file_inodes": 0, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_inode_file_caps": 0, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_grant": 0, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps": 0, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps_dirty": 0, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_cap_release": 0, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "process_request_cap_release": 0 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_cache": { 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update": 0, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update_receipt": 0, 2024-03-11T20:09:59.489 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_try_discover": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_send_discover": 1, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_handle_discover": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_delayed": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_enqueuing": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_created": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_enqueued": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_reintegrated": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_migrated": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_processing": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_enqueued": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_prioritized": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_started": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_completed": 0, 2024-03-11T20:09:59.490 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_exportdir": 0, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_flush": 0, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragmentdir": 0, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragstats": 0, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_inodestats": 0 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_log": { 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "evlrg": 0, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "evadd": 2, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "evex": 0, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "evtrm": 0, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "ev": 2, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "evexg": 0, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "evexd": 0, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "segadd": 2, 2024-03-11T20:09:59.491 INFO:teuthology.orchestra.run.smithi161.stdout: "segex": 0, 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: "segtrm": 0, 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: "seg": 2, 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: "segmjr": 2, 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: "segexg": 0, 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: "segexd": 0, 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: "expos": 4194304, 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: "wrpos": 4194892, 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: "rdpos": 0, 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: "jlat": { 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1, 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.001463810, 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.001463810 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: "replayed": 0 2024-03-11T20:09:59.492 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_mem": { 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "ino": 13, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "ino+": 13, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "ino-": 0, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "dir": 11, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "dir+": 12, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "dir-": 1, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "dn": 10, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "dn+": 10, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "dn-": 0, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "cap": 0, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "cap+": 0, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "cap-": 0, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "rss": 51700, 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: "heap": 207228 2024-03-11T20:09:59.493 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_server": { 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_client_request": 0, 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_server_request": 0, 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_request": 0, 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_session": 0, 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_peer_request": 0, 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "req_create_latency": { 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getattr_latency": { 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.494 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getfilelock_latency": { 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: "req_link_latency": { 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookup_latency": { 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.495 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookuphash_latency": { 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupino_latency": { 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupname_latency": { 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.496 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupparent_latency": { 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupsnap_latency": { 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lssnap_latency": { 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.497 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.498 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mkdir_latency": { 2024-03-11T20:09:59.498 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.498 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.498 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.498 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.499 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mknod_latency": { 2024-03-11T20:09:59.499 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.499 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.499 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.499 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.499 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mksnap_latency": { 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: "req_open_latency": { 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: "req_readdir_latency": { 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.500 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rename_latency": { 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: "req_renamesnap_latency": { 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: "req_snapdiff_latency": { 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.501 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmdir_latency": { 2024-03-11T20:09:59.502 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.502 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.502 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.502 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.502 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmsnap_latency": { 2024-03-11T20:09:59.502 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.502 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.502 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.502 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.502 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmxattr_latency": { 2024-03-11T20:09:59.502 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.502 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.502 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setattr_latency": { 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setdirlayout_latency": { 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setfilelock_latency": { 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.503 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setlayout_latency": { 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setxattr_latency": { 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: "req_symlink_latency": { 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.504 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: "req_unlink_latency": { 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_revoke_eviction": 0, 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_acquisition_throttle": 0, 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getvxattr_latency": { 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:59.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_sessions": { 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: "session_count": 0, 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: "session_add": 0, 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: "session_remove": 0, 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_open": 0, 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_stale": 0, 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: "total_load": 0, 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: "average_load": 0, 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: "avg_session_uptime": 0, 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: "mdthresh_evicted": 0 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: "mempool": { 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_bytes": 0, 2024-03-11T20:09:59.506 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_items": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_items": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_items": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_items": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_bytes": 0, 2024-03-11T20:09:59.507 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_items": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_bytes": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_items": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_items": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_bytes": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_items": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_bytes": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_items": 0, 2024-03-11T20:09:59.508 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_bytes": 0, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_items": 0, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_items": 0, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_items": 0, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_bytes": 239397, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_items": 86, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_bytes": 0, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_items": 0, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_bytes": 0, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_items": 0, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_bytes": 0, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_items": 0, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_bytes": 0, 2024-03-11T20:09:59.509 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_items": 0, 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_bytes": 9616, 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_items": 185, 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_items": 0, 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_bytes": 0, 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_items": 0, 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_bytes": 50040, 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_items": 198, 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_bytes": 0, 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_items": 0, 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_bytes": 0, 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_items": 0 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "objecter": { 2024-03-11T20:09:59.510 INFO:teuthology.orchestra.run.smithi161.stdout: "op_active": 0, 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: "op_laggy": 0, 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send": 35, 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send_bytes": 10710, 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: "op_resend": 33, 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: "op_reply": 35, 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: "op_latency": { 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 35, 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.340993079, 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.009742659 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: "op_inflight": 0, 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: "oplen_avg": { 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 35, 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 57 2024-03-11T20:09:59.511 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "op": 35, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "op_r": 0, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "op_w": 35, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "op_rmw": 0, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "op_pg": 0, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_stat": 0, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_create": 10, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_read": 0, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_write": 1, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writefull": 5, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writesame": 0, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_append": 0, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_zero": 0, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_truncate": 0, 2024-03-11T20:09:59.512 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_delete": 6, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_mapext": 0, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_sparse_read": 0, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_clonerange": 0, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_getxattr": 0, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_setxattr": 20, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_cmpxattr": 0, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_rmxattr": 0, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_resetxattrs": 0, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_call": 0, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_watch": 0, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_notify": 0, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls": 0, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls_filter": 0, 2024-03-11T20:09:59.513 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_other": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_active": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_send": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_resend": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_ping": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_active": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_send": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_resend": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_active": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_send": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_resend": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_active": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_send": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_resend": 0, 2024-03-11T20:09:59.514 INFO:teuthology.orchestra.run.smithi161.stdout: "command_active": 0, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "command_send": 0, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "command_resend": 0, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "map_epoch": 102, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "map_full": 0, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "map_inc": 13, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_sessions": 8, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_open": 8, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_close": 0, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_laggy": 0, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_wr": 14, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_rd": 0, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_del": 1 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "oft": { 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_objs": 1, 2024-03-11T20:09:59.515 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_kv_pairs": 0, 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_updates": 0, 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_removes": 0 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: "osd-slow-ops": { 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_ops_count": 0 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: "purge_queue": { 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops": 0, 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing": 0, 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_high_water": 0, 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed_ops": 0, 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed": 0, 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_item_in_journal": 0 2024-03-11T20:09:59.516 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 796, 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 1703277, 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 796, 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 796, 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 1703277, 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.517 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_bytes": { 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 35, 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 10710, 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 35, 2024-03-11T20:09:59.518 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 10710, 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_ops": { 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 1024, 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:59.519 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 35, 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 35, 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 35, 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 35, 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle": { 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:59.520 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle-0x55c257f386e0": { 2024-03-11T20:09:59.521 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 2, 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 588, 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 2, 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 1, 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 588, 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:09:59.522 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:59.523 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:09:59.523 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-11T20:10:00.169 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-11T20:10:00.169 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-11T20:10:00.169 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=2 2024-03-11T20:10:00.169 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-11T20:10:00.169 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T20:10:00.169 DEBUG:tasks.check_counter:Getting stats from l 2024-03-11T20:10:00.170 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-11T20:10:00.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:00 smithi097 ceph-mon[31898]: overall HEALTH_OK 2024-03-11T20:10:00.409 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T20:10:00.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:00 smithi114 ceph-mon[36376]: overall HEALTH_OK 2024-03-11T20:10:00.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:00 smithi161 ceph-mon[36332]: overall HEALTH_OK 2024-03-11T20:10:01.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:01 smithi097 ceph-mon[31898]: pgmap v333: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:10:01.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:01 smithi114 ceph-mon[36376]: pgmap v333: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:10:01.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:01 smithi161 ceph-mon[36332]: pgmap v333: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-11T20:10:02.087 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-11T20:10:02.087 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T20:10:02.087 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 822, 2024-03-11T20:10:02.087 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 824, 2024-03-11T20:10:02.087 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 276137, 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 255643, 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 6, 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 2, 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.103435418, 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.031330909, 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.027963824, 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.033316079, 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 822, 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.049842461, 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000060635 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.088 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 1472, 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 1654, 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 376570, 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 923323, 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 7, 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 5, 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.121823186, 2024-03-11T20:10:02.089 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.043217314, 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.029600342, 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.038549922, 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 1651, 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.076294097, 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000046210 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 466096 2024-03-11T20:10:02.090 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_messages": 276, 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages": 262, 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_bytes": 1607812, 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_bytes": 148640, 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_created_connections": 4, 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_active_connections": 3, 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_total_time": 0.053098172, 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_send_time": 0.016448914, 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_recv_time": 0.039152565, 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_running_fast_dispatch_time": 0.006443457, 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 262, 2024-03-11T20:10:02.091 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.016220124, 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000061908 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_handle_ack_lat": { 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_recv_encrypted_bytes": 1606464, 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: "msgr_send_encrypted_bytes": 147696 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: "cct": { 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: "total_workers": 1, 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: "unhealthy_workers": 0 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.092 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-MDSRank": { 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 3938, 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.111868079, 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000028407 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: "finisher-PurgeQueue": { 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: "queue_len": 0, 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: "complete_latency": { 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:10:02.093 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": { 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: "request": 0, 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: "reply": 0, 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: "reply_latency": { 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_reply": 0, 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: "forward": 0, 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_complete": 0, 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_fetch_keys": 0, 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_commit": 0, 2024-03-11T20:10:02.094 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_split": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_merge": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_top": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_bottom": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pin_tail": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_pinned": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_expired": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "inodes_with_caps": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "subtrees": 1, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_hit": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_forward": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_discover": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_dir_fetch": 0, 2024-03-11T20:10:02.095 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_remote_ino": 0, 2024-03-11T20:10:02.096 INFO:teuthology.orchestra.run.smithi161.stdout: "traverse_lock": 0, 2024-03-11T20:10:02.096 INFO:teuthology.orchestra.run.smithi161.stdout: "load_cent": 0, 2024-03-11T20:10:02.096 INFO:teuthology.orchestra.run.smithi161.stdout: "q": 0, 2024-03-11T20:10:02.096 INFO:teuthology.orchestra.run.smithi161.stdout: "exported": 0, 2024-03-11T20:10:02.096 INFO:teuthology.orchestra.run.smithi161.stdout: "exported_inodes": 0, 2024-03-11T20:10:02.096 INFO:teuthology.orchestra.run.smithi161.stdout: "imported": 0, 2024-03-11T20:10:02.096 INFO:teuthology.orchestra.run.smithi161.stdout: "imported_inodes": 0, 2024-03-11T20:10:02.096 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_dir_fetch": 0, 2024-03-11T20:10:02.096 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_backtrace_fetch": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "openino_peer_discover": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rfiles": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rbytes": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "root_rsnaps": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_set_tag": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_inotable_repaired": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_inodes": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "scrub_file_inodes": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_inode_file_caps": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T20:10:02.097 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_grant": 0, 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps": 0, 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_caps_dirty": 0, 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_cap_release": 0, 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "process_request_cap_release": 0 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_cache": { 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update": 0, 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_update_receipt": 0, 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_try_discover": 0, 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_send_discover": 0, 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "dir_handle_discover": 0, 2024-03-11T20:10:02.098 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_delayed": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "num_strays_enqueuing": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_created": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_enqueued": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_reintegrated": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "strays_migrated": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_processing": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_enqueued": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "num_recovering_prioritized": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_started": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "recovery_completed": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_exportdir": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_flush": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragmentdir": 0, 2024-03-11T20:10:02.099 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_fragstats": 0, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "ireq_inodestats": 0 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_log": { 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "evlrg": 0, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "evadd": 0, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "evex": 0, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "evtrm": 0, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "ev": 5, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "evexg": 0, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "evexd": 0, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "segadd": 0, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "segex": 0, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "segtrm": 0, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "seg": 2, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "segmjr": 2, 2024-03-11T20:10:02.100 INFO:teuthology.orchestra.run.smithi161.stdout: "segexg": 0, 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "segexd": 0, 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "expos": 4194304, 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "wrpos": 4195734, 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "rdpos": 4195593, 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "jlat": { 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "replayed": 5 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_mem": { 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "ino": 3, 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "ino+": 3, 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "ino-": 0, 2024-03-11T20:10:02.101 INFO:teuthology.orchestra.run.smithi161.stdout: "dir": 1, 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "dir+": 1, 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "dir-": 0, 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "dn": 0, 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "dn+": 0, 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "dn-": 0, 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "cap": 0, 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "cap+": 0, 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "cap-": 0, 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "rss": 46472, 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "heap": 215420 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_server": { 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_client_request": 0, 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "dispatch_server_request": 0, 2024-03-11T20:10:02.102 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_request": 0, 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_client_session": 0, 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: "handle_peer_request": 0, 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: "req_create_latency": { 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getattr_latency": { 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getfilelock_latency": { 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.103 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: "req_link_latency": { 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookup_latency": { 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookuphash_latency": { 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.104 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupino_latency": { 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupname_latency": { 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupparent_latency": { 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.105 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lookupsnap_latency": { 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: "req_lssnap_latency": { 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mkdir_latency": { 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.106 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.107 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.107 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.107 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mknod_latency": { 2024-03-11T20:10:02.107 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.107 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.107 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: "req_mksnap_latency": { 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: "req_open_latency": { 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: "req_readdir_latency": { 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.108 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rename_latency": { 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: "req_renamesnap_latency": { 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: "req_snapdiff_latency": { 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.109 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmdir_latency": { 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmsnap_latency": { 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: "req_rmxattr_latency": { 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.110 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.111 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setattr_latency": { 2024-03-11T20:10:02.111 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.111 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.111 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.111 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setdirlayout_latency": { 2024-03-11T20:10:02.111 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.111 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.111 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.111 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.111 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setfilelock_latency": { 2024-03-11T20:10:02.111 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setlayout_latency": { 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: "req_setxattr_latency": { 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: "req_symlink_latency": { 2024-03-11T20:10:02.112 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: "req_unlink_latency": { 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_revoke_eviction": 0, 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: "cap_acquisition_throttle": 0, 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: "req_getvxattr_latency": { 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.113 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_sessions": { 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: "session_count": 1, 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: "session_add": 2, 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: "session_remove": 1, 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_open": 2, 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: "sessions_stale": 0, 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: "total_load": 0, 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: "average_load": 0, 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: "avg_session_uptime": 4017, 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: "mdthresh_evicted": 0 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: "mempool": { 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_bytes": 0, 2024-03-11T20:10:02.114 INFO:teuthology.orchestra.run.smithi161.stdout: "bloom_filter_items": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_alloc_items": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_data_items": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_other_items": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_bytes": 0, 2024-03-11T20:10:02.115 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_extent_items": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_bytes": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_blob_items": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_fsck_items": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_bytes": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_txc_items": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_bytes": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluestore_writing_items": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_bytes": 0, 2024-03-11T20:10:02.116 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_items": 0, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_reader_items": 0, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "bluefs_file_writer_items": 0, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_bytes": 198236, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_anon_items": 57, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_bytes": 0, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "buffer_meta_items": 0, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_bytes": 0, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_items": 0, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_bytes": 0, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_mapbl_items": 0, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_bytes": 0, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_pglog_items": 0, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_bytes": 9616, 2024-03-11T20:10:02.117 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_items": 185, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_mapping_items": 0, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_bytes": 0, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "pgmap_items": 0, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_bytes": 8320, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_co_items": 13, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_bytes": 0, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_1_items": 0, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_bytes": 0, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "unittest_2_items": 0 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "objecter": { 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "op_active": 0, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "op_laggy": 0, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send": 2295, 2024-03-11T20:10:02.118 INFO:teuthology.orchestra.run.smithi161.stdout: "op_send_bytes": 21, 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: "op_resend": 3, 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: "op_reply": 2295, 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: "op_latency": { 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2295, 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 1.581967314, 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000689310 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: "op_inflight": 0, 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: "oplen_avg": { 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 2295, 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 2296 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: "op": 2295, 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: "op_r": 2295, 2024-03-11T20:10:02.119 INFO:teuthology.orchestra.run.smithi161.stdout: "op_w": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "op_rmw": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "op_pg": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_stat": 1465, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_create": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_read": 828, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_write": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writefull": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_writesame": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_append": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_zero": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_truncate": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_delete": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_mapext": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_sparse_read": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_clonerange": 0, 2024-03-11T20:10:02.120 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_getxattr": 1, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_setxattr": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_cmpxattr": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_rmxattr": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_resetxattrs": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_call": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_watch": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_notify": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_pgls_filter": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "osdop_other": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_active": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_send": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_resend": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "linger_ping": 0, 2024-03-11T20:10:02.121 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_active": 0, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_send": 0, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "poolop_resend": 0, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_active": 0, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_send": 0, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "poolstat_resend": 0, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_active": 0, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_send": 0, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "statfs_resend": 0, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "command_active": 0, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "command_send": 0, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "command_resend": 0, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "map_epoch": 102, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "map_full": 0, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "map_inc": 13, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_sessions": 6, 2024-03-11T20:10:02.122 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_open": 6, 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_session_close": 0, 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_laggy": 0, 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_wr": 0, 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_rd": 2, 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_del": 0 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: "oft": { 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_objs": 0, 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_kv_pairs": 0, 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_updates": 0, 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: "omap_total_removes": 0 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: "osd-slow-ops": { 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: "slow_ops_count": 0 2024-03-11T20:10:02.123 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "purge_queue": { 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops": 0, 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing": 0, 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executing_high_water": 0, 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed_ops": 0, 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_executed": 0, 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "pq_item_in_journal": 0 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 2566, 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 2012800, 2024-03-11T20:10:02.124 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 2566, 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2566, 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 2012800, 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_bytes": { 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 104857600, 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:10:02.125 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 2295, 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 1435, 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2295, 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 1435, 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-objecter_ops": { 2024-03-11T20:10:02.126 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 1024, 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 2295, 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 2295, 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 2295, 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 2295, 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:10:02.127 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle": { 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.128 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: "throttle-write_buf_throttle-0x559690856960": { 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: "val": 0, 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: "max": 3758096384, 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: "get_started": 0, 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: "get": 0, 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: "get_sum": 0, 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_fail": 0, 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: "get_or_fail_success": 0, 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: "take": 0, 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: "take_sum": 0, 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: "put": 0, 2024-03-11T20:10:02.129 INFO:teuthology.orchestra.run.smithi161.stdout: "put_sum": 0, 2024-03-11T20:10:02.130 INFO:teuthology.orchestra.run.smithi161.stdout: "wait": { 2024-03-11T20:10:02.130 INFO:teuthology.orchestra.run.smithi161.stdout: "avgcount": 0, 2024-03-11T20:10:02.130 INFO:teuthology.orchestra.run.smithi161.stdout: "sum": 0.000000000, 2024-03-11T20:10:02.130 INFO:teuthology.orchestra.run.smithi161.stdout: "avgtime": 0.000000000 2024-03-11T20:10:02.130 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:10:02.130 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-11T20:10:02.130 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-11T20:10:02.894 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-11T20:10:02.894 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-11T20:10:02.894 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-03-11T20:10:02.894 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-03-11T20:10:02.894 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T20:10:02.895 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-11T20:10:02.906 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-11T20:10:02.906 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T20:10:02.906 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T20:10:02.967 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T20:10:02.968 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T20:10:03.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T20:10:03.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.238+0000 7f020c200640 1 -- 172.21.15.97:0/641307166 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0204108670 msgr2=0x7f0204110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:03.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.238+0000 7f020c200640 1 --2- 172.21.15.97:0/641307166 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0204108670 0x7f0204110240 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009be0 tx=0x7f01fc02bef0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.238+0000 7f020c200640 1 -- 172.21.15.97:0/641307166 shutdown_connections 2024-03-11T20:10:03.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.238+0000 7f020c200640 1 --2- 172.21.15.97:0/641307166 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0204108670 0x7f0204110240 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.238+0000 7f020c200640 1 --2- 172.21.15.97:0/641307166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0204103f50 0x7f0204107f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.238+0000 7f020c200640 1 --2- 172.21.15.97:0/641307166 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02041035a0 0x7f0204103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.238+0000 7f020c200640 1 -- 172.21.15.97:0/641307166 >> 172.21.15.97:0/641307166 conn(0x7f02040fee00 msgr2=0x7f0204101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:03.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.238+0000 7f020c200640 1 -- 172.21.15.97:0/641307166 shutdown_connections 2024-03-11T20:10:03.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.238+0000 7f020c200640 1 -- 172.21.15.97:0/641307166 wait complete. 2024-03-11T20:10:03.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.239+0000 7f020c200640 1 Processor -- start 2024-03-11T20:10:03.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.239+0000 7f020c200640 1 -- start start 2024-03-11T20:10:03.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.239+0000 7f020c200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02041035a0 0x7f020419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:03.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.239+0000 7f020c200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0204108670 0x7f020419fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:03.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.239+0000 7f020c200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02041998c0 0x7f0204199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:03.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.240+0000 7f020c200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020419a2b0 con 0x7f02041998c0 2024-03-11T20:10:03.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.240+0000 7f020c200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020419a420 con 0x7f02041035a0 2024-03-11T20:10:03.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.240+0000 7f020c200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020419a5e0 con 0x7f0204108670 2024-03-11T20:10:03.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.240+0000 7f020a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02041998c0 0x7f0204199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:03.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.240+0000 7f020a200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02041998c0 0x7f0204199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55338/0 (socket says 172.21.15.97:55338) 2024-03-11T20:10:03.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.240+0000 7f020a200640 1 -- 172.21.15.97:0/4227417073 learned_addr learned my addr 172.21.15.97:0/4227417073 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:03.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.240+0000 7f0208e00640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0204108670 0x7f020419fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:03.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.240+0000 7f0209800640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02041035a0 0x7f020419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:03.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.240+0000 7f020a200640 1 -- 172.21.15.97:0/4227417073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0204108670 msgr2=0x7f020419fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:03.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.240+0000 7f020a200640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0204108670 0x7f020419fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.240+0000 7f020a200640 1 -- 172.21.15.97:0/4227417073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02041035a0 msgr2=0x7f020419f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:03.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.240+0000 7f020a200640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02041035a0 0x7f020419f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.240+0000 7f020a200640 1 -- 172.21.15.97:0/4227417073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc009840 con 0x7f02041998c0 2024-03-11T20:10:03.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.241+0000 7f020a200640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02041998c0 0x7f0204199d70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f01fc036040 tx=0x7f01fc0333e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:03.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.241+0000 7f0202000640 1 -- 172.21.15.97:0/4227417073 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc003b70 con 0x7f02041998c0 2024-03-11T20:10:03.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.241+0000 7f020c200640 1 -- 172.21.15.97:0/4227417073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02041a6580 con 0x7f02041998c0 2024-03-11T20:10:03.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.241+0000 7f0202000640 1 -- 172.21.15.97:0/4227417073 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01fc033810 con 0x7f02041998c0 2024-03-11T20:10:03.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.241+0000 7f020c200640 1 -- 172.21.15.97:0/4227417073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02041a6a30 con 0x7f02041998c0 2024-03-11T20:10:03.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.241+0000 7f0202000640 1 -- 172.21.15.97:0/4227417073 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc034730 con 0x7f02041998c0 2024-03-11T20:10:03.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.242+0000 7f020c200640 1 -- 172.21.15.97:0/4227417073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02040683c0 con 0x7f02041998c0 2024-03-11T20:10:03.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.244+0000 7f0202000640 1 -- 172.21.15.97:0/4227417073 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f01fc034890 con 0x7f02041998c0 2024-03-11T20:10:03.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.246+0000 7f0202000640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f01ec0a1ba0 0x7f01ec0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:03.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.246+0000 7f0202000640 1 -- 172.21.15.97:0/4227417073 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7f01fc0f03c0 con 0x7f02041998c0 2024-03-11T20:10:03.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.246+0000 7f0209800640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f01ec0a1ba0 0x7f01ec0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:03.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.246+0000 7f0209800640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f01ec0a1ba0 0x7f01ec0a4060 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f02041084a0 tx=0x7f01f8009260 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:03.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.248+0000 7f0202000640 1 -- 172.21.15.97:0/4227417073 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f01fc0b8fb0 con 0x7f02041998c0 2024-03-11T20:10:03.274 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:03 smithi161 ceph-mon[36332]: pgmap v334: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:10:03.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:03 smithi097 ceph-mon[31898]: pgmap v334: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:10:03.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.446+0000 7f020c200640 1 -- 172.21.15.97:0/4227417073 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f020419ada0 con 0x7f02041998c0 2024-03-11T20:10:03.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.450+0000 7f0202000640 1 -- 172.21.15.97:0/4227417073 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+25400 (secure 0 0 0) 0x7f01fc0bc670 con 0x7f02041998c0 2024-03-11T20:10:03.440 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T20:10:03.440 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":102,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T20:09:13.920214+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/206978801":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/3511489140":"2024-03-12T19:59:16.643130+0000","172.21.15.97:6801/3389151509":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/3879069067":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6800/3389151509":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/839620487":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T20:10:03.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.453+0000 7f020c200640 1 -- 172.21.15.97:0/4227417073 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f01ec0a1ba0 msgr2=0x7f01ec0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.453+0000 7f020c200640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f01ec0a1ba0 0x7f01ec0a4060 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f02041084a0 tx=0x7f01f8009260 comp rx=0 tx=0).stop 2024-03-11T20:10:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.453+0000 7f020c200640 1 -- 172.21.15.97:0/4227417073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02041998c0 msgr2=0x7f0204199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.453+0000 7f020c200640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02041998c0 0x7f0204199d70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f01fc036040 tx=0x7f01fc0333e0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.453+0000 7f020c200640 1 -- 172.21.15.97:0/4227417073 shutdown_connections 2024-03-11T20:10:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.453+0000 7f020c200640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f01ec0a1ba0 0x7f01ec0a4060 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.453+0000 7f020c200640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f02041998c0 0x7f0204199d70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.453+0000 7f020c200640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0204108670 0x7f020419fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.453+0000 7f020c200640 1 --2- 172.21.15.97:0/4227417073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02041035a0 0x7f020419f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.453+0000 7f020c200640 1 -- 172.21.15.97:0/4227417073 >> 172.21.15.97:0/4227417073 conn(0x7f02040fee00 msgr2=0x7f0204101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.454+0000 7f020c200640 1 -- 172.21.15.97:0/4227417073 shutdown_connections 2024-03-11T20:10:03.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.454+0000 7f020c200640 1 -- 172.21.15.97:0/4227417073 wait complete. 2024-03-11T20:10:03.463 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T19:50:29.051985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T19:55:43.424076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T19:55:44.652160+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T19:55:48.598077+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T20:10:03.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T20:10:03.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:03 smithi114 ceph-mon[36376]: pgmap v334: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-11T20:10:03.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.696+0000 7f4cf9400640 1 -- 172.21.15.97:0/2151828603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cf4106300 msgr2=0x7f4cf410ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:03.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.696+0000 7f4cf9400640 1 --2- 172.21.15.97:0/2151828603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cf4106300 0x7f4cf410ded0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009cd0 tx=0x7f4cec02c0e0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.696+0000 7f4cf9400640 1 -- 172.21.15.97:0/2151828603 shutdown_connections 2024-03-11T20:10:03.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.696+0000 7f4cf9400640 1 --2- 172.21.15.97:0/2151828603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cf4106300 0x7f4cf410ded0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.696+0000 7f4cf9400640 1 --2- 172.21.15.97:0/2151828603 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cf4101be0 0x7f4cf4105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.696+0000 7f4cf9400640 1 --2- 172.21.15.97:0/2151828603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cf4101230 0x7f4cf4101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.696+0000 7f4cf9400640 1 -- 172.21.15.97:0/2151828603 >> 172.21.15.97:0/2151828603 conn(0x7f4cf40fcaf0 msgr2=0x7f4cf40fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:03.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.696+0000 7f4cf9400640 1 -- 172.21.15.97:0/2151828603 shutdown_connections 2024-03-11T20:10:03.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.696+0000 7f4cf9400640 1 -- 172.21.15.97:0/2151828603 wait complete. 2024-03-11T20:10:03.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.697+0000 7f4cf9400640 1 Processor -- start 2024-03-11T20:10:03.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.697+0000 7f4cf9400640 1 -- start start 2024-03-11T20:10:03.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.697+0000 7f4cf9400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cf4101230 0x7f4cf419f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:03.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.697+0000 7f4cf9400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cf4106300 0x7f4cf419fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:03.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.697+0000 7f4cf9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cf4199630 0x7f4cf4199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:03.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.697+0000 7f4cf9400640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf419a100 con 0x7f4cf4106300 2024-03-11T20:10:03.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.697+0000 7f4cf9400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf419a270 con 0x7f4cf4199630 2024-03-11T20:10:03.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.697+0000 7f4cf9400640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf419a430 con 0x7f4cf4101230 2024-03-11T20:10:03.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.698+0000 7f4cebe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cf4106300 0x7f4cf419fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:03.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.698+0000 7f4cebe00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cf4106300 0x7f4cf419fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:55352/0 (socket says 172.21.15.97:55352) 2024-03-11T20:10:03.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.698+0000 7f4cf8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cf4199630 0x7f4cf4199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:03.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.698+0000 7f4cebe00640 1 -- 172.21.15.97:0/643877301 learned_addr learned my addr 172.21.15.97:0/643877301 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:03.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.698+0000 7f4cf3e00640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cf4101230 0x7f4cf419f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:03.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.698+0000 7f4cf8a00640 1 -- 172.21.15.97:0/643877301 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cf4101230 msgr2=0x7f4cf419f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:03.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.698+0000 7f4cf8a00640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cf4101230 0x7f4cf419f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.698+0000 7f4cf8a00640 1 -- 172.21.15.97:0/643877301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cf4106300 msgr2=0x7f4cf419fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:03.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.698+0000 7f4cf8a00640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cf4106300 0x7f4cf419fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.698+0000 7f4cf8a00640 1 -- 172.21.15.97:0/643877301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec009900 con 0x7f4cf4199630 2024-03-11T20:10:03.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.698+0000 7f4cebe00640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cf4106300 0x7f4cf419fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:10:03.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.698+0000 7f4cf8a00640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cf4199630 0x7f4cf4199ab0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4cec037040 tx=0x7f4cec0325e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:03.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.699+0000 7f4cf1600640 1 -- 172.21.15.97:0/643877301 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec034e20 con 0x7f4cf4199630 2024-03-11T20:10:03.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.699+0000 7f4cf9400640 1 -- 172.21.15.97:0/643877301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf41a6370 con 0x7f4cf4199630 2024-03-11T20:10:03.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.699+0000 7f4cf1600640 1 -- 172.21.15.97:0/643877301 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4cec003e50 con 0x7f4cf4199630 2024-03-11T20:10:03.690 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.699+0000 7f4cf1600640 1 -- 172.21.15.97:0/643877301 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec004790 con 0x7f4cf4199630 2024-03-11T20:10:03.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.699+0000 7f4cf9400640 1 -- 172.21.15.97:0/643877301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf41a6840 con 0x7f4cf4199630 2024-03-11T20:10:03.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.700+0000 7f4cf9400640 1 -- 172.21.15.97:0/643877301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf4102700 con 0x7f4cf4199630 2024-03-11T20:10:03.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.702+0000 7f4cf1600640 1 -- 172.21.15.97:0/643877301 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4cec00a040 con 0x7f4cf4199630 2024-03-11T20:10:03.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.703+0000 7f4cf1600640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f4cd80a1c20 0x7f4cd80a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:03.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.703+0000 7f4cf1600640 1 -- 172.21.15.97:0/643877301 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7f4cec0f0a90 con 0x7f4cf4199630 2024-03-11T20:10:03.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.704+0000 7f4cf3e00640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f4cd80a1c20 0x7f4cd80a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:03.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.704+0000 7f4cf3e00640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f4cd80a1c20 0x7f4cd80a40e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009bd0 tx=0x7f4ce4005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:03.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.706+0000 7f4cf1600640 1 -- 172.21.15.97:0/643877301 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4cec0b97c0 con 0x7f4cf4199630 2024-03-11T20:10:03.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.903+0000 7f4cf9400640 1 -- 172.21.15.97:0/643877301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4cf4101be0 con 0x7f4cf4199630 2024-03-11T20:10:03.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.903+0000 7f4cf1600640 1 -- 172.21.15.97:0/643877301 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4cec0440a0 con 0x7f4cf4199630 2024-03-11T20:10:03.892 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-03-11T20:10:03.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.907+0000 7f4cf9400640 1 -- 172.21.15.97:0/643877301 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f4cd80a1c20 msgr2=0x7f4cd80a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:03.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.907+0000 7f4cf9400640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f4cd80a1c20 0x7f4cd80a40e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009bd0 tx=0x7f4ce4005f90 comp rx=0 tx=0).stop 2024-03-11T20:10:03.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.907+0000 7f4cf9400640 1 -- 172.21.15.97:0/643877301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cf4199630 msgr2=0x7f4cf4199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:03.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.907+0000 7f4cf9400640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cf4199630 0x7f4cf4199ab0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4cec037040 tx=0x7f4cec0325e0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.907+0000 7f4cf9400640 1 -- 172.21.15.97:0/643877301 shutdown_connections 2024-03-11T20:10:03.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.907+0000 7f4cf9400640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f4cd80a1c20 0x7f4cd80a40e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.907+0000 7f4cf9400640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cf4199630 0x7f4cf4199ab0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.907+0000 7f4cf9400640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4cf4106300 0x7f4cf419fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.907+0000 7f4cf9400640 1 --2- 172.21.15.97:0/643877301 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cf4101230 0x7f4cf419f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:03.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.907+0000 7f4cf9400640 1 -- 172.21.15.97:0/643877301 >> 172.21.15.97:0/643877301 conn(0x7f4cf40fcaf0 msgr2=0x7f4cf40fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:03.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.907+0000 7f4cf9400640 1 -- 172.21.15.97:0/643877301 shutdown_connections 2024-03-11T20:10:03.897 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:03.907+0000 7f4cf9400640 1 -- 172.21.15.97:0/643877301 wait complete. 2024-03-11T20:10:03.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T20:10:04.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.145+0000 7f7ad0000640 1 -- 172.21.15.97:0/3775660196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ac8103f40 msgr2=0x7f7ac8107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:04.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.145+0000 7f7ad0000640 1 --2- 172.21.15.97:0/3775660196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ac8103f40 0x7f7ac8107f80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4009930 tx=0x7f7ab402bc20 comp rx=0 tx=0).stop 2024-03-11T20:10:04.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.145+0000 7f7ad0000640 1 -- 172.21.15.97:0/3775660196 shutdown_connections 2024-03-11T20:10:04.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.145+0000 7f7ad0000640 1 --2- 172.21.15.97:0/3775660196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac8108660 0x7f7ac8110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.145+0000 7f7ad0000640 1 --2- 172.21.15.97:0/3775660196 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ac8103f40 0x7f7ac8107f80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.145+0000 7f7ad0000640 1 --2- 172.21.15.97:0/3775660196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ac8103590 0x7f7ac8103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.145+0000 7f7ad0000640 1 -- 172.21.15.97:0/3775660196 >> 172.21.15.97:0/3775660196 conn(0x7f7ac80fedd0 msgr2=0x7f7ac81011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:04.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.145+0000 7f7ad0000640 1 -- 172.21.15.97:0/3775660196 shutdown_connections 2024-03-11T20:10:04.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.146+0000 7f7ad0000640 1 -- 172.21.15.97:0/3775660196 wait complete. 2024-03-11T20:10:04.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.146+0000 7f7ad0000640 1 Processor -- start 2024-03-11T20:10:04.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.147+0000 7f7ad0000640 1 -- start start 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.147+0000 7f7ad0000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac8103590 0x7f7ac819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.147+0000 7f7ad0000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ac8108660 0x7f7ac819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.147+0000 7f7ad0000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ac8199860 0x7f7ac8199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.147+0000 7f7ad0000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac819a250 con 0x7f7ac8199860 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.147+0000 7f7ad0000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac819a3c0 con 0x7f7ac8103590 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.147+0000 7f7ad0000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac819a580 con 0x7f7ac8108660 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.147+0000 7f7ace000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ac8199860 0x7f7ac8199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.147+0000 7f7acd600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac8103590 0x7f7ac819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.147+0000 7f7accc00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ac8108660 0x7f7ac819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.147+0000 7f7acd600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac8103590 0x7f7ac819f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:35888/0 (socket says 172.21.15.97:35888) 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.147+0000 7f7acd600640 1 -- 172.21.15.97:0/3930493019 learned_addr learned my addr 172.21.15.97:0/3930493019 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.148+0000 7f7acd600640 1 -- 172.21.15.97:0/3930493019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ac8108660 msgr2=0x7f7ac819fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.148+0000 7f7acd600640 1 --2- 172.21.15.97:0/3930493019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ac8108660 0x7f7ac819fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.148+0000 7f7acd600640 1 -- 172.21.15.97:0/3930493019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ac8199860 msgr2=0x7f7ac8199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:04.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.148+0000 7f7acd600640 1 --2- 172.21.15.97:0/3930493019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ac8199860 0x7f7ac8199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.148+0000 7f7acd600640 1 -- 172.21.15.97:0/3930493019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc009490 con 0x7f7ac8103590 2024-03-11T20:10:04.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.148+0000 7f7acd600640 1 --2- 172.21.15.97:0/3930493019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac8103590 0x7f7ac819f6b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7abc00ea60 tx=0x7f7abc00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:04.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.148+0000 7f7ac6000640 1 -- 172.21.15.97:0/3930493019 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc00cd20 con 0x7f7ac8103590 2024-03-11T20:10:04.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.148+0000 7f7ac6000640 1 -- 172.21.15.97:0/3930493019 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7abc00ce80 con 0x7f7ac8103590 2024-03-11T20:10:04.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.148+0000 7f7ad0000640 1 -- 172.21.15.97:0/3930493019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac0009840 con 0x7f7ac8103590 2024-03-11T20:10:04.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.148+0000 7f7ac6000640 1 -- 172.21.15.97:0/3930493019 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc010690 con 0x7f7ac8103590 2024-03-11T20:10:04.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.148+0000 7f7ad0000640 1 -- 172.21.15.97:0/3930493019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab4009560 con 0x7f7ac8103590 2024-03-11T20:10:04.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.149+0000 7f7ad0000640 1 -- 172.21.15.97:0/3930493019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac80683c0 con 0x7f7ac8103590 2024-03-11T20:10:04.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.151+0000 7f7ac6000640 1 -- 172.21.15.97:0/3930493019 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f7abc003ee0 con 0x7f7ac8103590 2024-03-11T20:10:04.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.153+0000 7f7ac6000640 1 --2- 172.21.15.97:0/3930493019 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f7ab00a1ba0 0x7f7ab00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:04.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.153+0000 7f7ac6000640 1 -- 172.21.15.97:0/3930493019 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7f7abc014070 con 0x7f7ac8103590 2024-03-11T20:10:04.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.153+0000 7f7accc00640 1 --2- 172.21.15.97:0/3930493019 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f7ab00a1ba0 0x7f7ab00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:04.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.154+0000 7f7accc00640 1 --2- 172.21.15.97:0/3930493019 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f7ab00a1ba0 0x7f7ab00a4060 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4000c00 tx=0x7f7ab4005fd0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:04.147 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.157+0000 7f7ac6000640 1 -- 172.21.15.97:0/3930493019 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7abc096b70 con 0x7f7ac8103590 2024-03-11T20:10:04.271 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:04 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/4227417073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:10:04.271 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:04 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/643877301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T20:10:04.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.358+0000 7f7ad0000640 1 -- 172.21.15.97:0/3930493019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7ac810b8f0 con 0x7f7ac8103590 2024-03-11T20:10:04.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.359+0000 7f7ac6000640 1 -- 172.21.15.97:0/3930493019 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7abc09a230 con 0x7f7ac8103590 2024-03-11T20:10:04.348 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T20:10:04.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.362+0000 7f7ad0000640 1 -- 172.21.15.97:0/3930493019 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f7ab00a1ba0 msgr2=0x7f7ab00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:04.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.362+0000 7f7ad0000640 1 --2- 172.21.15.97:0/3930493019 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f7ab00a1ba0 0x7f7ab00a4060 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4000c00 tx=0x7f7ab4005fd0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.362+0000 7f7ad0000640 1 -- 172.21.15.97:0/3930493019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac8103590 msgr2=0x7f7ac819f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:04.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.362+0000 7f7ad0000640 1 --2- 172.21.15.97:0/3930493019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac8103590 0x7f7ac819f6b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7abc00ea60 tx=0x7f7abc00ee60 comp rx=0 tx=0).stop 2024-03-11T20:10:04.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.362+0000 7f7ad0000640 1 -- 172.21.15.97:0/3930493019 shutdown_connections 2024-03-11T20:10:04.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.362+0000 7f7ad0000640 1 --2- 172.21.15.97:0/3930493019 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f7ab00a1ba0 0x7f7ab00a4060 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.362+0000 7f7ad0000640 1 --2- 172.21.15.97:0/3930493019 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ac8199860 0x7f7ac8199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.351 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.362+0000 7f7ad0000640 1 --2- 172.21.15.97:0/3930493019 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ac8108660 0x7f7ac819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.362+0000 7f7ad0000640 1 --2- 172.21.15.97:0/3930493019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac8103590 0x7f7ac819f6b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.362+0000 7f7ad0000640 1 -- 172.21.15.97:0/3930493019 >> 172.21.15.97:0/3930493019 conn(0x7f7ac80fedd0 msgr2=0x7f7ac8100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:04.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.362+0000 7f7ad0000640 1 -- 172.21.15.97:0/3930493019 shutdown_connections 2024-03-11T20:10:04.352 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.362+0000 7f7ad0000640 1 -- 172.21.15.97:0/3930493019 wait complete. 2024-03-11T20:10:04.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T20:10:04.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:04 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/4227417073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:10:04.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:04 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/643877301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T20:10:04.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:04 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/4227417073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:10:04.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:04 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/643877301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T20:10:04.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.595+0000 7fd1b0000640 1 -- 172.21.15.97:0/3530269550 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1a8103590 msgr2=0x7fd1a8103970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:04.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.595+0000 7fd1b0000640 1 --2- 172.21.15.97:0/3530269550 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1a8103590 0x7fd1a8103970 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009be0 tx=0x7fd1a002bef0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.596+0000 7fd1b0000640 1 -- 172.21.15.97:0/3530269550 shutdown_connections 2024-03-11T20:10:04.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.596+0000 7fd1b0000640 1 --2- 172.21.15.97:0/3530269550 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1a8108660 0x7fd1a8110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.596+0000 7fd1b0000640 1 --2- 172.21.15.97:0/3530269550 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1a8103f40 0x7fd1a8107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.596+0000 7fd1b0000640 1 --2- 172.21.15.97:0/3530269550 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1a8103590 0x7fd1a8103970 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.596+0000 7fd1b0000640 1 -- 172.21.15.97:0/3530269550 >> 172.21.15.97:0/3530269550 conn(0x7fd1a80fedf0 msgr2=0x7fd1a8101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:04.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.596+0000 7fd1b0000640 1 -- 172.21.15.97:0/3530269550 shutdown_connections 2024-03-11T20:10:04.585 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.596+0000 7fd1b0000640 1 -- 172.21.15.97:0/3530269550 wait complete. 2024-03-11T20:10:04.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.597+0000 7fd1b0000640 1 Processor -- start 2024-03-11T20:10:04.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.597+0000 7fd1b0000640 1 -- start start 2024-03-11T20:10:04.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.597+0000 7fd1b0000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1a8103590 0x7fd1a819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:04.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.597+0000 7fd1b0000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1a8108660 0x7fd1a819fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:04.587 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.597+0000 7fd1b0000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1a8199840 0x7fd1a8199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.597+0000 7fd1b0000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a819a2f0 con 0x7fd1a8199840 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.597+0000 7fd1b0000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a819a460 con 0x7fd1a8108660 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.597+0000 7fd1b0000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a819a5d0 con 0x7fd1a8103590 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.597+0000 7fd1ae000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1a8199840 0x7fd1a8199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.597+0000 7fd1acc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1a8108660 0x7fd1a819fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.598+0000 7fd1ae000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1a8199840 0x7fd1a8199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:44994/0 (socket says 172.21.15.97:44994) 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.598+0000 7fd1ae000640 1 -- 172.21.15.97:0/969387909 learned_addr learned my addr 172.21.15.97:0/969387909 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.598+0000 7fd1ad600640 1 --2- 172.21.15.97:0/969387909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1a8103590 0x7fd1a819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.598+0000 7fd1ae000640 1 -- 172.21.15.97:0/969387909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1a8103590 msgr2=0x7fd1a819f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.598+0000 7fd1ae000640 1 --2- 172.21.15.97:0/969387909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1a8103590 0x7fd1a819f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.598+0000 7fd1ae000640 1 -- 172.21.15.97:0/969387909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1a8108660 msgr2=0x7fd1a819fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.598+0000 7fd1ae000640 1 --2- 172.21.15.97:0/969387909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1a8108660 0x7fd1a819fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.598+0000 7fd1ae000640 1 -- 172.21.15.97:0/969387909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c009560 con 0x7fd1a8199840 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.598+0000 7fd1ae000640 1 --2- 172.21.15.97:0/969387909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1a8199840 0x7fd1a8199cf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00e940 tx=0x7fd19c00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.598+0000 7fd1a6000640 1 -- 172.21.15.97:0/969387909 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c00cd80 con 0x7fd1a8199840 2024-03-11T20:10:04.588 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.598+0000 7fd1a6000640 1 -- 172.21.15.97:0/969387909 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd19c0035f0 con 0x7fd1a8199840 2024-03-11T20:10:04.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.598+0000 7fd1a6000640 1 -- 172.21.15.97:0/969387909 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c010690 con 0x7fd1a8199840 2024-03-11T20:10:04.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.598+0000 7fd1b0000640 1 -- 172.21.15.97:0/969387909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a0009840 con 0x7fd1a8199840 2024-03-11T20:10:04.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.599+0000 7fd1b0000640 1 -- 172.21.15.97:0/969387909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a81a6880 con 0x7fd1a8199840 2024-03-11T20:10:04.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.600+0000 7fd1b0000640 1 -- 172.21.15.97:0/969387909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a8104a60 con 0x7fd1a8199840 2024-03-11T20:10:04.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.602+0000 7fd1a6000640 1 -- 172.21.15.97:0/969387909 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd19c003af0 con 0x7fd1a8199840 2024-03-11T20:10:04.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.603+0000 7fd1a6000640 1 --2- 172.21.15.97:0/969387909 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fd1900a1ba0 0x7fd1900a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:04.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.603+0000 7fd1a6000640 1 -- 172.21.15.97:0/969387909 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7fd19c014070 con 0x7fd1a8199840 2024-03-11T20:10:04.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.603+0000 7fd1ad600640 1 --2- 172.21.15.97:0/969387909 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fd1900a1ba0 0x7fd1900a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:04.592 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.604+0000 7fd1ad600640 1 --2- 172.21.15.97:0/969387909 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fd1900a1ba0 0x7fd1900a4060 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009370 tx=0x7fd1a0005c30 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:04.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.607+0000 7fd1a6000640 1 -- 172.21.15.97:0/969387909 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd19c096a90 con 0x7fd1a8199840 2024-03-11T20:10:04.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.801+0000 7fd1b0000640 1 -- 172.21.15.97:0/969387909 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd1a810b930 con 0x7fd1a8199840 2024-03-11T20:10:04.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.802+0000 7fd1a6000640 1 -- 172.21.15.97:0/969387909 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd19c09a150 con 0x7fd1a8199840 2024-03-11T20:10:04.791 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 64 2024-03-11T20:10:04.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.804+0000 7fd1b0000640 1 -- 172.21.15.97:0/969387909 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fd1900a1ba0 msgr2=0x7fd1900a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:04.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.804+0000 7fd1b0000640 1 --2- 172.21.15.97:0/969387909 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fd1900a1ba0 0x7fd1900a4060 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009370 tx=0x7fd1a0005c30 comp rx=0 tx=0).stop 2024-03-11T20:10:04.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.804+0000 7fd1b0000640 1 -- 172.21.15.97:0/969387909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1a8199840 msgr2=0x7fd1a8199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:04.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.804+0000 7fd1b0000640 1 --2- 172.21.15.97:0/969387909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1a8199840 0x7fd1a8199cf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00e940 tx=0x7fd19c00ee00 comp rx=0 tx=0).stop 2024-03-11T20:10:04.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.805+0000 7fd1b0000640 1 -- 172.21.15.97:0/969387909 shutdown_connections 2024-03-11T20:10:04.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.805+0000 7fd1b0000640 1 --2- 172.21.15.97:0/969387909 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fd1900a1ba0 0x7fd1900a4060 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.805+0000 7fd1b0000640 1 --2- 172.21.15.97:0/969387909 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fd1a8199840 0x7fd1a8199cf0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.805+0000 7fd1b0000640 1 --2- 172.21.15.97:0/969387909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd1a8108660 0x7fd1a819fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.805+0000 7fd1b0000640 1 --2- 172.21.15.97:0/969387909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1a8103590 0x7fd1a819f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:04.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.805+0000 7fd1b0000640 1 -- 172.21.15.97:0/969387909 >> 172.21.15.97:0/969387909 conn(0x7fd1a80fedf0 msgr2=0x7fd1a8100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:04.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.805+0000 7fd1b0000640 1 -- 172.21.15.97:0/969387909 shutdown_connections 2024-03-11T20:10:04.794 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:04.805+0000 7fd1b0000640 1 -- 172.21.15.97:0/969387909 wait complete. 2024-03-11T20:10:04.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T20:10:05.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.035+0000 7f4479000640 1 -- 172.21.15.97:0/655692124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447410d730 msgr2=0x7f447410db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:05.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.035+0000 7f4479000640 1 --2- 172.21.15.97:0/655692124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447410d730 0x7f447410db10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4464009930 tx=0x7f446402bc30 comp rx=0 tx=0).stop 2024-03-11T20:10:05.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.035+0000 7f4479000640 1 -- 172.21.15.97:0/655692124 shutdown_connections 2024-03-11T20:10:05.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.035+0000 7f4479000640 1 --2- 172.21.15.97:0/655692124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44741080e0 0x7f44741145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.024 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.035+0000 7f4479000640 1 --2- 172.21.15.97:0/655692124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4474107720 0x7f4474107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.035+0000 7f4479000640 1 --2- 172.21.15.97:0/655692124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447410d730 0x7f447410db10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.035+0000 7f4479000640 1 -- 172.21.15.97:0/655692124 >> 172.21.15.97:0/655692124 conn(0x7f44740756c0 msgr2=0x7f4474077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:05.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.035+0000 7f4479000640 1 -- 172.21.15.97:0/655692124 shutdown_connections 2024-03-11T20:10:05.025 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.035+0000 7f4479000640 1 -- 172.21.15.97:0/655692124 wait complete. 2024-03-11T20:10:05.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.036+0000 7f4479000640 1 Processor -- start 2024-03-11T20:10:05.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.036+0000 7f4479000640 1 -- start start 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.036+0000 7f4479000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4474107720 0x7f447419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.036+0000 7f4479000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44741080e0 0x7f447419e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.036+0000 7f4479000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447410d730 0x7f44741a2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.036+0000 7f4479000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44741a2c80 con 0x7f44741080e0 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.036+0000 7f4479000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44741a2df0 con 0x7f4474107720 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.036+0000 7f4479000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44741a2fb0 con 0x7f447410d730 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.037+0000 7f4472800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447410d730 0x7f44741a2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.037+0000 7f4471e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4474107720 0x7f447419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.037+0000 7f4471400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44741080e0 0x7f447419e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.037+0000 7f4472800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447410d730 0x7f44741a2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:55380/0 (socket says 172.21.15.97:55380) 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.037+0000 7f4471e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4474107720 0x7f447419ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:35922/0 (socket says 172.21.15.97:35922) 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.037+0000 7f4472800640 1 -- 172.21.15.97:0/3946514607 learned_addr learned my addr 172.21.15.97:0/3946514607 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.037+0000 7f4471e00640 1 -- 172.21.15.97:0/3946514607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447410d730 msgr2=0x7f44741a2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.037+0000 7f4471e00640 1 --2- 172.21.15.97:0/3946514607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447410d730 0x7f44741a2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.027 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.037+0000 7f4471e00640 1 -- 172.21.15.97:0/3946514607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44741080e0 msgr2=0x7f447419e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:05.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.037+0000 7f4471e00640 1 --2- 172.21.15.97:0/3946514607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44741080e0 0x7f447419e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.037+0000 7f4471e00640 1 -- 172.21.15.97:0/3946514607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4464009560 con 0x7f4474107720 2024-03-11T20:10:05.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.037+0000 7f4471e00640 1 --2- 172.21.15.97:0/3946514607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4474107720 0x7f447419ddc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4464009230 tx=0x7f446402c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:05.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.038+0000 7f446aa00640 1 -- 172.21.15.97:0/3946514607 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44640042d0 con 0x7f4474107720 2024-03-11T20:10:05.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.038+0000 7f446aa00640 1 -- 172.21.15.97:0/3946514607 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4464004430 con 0x7f4474107720 2024-03-11T20:10:05.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.038+0000 7f446aa00640 1 -- 172.21.15.97:0/3946514607 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4464003bd0 con 0x7f4474107720 2024-03-11T20:10:05.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.038+0000 7f4479000640 1 -- 172.21.15.97:0/3946514607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44741a3230 con 0x7f4474107720 2024-03-11T20:10:05.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.038+0000 7f4479000640 1 -- 172.21.15.97:0/3946514607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44741aab10 con 0x7f4474107720 2024-03-11T20:10:05.028 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.039+0000 7f4479000640 1 -- 172.21.15.97:0/3946514607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44740683c0 con 0x7f4474107720 2024-03-11T20:10:05.029 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.041+0000 7f446aa00640 1 -- 172.21.15.97:0/3946514607 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4464003d30 con 0x7f4474107720 2024-03-11T20:10:05.030 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.042+0000 7f446aa00640 1 --2- 172.21.15.97:0/3946514607 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f44580a1c20 0x7f44580a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:05.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.042+0000 7f446aa00640 1 -- 172.21.15.97:0/3946514607 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7f44640efd50 con 0x7f4474107720 2024-03-11T20:10:05.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.042+0000 7f4471400640 1 --2- 172.21.15.97:0/3946514607 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f44580a1c20 0x7f44580a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:05.031 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.043+0000 7f4471400640 1 --2- 172.21.15.97:0/3946514607 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f44580a1c20 0x7f44580a40e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4474077d80 tx=0x7f4460005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:05.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.044+0000 7f446aa00640 1 -- 172.21.15.97:0/3946514607 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f44640b88e0 con 0x7f4474107720 2024-03-11T20:10:05.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:05 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3930493019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T20:10:05.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:05 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/969387909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T20:10:05.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:05 smithi097 ceph-mon[31898]: pgmap v335: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:10:05.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.238+0000 7f4479000640 1 -- 172.21.15.97:0/3946514607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f447410fc90 con 0x7f4474107720 2024-03-11T20:10:05.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.239+0000 7f446aa00640 1 -- 172.21.15.97:0/3946514607 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) v1 ==== 103+0+11 (secure 0 0 0) 0x7f44640bbfa0 con 0x7f4474107720 2024-03-11T20:10:05.228 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 32 2024-03-11T20:10:05.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.241+0000 7f4479000640 1 -- 172.21.15.97:0/3946514607 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f44580a1c20 msgr2=0x7f44580a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:05.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.241+0000 7f4479000640 1 --2- 172.21.15.97:0/3946514607 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f44580a1c20 0x7f44580a40e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4474077d80 tx=0x7f4460005f90 comp rx=0 tx=0).stop 2024-03-11T20:10:05.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.241+0000 7f4479000640 1 -- 172.21.15.97:0/3946514607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4474107720 msgr2=0x7f447419ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:05.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.241+0000 7f4479000640 1 --2- 172.21.15.97:0/3946514607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4474107720 0x7f447419ddc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4464009230 tx=0x7f446402c5d0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.241+0000 7f4479000640 1 -- 172.21.15.97:0/3946514607 shutdown_connections 2024-03-11T20:10:05.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.241+0000 7f4479000640 1 --2- 172.21.15.97:0/3946514607 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f44580a1c20 0x7f44580a40e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.241+0000 7f4479000640 1 --2- 172.21.15.97:0/3946514607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f447410d730 0x7f44741a2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.241+0000 7f4479000640 1 --2- 172.21.15.97:0/3946514607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f44741080e0 0x7f447419e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.241+0000 7f4479000640 1 --2- 172.21.15.97:0/3946514607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4474107720 0x7f447419ddc0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.241+0000 7f4479000640 1 -- 172.21.15.97:0/3946514607 >> 172.21.15.97:0/3946514607 conn(0x7f44740756c0 msgr2=0x7f4474075f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:05.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.241+0000 7f4479000640 1 -- 172.21.15.97:0/3946514607 shutdown_connections 2024-03-11T20:10:05.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.241+0000 7f4479000640 1 -- 172.21.15.97:0/3946514607 wait complete. 2024-03-11T20:10:05.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-11T20:10:05.274 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:05 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3930493019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T20:10:05.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:05 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/969387909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T20:10:05.275 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:05 smithi161 ceph-mon[36332]: pgmap v335: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:10:05.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.462+0000 7f5755e00640 1 -- 172.21.15.97:0/1686676268 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5750106430 msgr2=0x7f575010e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:05.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.462+0000 7f5755e00640 1 --2- 172.21.15.97:0/1686676268 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5750106430 0x7f575010e000 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5748009860 tx=0x7f574802bb70 comp rx=0 tx=0).stop 2024-03-11T20:10:05.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.462+0000 7f5755e00640 1 -- 172.21.15.97:0/1686676268 shutdown_connections 2024-03-11T20:10:05.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.462+0000 7f5755e00640 1 --2- 172.21.15.97:0/1686676268 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5750106430 0x7f575010e000 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.462+0000 7f5755e00640 1 --2- 172.21.15.97:0/1686676268 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750068f80 0x7f5750105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.462+0000 7f5755e00640 1 --2- 172.21.15.97:0/1686676268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57500685d0 0x7f57500689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.463+0000 7f5755e00640 1 -- 172.21.15.97:0/1686676268 >> 172.21.15.97:0/1686676268 conn(0x7f5750077180 msgr2=0x7f5750077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:05.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.463+0000 7f5755e00640 1 -- 172.21.15.97:0/1686676268 shutdown_connections 2024-03-11T20:10:05.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.463+0000 7f5755e00640 1 -- 172.21.15.97:0/1686676268 wait complete. 2024-03-11T20:10:05.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.464+0000 7f5755e00640 1 Processor -- start 2024-03-11T20:10:05.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.464+0000 7f5755e00640 1 -- start start 2024-03-11T20:10:05.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.464+0000 7f5755e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57500685d0 0x7f5750100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:05.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.464+0000 7f5755e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750106430 0x7f5750100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:05.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.464+0000 7f5755e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5750100ac0 0x7f57501a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:05.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.464+0000 7f5755e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57501a5f40 con 0x7f5750106430 2024-03-11T20:10:05.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.464+0000 7f5755e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57501a60b0 con 0x7f57500685d0 2024-03-11T20:10:05.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.464+0000 7f5755e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57501a6270 con 0x7f5750100ac0 2024-03-11T20:10:05.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.465+0000 7f574f400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5750100ac0 0x7f57501a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:05.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.465+0000 7f574e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750106430 0x7f5750100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:05.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.465+0000 7f574f400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5750100ac0 0x7f57501a5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:55392/0 (socket says 172.21.15.97:55392) 2024-03-11T20:10:05.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.465+0000 7f574e000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750106430 0x7f5750100580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45020/0 (socket says 172.21.15.97:45020) 2024-03-11T20:10:05.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.465+0000 7f574f400640 1 -- 172.21.15.97:0/735259117 learned_addr learned my addr 172.21.15.97:0/735259117 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:05.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.465+0000 7f574ea00640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57500685d0 0x7f5750100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:05.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.465+0000 7f574f400640 1 -- 172.21.15.97:0/735259117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57500685d0 msgr2=0x7f5750100040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:05.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.465+0000 7f574f400640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57500685d0 0x7f5750100040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.465+0000 7f574f400640 1 -- 172.21.15.97:0/735259117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750106430 msgr2=0x7f5750100580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:05.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.465+0000 7f574f400640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750106430 0x7f5750100580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.465+0000 7f574f400640 1 -- 172.21.15.97:0/735259117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5738009900 con 0x7f5750100ac0 2024-03-11T20:10:05.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.465+0000 7f574e000640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750106430 0x7f5750100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:10:05.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.466+0000 7f574f400640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5750100ac0 0x7f57501a5970 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f574802c180 tx=0x7f5748004530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:05.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.466+0000 7f5754a00640 1 -- 172.21.15.97:0/735259117 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748034a10 con 0x7f5750100ac0 2024-03-11T20:10:05.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.466+0000 7f5755e00640 1 -- 172.21.15.97:0/735259117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5748009490 con 0x7f5750100ac0 2024-03-11T20:10:05.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.466+0000 7f5754a00640 1 -- 172.21.15.97:0/735259117 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5748005510 con 0x7f5750100ac0 2024-03-11T20:10:05.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.466+0000 7f5754a00640 1 -- 172.21.15.97:0/735259117 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748004920 con 0x7f5750100ac0 2024-03-11T20:10:05.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.466+0000 7f5755e00640 1 -- 172.21.15.97:0/735259117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57501a6810 con 0x7f5750100ac0 2024-03-11T20:10:05.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.467+0000 7f5755e00640 1 -- 172.21.15.97:0/735259117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5750067f70 con 0x7f5750100ac0 2024-03-11T20:10:05.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.469+0000 7f5754a00640 1 -- 172.21.15.97:0/735259117 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f574803b040 con 0x7f5750100ac0 2024-03-11T20:10:05.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.470+0000 7f5754a00640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f57300a1bd0 0x7f57300a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:05.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.470+0000 7f5754a00640 1 -- 172.21.15.97:0/735259117 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7f57480ef700 con 0x7f5750100ac0 2024-03-11T20:10:05.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.470+0000 7f574ea00640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f57300a1bd0 0x7f57300a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:05.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.471+0000 7f574ea00640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f57300a1bd0 0x7f57300a4090 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5740009bd0 tx=0x7f5740005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:05.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.473+0000 7f5754a00640 1 -- 172.21.15.97:0/735259117 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f57480b8290 con 0x7f5750100ac0 2024-03-11T20:10:05.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:05 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3930493019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T20:10:05.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:05 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/969387909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T20:10:05.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:05 smithi114 ceph-mon[36376]: pgmap v335: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:10:05.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.682+0000 7f5755e00640 1 -- 172.21.15.97:0/735259117 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f57501096c0 con 0x7f5750100ac0 2024-03-11T20:10:05.672 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.683+0000 7f5754a00640 1 -- 172.21.15.97:0/735259117 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v102) v1 ==== 81+0+942 (secure 0 0 0) 0x7f57480413c0 con 0x7f5750100ac0 2024-03-11T20:10:05.672 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/206978801 2024-03-12T19:59:16.643130+0000 2024-03-11T20:10:05.672 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/3511489140 2024-03-12T19:59:16.643130+0000 2024-03-11T20:10:05.672 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6801/3389151509 2024-03-12T19:59:16.643130+0000 2024-03-11T20:10:05.672 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/3879069067 2024-03-12T19:59:16.643130+0000 2024-03-11T20:10:05.672 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/4129219534 2024-03-12T19:47:44.356323+0000 2024-03-11T20:10:05.672 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/3932792931 2024-03-12T19:47:44.356323+0000 2024-03-11T20:10:05.673 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6801/1708150324 2024-03-12T19:47:44.356323+0000 2024-03-11T20:10:05.673 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6800/1708150324 2024-03-12T19:47:44.356323+0000 2024-03-11T20:10:05.673 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6800/3216682632 2024-03-12T19:47:09.368819+0000 2024-03-11T20:10:05.673 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6800/3389151509 2024-03-12T19:59:16.643130+0000 2024-03-11T20:10:05.673 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/3236990094 2024-03-12T19:47:09.368819+0000 2024-03-11T20:10:05.673 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/839620487 2024-03-12T19:59:16.643130+0000 2024-03-11T20:10:05.673 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/1301912133 2024-03-12T19:47:09.368819+0000 2024-03-11T20:10:05.673 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:6801/3216682632 2024-03-12T19:47:09.368819+0000 2024-03-11T20:10:05.673 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/46019613 2024-03-12T19:47:44.356323+0000 2024-03-11T20:10:05.673 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97:0/4194903751 2024-03-12T19:47:09.368819+0000 2024-03-11T20:10:05.673 INFO:teuthology.orchestra.run.smithi097.stderr:listed 16 entries 2024-03-11T20:10:05.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.686+0000 7f5755e00640 1 -- 172.21.15.97:0/735259117 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f57300a1bd0 msgr2=0x7f57300a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:05.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.686+0000 7f5755e00640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f57300a1bd0 0x7f57300a4090 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5740009bd0 tx=0x7f5740005f90 comp rx=0 tx=0).stop 2024-03-11T20:10:05.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.686+0000 7f5755e00640 1 -- 172.21.15.97:0/735259117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5750100ac0 msgr2=0x7f57501a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:05.674 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.686+0000 7f5755e00640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5750100ac0 0x7f57501a5970 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f574802c180 tx=0x7f5748004530 comp rx=0 tx=0).stop 2024-03-11T20:10:05.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.686+0000 7f5755e00640 1 -- 172.21.15.97:0/735259117 shutdown_connections 2024-03-11T20:10:05.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.686+0000 7f5755e00640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f57300a1bd0 0x7f57300a4090 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.686+0000 7f5755e00640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5750100ac0 0x7f57501a5970 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.686+0000 7f5755e00640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5750106430 0x7f5750100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.686+0000 7f5755e00640 1 --2- 172.21.15.97:0/735259117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57500685d0 0x7f5750100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:05.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.686+0000 7f5755e00640 1 -- 172.21.15.97:0/735259117 >> 172.21.15.97:0/735259117 conn(0x7f5750077180 msgr2=0x7f57500fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:05.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.686+0000 7f5755e00640 1 -- 172.21.15.97:0/735259117 shutdown_connections 2024-03-11T20:10:05.675 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:05.686+0000 7f5755e00640 1 -- 172.21.15.97:0/735259117 wait complete. 2024-03-11T20:10:05.694 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-11T20:10:05.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T20:10:05.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-11T20:10:05.803 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com 2024-03-11T20:10:05.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T20:10:05.804 DEBUG:teuthology.orchestra.run.smithi097:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-11T20:10:05.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T20:10:05.863 DEBUG:teuthology.orchestra.run.smithi097:> ip netns list 2024-03-11T20:10:05.900 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-11T20:10:05.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T20:10:05.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-11T20:10:06.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T20:10:06.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip link delete ceph-brx 2024-03-11T20:10:06.139 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T20:10:06.150 INFO:tasks.cephadm:Teardown begin 2024-03-11T20:10:06.150 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-11T20:10:06.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T20:10:06.338 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.349+0000 7fe544000640 1 -- 172.21.15.97:0/2840223464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe53c103f30 msgr2=0x7fe53c107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:06.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.349+0000 7fe544000640 1 --2- 172.21.15.97:0/2840223464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe53c103f30 0x7fe53c107f70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe530009930 tx=0x7fe53002bc30 comp rx=0 tx=0).stop 2024-03-11T20:10:06.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.349+0000 7fe544000640 1 -- 172.21.15.97:0/2840223464 shutdown_connections 2024-03-11T20:10:06.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.349+0000 7fe544000640 1 --2- 172.21.15.97:0/2840223464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c108650 0x7fe53c110220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.349+0000 7fe544000640 1 --2- 172.21.15.97:0/2840223464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe53c103f30 0x7fe53c107f70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.349+0000 7fe544000640 1 --2- 172.21.15.97:0/2840223464 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe53c103580 0x7fe53c103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.349+0000 7fe544000640 1 -- 172.21.15.97:0/2840223464 >> 172.21.15.97:0/2840223464 conn(0x7fe53c0fede0 msgr2=0x7fe53c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:06.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.350+0000 7fe544000640 1 -- 172.21.15.97:0/2840223464 shutdown_connections 2024-03-11T20:10:06.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.350+0000 7fe544000640 1 -- 172.21.15.97:0/2840223464 wait complete. 2024-03-11T20:10:06.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.351+0000 7fe544000640 1 Processor -- start 2024-03-11T20:10:06.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.351+0000 7fe544000640 1 -- start start 2024-03-11T20:10:06.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.351+0000 7fe544000640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe53c103580 0x7fe53c199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:06.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.351+0000 7fe544000640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c108650 0x7fe53c19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:06.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.351+0000 7fe544000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe53c19e3c0 0x7fe53c19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:06.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.351+0000 7fe544000640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c19edb0 con 0x7fe53c103580 2024-03-11T20:10:06.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.351+0000 7fe544000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c19ef20 con 0x7fe53c19e3c0 2024-03-11T20:10:06.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.351+0000 7fe544000640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c19f0e0 con 0x7fe53c108650 2024-03-11T20:10:06.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.351+0000 7fe541600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe53c103580 0x7fe53c199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:06.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.351+0000 7fe540c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c108650 0x7fe53c19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:06.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.352+0000 7fe540c00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c108650 0x7fe53c19a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.97:55412/0 (socket says 172.21.15.97:55412) 2024-03-11T20:10:06.341 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.352+0000 7fe540c00640 1 -- 172.21.15.97:0/2199508297 learned_addr learned my addr 172.21.15.97:0/2199508297 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:06.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.352+0000 7fe542000640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe53c19e3c0 0x7fe53c19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:06.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.352+0000 7fe540c00640 1 -- 172.21.15.97:0/2199508297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe53c19e3c0 msgr2=0x7fe53c19e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:06.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.352+0000 7fe540c00640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe53c19e3c0 0x7fe53c19e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.352+0000 7fe540c00640 1 -- 172.21.15.97:0/2199508297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe53c103580 msgr2=0x7fe53c199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:06.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.352+0000 7fe540c00640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe53c103580 0x7fe53c199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.352+0000 7fe540c00640 1 -- 172.21.15.97:0/2199508297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe530009560 con 0x7fe53c108650 2024-03-11T20:10:06.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.352+0000 7fe541600640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe53c103580 0x7fe53c199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:10:06.342 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.352+0000 7fe540c00640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c108650 0x7fe53c19a010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe530009230 tx=0x7fe53002c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.353+0000 7fe53a000640 1 -- 172.21.15.97:0/2199508297 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530004350 con 0x7fe53c108650 2024-03-11T20:10:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.353+0000 7fe53a000640 1 -- 172.21.15.97:0/2199508297 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5300044b0 con 0x7fe53c108650 2024-03-11T20:10:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.353+0000 7fe53a000640 1 -- 172.21.15.97:0/2199508297 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530003da0 con 0x7fe53c108650 2024-03-11T20:10:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.353+0000 7fe544000640 1 -- 172.21.15.97:0/2199508297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53c1a6670 con 0x7fe53c108650 2024-03-11T20:10:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.353+0000 7fe544000640 1 -- 172.21.15.97:0/2199508297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53c1a6b20 con 0x7fe53c108650 2024-03-11T20:10:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.354+0000 7fe544000640 1 -- 172.21.15.97:0/2199508297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53c0683c0 con 0x7fe53c108650 2024-03-11T20:10:06.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.356+0000 7fe53a000640 1 -- 172.21.15.97:0/2199508297 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe530004df0 con 0x7fe53c108650 2024-03-11T20:10:06.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.357+0000 7fe53a000640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe5240a1c20 0x7fe5240a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:06.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.357+0000 7fe53a000640 1 -- 172.21.15.97:0/2199508297 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7fe5300efa80 con 0x7fe53c108650 2024-03-11T20:10:06.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.357+0000 7fe541600640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe5240a1c20 0x7fe5240a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:06.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.358+0000 7fe541600640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe5240a1c20 0x7fe5240a40e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009fc0 tx=0x7fe52c005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:06.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.361+0000 7fe53a000640 1 -- 172.21.15.97:0/2199508297 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe5300b8610 con 0x7fe53c108650 2024-03-11T20:10:06.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:06 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/3946514607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T20:10:06.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:06 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/735259117' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T20:10:06.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:06 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/3946514607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T20:10:06.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:06 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/735259117' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T20:10:06.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:06 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/3946514607' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T20:10:06.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:06 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/735259117' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T20:10:06.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.597+0000 7fe544000640 1 -- 172.21.15.97:0/2199508297 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe53c10b880 con 0x7fe53c108650 2024-03-11T20:10:06.586 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.598+0000 7fe53a000640 1 -- 172.21.15.97:0/2199508297 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11401 (secure 0 0 0) 0x7fe5300bbcd0 con 0x7fe53c108650 2024-03-11T20:10:06.587 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T20:10:06.587 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:55:45.149335+0000","modified":"2024-03-11T19:59:05.947695+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24583,"mds_2":24547,"mds_3":24545,"mds_4":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24583":{"gid":24583,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.97:6841/2531110572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2531110572},{"type":"v1","addr":"172.21.15.97:6841","nonce":2531110572}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6839/1624532248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":1624532248},{"type":"v1","addr":"172.21.15.114:6839","nonce":1624532248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T20:10:06.587 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 29 2024-03-11T20:10:06.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.600+0000 7fe544000640 1 -- 172.21.15.97:0/2199508297 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe5240a1c20 msgr2=0x7fe5240a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:06.589 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.600+0000 7fe544000640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe5240a1c20 0x7fe5240a40e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009fc0 tx=0x7fe52c005f90 comp rx=0 tx=0).stop 2024-03-11T20:10:06.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.600+0000 7fe544000640 1 -- 172.21.15.97:0/2199508297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c108650 msgr2=0x7fe53c19a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:06.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.600+0000 7fe544000640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c108650 0x7fe53c19a010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe530009230 tx=0x7fe53002c680 comp rx=0 tx=0).stop 2024-03-11T20:10:06.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.601+0000 7fe544000640 1 -- 172.21.15.97:0/2199508297 shutdown_connections 2024-03-11T20:10:06.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.601+0000 7fe544000640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fe5240a1c20 0x7fe5240a40e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.601+0000 7fe544000640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe53c19e3c0 0x7fe53c19e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.601+0000 7fe544000640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe53c108650 0x7fe53c19a010 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.601+0000 7fe544000640 1 --2- 172.21.15.97:0/2199508297 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe53c103580 0x7fe53c199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.601+0000 7fe544000640 1 -- 172.21.15.97:0/2199508297 >> 172.21.15.97:0/2199508297 conn(0x7fe53c0fede0 msgr2=0x7fe53c1000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:06.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.601+0000 7fe544000640 1 -- 172.21.15.97:0/2199508297 shutdown_connections 2024-03-11T20:10:06.590 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.601+0000 7fe544000640 1 -- 172.21.15.97:0/2199508297 wait complete. 2024-03-11T20:10:06.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T20:10:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.832+0000 7f4349e00640 1 -- 172.21.15.97:0/4059907881 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43441080f0 msgr2=0x7f4344114600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.832+0000 7f4349e00640 1 --2- 172.21.15.97:0/4059907881 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43441080f0 0x7f4344114600 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f433c009c10 tx=0x7f433c02bef0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.833+0000 7f4349e00640 1 -- 172.21.15.97:0/4059907881 shutdown_connections 2024-03-11T20:10:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.833+0000 7f4349e00640 1 --2- 172.21.15.97:0/4059907881 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43441080f0 0x7f4344114600 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.833+0000 7f4349e00640 1 --2- 172.21.15.97:0/4059907881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4344107730 0x7f4344107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.833+0000 7f4349e00640 1 --2- 172.21.15.97:0/4059907881 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f434410d740 0x7f434410db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.833+0000 7f4349e00640 1 -- 172.21.15.97:0/4059907881 >> 172.21.15.97:0/4059907881 conn(0x7f43440756d0 msgr2=0x7f4344077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.833+0000 7f4349e00640 1 -- 172.21.15.97:0/4059907881 shutdown_connections 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.833+0000 7f4349e00640 1 -- 172.21.15.97:0/4059907881 wait complete. 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.834+0000 7f4349e00640 1 Processor -- start 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.834+0000 7f4349e00640 1 -- start start 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.834+0000 7f4349e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4344107730 0x7f434419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.834+0000 7f4349e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43441080f0 0x7f434419e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.834+0000 7f4349e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f434410d740 0x7f43441a2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.834+0000 7f4349e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43441a2c80 con 0x7f4344107730 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.834+0000 7f4349e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43441a2df0 con 0x7f43441080f0 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.834+0000 7f4349e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43441a2fb0 con 0x7f434410d740 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.834+0000 7f4342a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4344107730 0x7f434419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.834+0000 7f4342a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4344107730 0x7f434419ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45062/0 (socket says 172.21.15.97:45062) 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.834+0000 7f4342000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43441080f0 0x7f434419e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:06.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.834+0000 7f4342a00640 1 -- 172.21.15.97:0/940373968 learned_addr learned my addr 172.21.15.97:0/940373968 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:06.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.834+0000 7f4343400640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f434410d740 0x7f43441a2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.835+0000 7f4343400640 1 -- 172.21.15.97:0/940373968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43441080f0 msgr2=0x7f434419e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.835+0000 7f4343400640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43441080f0 0x7f434419e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.835+0000 7f4343400640 1 -- 172.21.15.97:0/940373968 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4344107730 msgr2=0x7f434419ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.835+0000 7f4343400640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4344107730 0x7f434419ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.835+0000 7f4343400640 1 -- 172.21.15.97:0/940373968 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c009840 con 0x7f434410d740 2024-03-11T20:10:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.835+0000 7f4342000640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43441080f0 0x7f434419e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:10:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.835+0000 7f4343400640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f434410d740 0x7f43441a2680 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f433c036040 tx=0x7f433c033610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.835+0000 7f4348a00640 1 -- 172.21.15.97:0/940373968 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c003b70 con 0x7f434410d740 2024-03-11T20:10:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.836+0000 7f4349e00640 1 -- 172.21.15.97:0/940373968 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43441a3230 con 0x7f434410d740 2024-03-11T20:10:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.836+0000 7f4348a00640 1 -- 172.21.15.97:0/940373968 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f433c02ce40 con 0x7f434410d740 2024-03-11T20:10:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.836+0000 7f4348a00640 1 -- 172.21.15.97:0/940373968 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c03d720 con 0x7f434410d740 2024-03-11T20:10:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.836+0000 7f4349e00640 1 -- 172.21.15.97:0/940373968 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43441aab10 con 0x7f434410d740 2024-03-11T20:10:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.837+0000 7f4349e00640 1 -- 172.21.15.97:0/940373968 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43440683c0 con 0x7f434410d740 2024-03-11T20:10:06.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.839+0000 7f4348a00640 1 -- 172.21.15.97:0/940373968 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f433c004040 con 0x7f434410d740 2024-03-11T20:10:06.828 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.840+0000 7f4348a00640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f43280a1ba0 0x7f43280a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:06.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.840+0000 7f4348a00640 1 -- 172.21.15.97:0/940373968 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7f433c0f0260 con 0x7f434410d740 2024-03-11T20:10:06.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.840+0000 7f4342a00640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f43280a1ba0 0x7f43280a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:06.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.841+0000 7f4342a00640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f43280a1ba0 0x7f43280a4060 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4334009c10 tx=0x7f4334005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:06.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:06.844+0000 7f4348a00640 1 -- 172.21.15.97:0/940373968 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f433c0b8e50 con 0x7f434410d740 2024-03-11T20:10:07.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.081+0000 7f4349e00640 1 -- 172.21.15.97:0/940373968 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f434410fd00 con 0x7f434410d740 2024-03-11T20:10:07.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.082+0000 7f4348a00640 1 -- 172.21.15.97:0/940373968 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11401 (secure 0 0 0) 0x7f433c0bc510 con 0x7f434410d740 2024-03-11T20:10:07.072 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T20:10:07.072 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6835/207363921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":207363921},{"type":"v1","addr":"172.21.15.97:6835","nonce":207363921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2355075568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6838","nonce":2355075568},{"type":"v1","addr":"172.21.15.97:6839","nonce":2355075568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T19:55:45.149335+0000","modified":"2024-03-11T19:59:05.947695+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24583,"mds_2":24547,"mds_3":24545,"mds_4":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6833/2213740234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2213740234},{"type":"v1","addr":"172.21.15.161:6833","nonce":2213740234}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.114:6833/2550912009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2550912009},{"type":"v1","addr":"172.21.15.114:6833","nonce":2550912009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.97:6837/267259401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":267259401},{"type":"v1","addr":"172.21.15.97:6837","nonce":267259401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24512":{"gid":24512,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2623690462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2623690462},{"type":"v1","addr":"172.21.15.114:6835","nonce":2623690462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6835/3358622196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":3358622196},{"type":"v1","addr":"172.21.15.161:6835","nonce":3358622196}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24529":{"gid":24529,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.161:6837/1600156948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1600156948},{"type":"v1","addr":"172.21.15.161:6837","nonce":1600156948}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6837/1112816294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1112816294},{"type":"v1","addr":"172.21.15.114:6837","nonce":1112816294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.161:6839/565994162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":565994162},{"type":"v1","addr":"172.21.15.161:6839","nonce":565994162}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24583":{"gid":24583,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.97:6841/2531110572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2531110572},{"type":"v1","addr":"172.21.15.97:6841","nonce":2531110572}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.114:6839/1624532248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":1624532248},{"type":"v1","addr":"172.21.15.114:6839","nonce":1624532248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T20:10:07.072 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 29 2024-03-11T20:10:07.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.085+0000 7f4349e00640 1 -- 172.21.15.97:0/940373968 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f43280a1ba0 msgr2=0x7f43280a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:07.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.085+0000 7f4349e00640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f43280a1ba0 0x7f43280a4060 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4334009c10 tx=0x7f4334005f90 comp rx=0 tx=0).stop 2024-03-11T20:10:07.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.085+0000 7f4349e00640 1 -- 172.21.15.97:0/940373968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f434410d740 msgr2=0x7f43441a2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:07.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.085+0000 7f4349e00640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f434410d740 0x7f43441a2680 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f433c036040 tx=0x7f433c033610 comp rx=0 tx=0).stop 2024-03-11T20:10:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.085+0000 7f4349e00640 1 -- 172.21.15.97:0/940373968 shutdown_connections 2024-03-11T20:10:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.085+0000 7f4349e00640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f43280a1ba0 0x7f43280a4060 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.085+0000 7f4349e00640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f434410d740 0x7f43441a2680 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.085+0000 7f4349e00640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43441080f0 0x7f434419e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.085+0000 7f4349e00640 1 --2- 172.21.15.97:0/940373968 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4344107730 0x7f434419ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.085+0000 7f4349e00640 1 -- 172.21.15.97:0/940373968 >> 172.21.15.97:0/940373968 conn(0x7f43440756d0 msgr2=0x7f4344075d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:07.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.085+0000 7f4349e00640 1 -- 172.21.15.97:0/940373968 shutdown_connections 2024-03-11T20:10:07.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.085+0000 7f4349e00640 1 -- 172.21.15.97:0/940373968 wait complete. 2024-03-11T20:10:07.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T20:10:07.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.300+0000 7fcd12e00640 1 -- 172.21.15.97:0/782909083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd0c103580 msgr2=0x7fcd0c103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:07.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.300+0000 7fcd12e00640 1 --2- 172.21.15.97:0/782909083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd0c103580 0x7fcd0c103960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcd04009be0 tx=0x7fcd0402bef0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.300+0000 7fcd12e00640 1 -- 172.21.15.97:0/782909083 shutdown_connections 2024-03-11T20:10:07.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.300+0000 7fcd12e00640 1 --2- 172.21.15.97:0/782909083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0c108650 0x7fcd0c110220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.300+0000 7fcd12e00640 1 --2- 172.21.15.97:0/782909083 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd0c103f30 0x7fcd0c107f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.300+0000 7fcd12e00640 1 --2- 172.21.15.97:0/782909083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd0c103580 0x7fcd0c103960 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.300+0000 7fcd12e00640 1 -- 172.21.15.97:0/782909083 >> 172.21.15.97:0/782909083 conn(0x7fcd0c0fede0 msgr2=0x7fcd0c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:07.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.300+0000 7fcd12e00640 1 -- 172.21.15.97:0/782909083 shutdown_connections 2024-03-11T20:10:07.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.301+0000 7fcd12e00640 1 -- 172.21.15.97:0/782909083 wait complete. 2024-03-11T20:10:07.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.302+0000 7fcd12e00640 1 Processor -- start 2024-03-11T20:10:07.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.302+0000 7fcd12e00640 1 -- start start 2024-03-11T20:10:07.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.302+0000 7fcd12e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0c103580 0x7fcd0c19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:07.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.302+0000 7fcd12e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd0c108650 0x7fcd0c19fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:07.291 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.302+0000 7fcd12e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd0c1998c0 0x7fcd0c199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:07.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.302+0000 7fcd12e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c19a2b0 con 0x7fcd0c1998c0 2024-03-11T20:10:07.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.302+0000 7fcd12e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c19a420 con 0x7fcd0c103580 2024-03-11T20:10:07.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.302+0000 7fcd12e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c19a5e0 con 0x7fcd0c108650 2024-03-11T20:10:07.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.303+0000 7fcd0be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0c103580 0x7fcd0c19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:07.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.303+0000 7fcd0b400640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd0c108650 0x7fcd0c19fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:07.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.303+0000 7fcd0be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0c103580 0x7fcd0c19f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:36010/0 (socket says 172.21.15.97:36010) 2024-03-11T20:10:07.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.303+0000 7fcd0be00640 1 -- 172.21.15.97:0/282622601 learned_addr learned my addr 172.21.15.97:0/282622601 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:07.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.303+0000 7fcd10e00640 1 --2- 172.21.15.97:0/282622601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd0c1998c0 0x7fcd0c199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:07.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.303+0000 7fcd0be00640 1 -- 172.21.15.97:0/282622601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd0c108650 msgr2=0x7fcd0c19fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:07.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.303+0000 7fcd0be00640 1 --2- 172.21.15.97:0/282622601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd0c108650 0x7fcd0c19fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.303+0000 7fcd0be00640 1 -- 172.21.15.97:0/282622601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd0c1998c0 msgr2=0x7fcd0c199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:07.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.303+0000 7fcd0be00640 1 --2- 172.21.15.97:0/282622601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd0c1998c0 0x7fcd0c199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.293 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.303+0000 7fcd0be00640 1 -- 172.21.15.97:0/282622601 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd04009840 con 0x7fcd0c103580 2024-03-11T20:10:07.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.303+0000 7fcd0be00640 1 --2- 172.21.15.97:0/282622601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0c103580 0x7fcd0c19f710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcd04036040 tx=0x7fcd04033610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:07.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.304+0000 7fcd08c00640 1 -- 172.21.15.97:0/282622601 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd04003d80 con 0x7fcd0c103580 2024-03-11T20:10:07.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.304+0000 7fcd12e00640 1 -- 172.21.15.97:0/282622601 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0c1a65d0 con 0x7fcd0c103580 2024-03-11T20:10:07.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.304+0000 7fcd08c00640 1 -- 172.21.15.97:0/282622601 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd0402ce40 con 0x7fcd0c103580 2024-03-11T20:10:07.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.304+0000 7fcd08c00640 1 -- 172.21.15.97:0/282622601 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd040348f0 con 0x7fcd0c103580 2024-03-11T20:10:07.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.304+0000 7fcd12e00640 1 -- 172.21.15.97:0/282622601 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd0c1a6a80 con 0x7fcd0c103580 2024-03-11T20:10:07.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.305+0000 7fcd12e00640 1 -- 172.21.15.97:0/282622601 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0c0683c0 con 0x7fcd0c103580 2024-03-11T20:10:07.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.307+0000 7fcd08c00640 1 -- 172.21.15.97:0/282622601 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fcd040045b0 con 0x7fcd0c103580 2024-03-11T20:10:07.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.308+0000 7fcd08c00640 1 --2- 172.21.15.97:0/282622601 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fccf00a1c20 0x7fccf00a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:07.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.308+0000 7fcd08c00640 1 -- 172.21.15.97:0/282622601 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7fcd040f0080 con 0x7fcd0c103580 2024-03-11T20:10:07.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.309+0000 7fcd0b400640 1 --2- 172.21.15.97:0/282622601 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fccf00a1c20 0x7fccf00a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:07.298 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.309+0000 7fcd0b400640 1 --2- 172.21.15.97:0/282622601 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fccf00a1c20 0x7fccf00a40e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c102650 tx=0x7fccf4009260 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:07.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.312+0000 7fcd08c00640 1 -- 172.21.15.97:0/282622601 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcd040b8c10 con 0x7fcd0c103580 2024-03-11T20:10:07.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:07 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2199508297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T20:10:07.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:07 smithi097 ceph-mon[31898]: pgmap v336: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:10:07.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:07 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2199508297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T20:10:07.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:07 smithi114 ceph-mon[36376]: pgmap v336: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:10:07.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.511+0000 7fcd12e00640 1 -- 172.21.15.97:0/282622601 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd0c10b8b0 con 0x7fcd0c103580 2024-03-11T20:10:07.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.513+0000 7fcd08c00640 1 -- 172.21.15.97:0/282622601 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+25400 (secure 0 0 0) 0x7fcd040bc2d0 con 0x7fcd0c103580 2024-03-11T20:10:07.502 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T20:10:07.502 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":102,"fsid":"e81ca0d2-dfdf-11ee-95c8-87774f69a715","created":"2024-03-11T19:46:35.573122+0000","modified":"2024-03-11T20:09:13.920214+0000","last_up_change":"2024-03-11T19:54:43.082568+0000","last_in_change":"2024-03-11T19:54:22.590035+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T19:50:29.051985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T19:55:43.424076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T19:55:44.652160+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T19:55:48.598077+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaa3b17b-f7a0-4cf1-857c-5e3415f224a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6803","nonce":2786748520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6805","nonce":2786748520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6809","nonce":2786748520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2786748520},{"type":"v1","addr":"172.21.15.97:6807","nonce":2786748520}]},"public_addr":"172.21.15.97:6803/2786748520","cluster_addr":"172.21.15.97:6805/2786748520","heartbeat_back_addr":"172.21.15.97:6809/2786748520","heartbeat_front_addr":"172.21.15.97:6807/2786748520","state":["exists","up"]},{"osd":1,"uuid":"259f580e-e340-4048-a461-5da3946ed1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6801","nonce":2861651844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6803","nonce":2861651844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6807","nonce":2861651844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2861651844},{"type":"v1","addr":"172.21.15.114:6805","nonce":2861651844}]},"public_addr":"172.21.15.114:6801/2861651844","cluster_addr":"172.21.15.114:6803/2861651844","heartbeat_back_addr":"172.21.15.114:6807/2861651844","heartbeat_front_addr":"172.21.15.114:6805/2861651844","state":["exists","up"]},{"osd":2,"uuid":"64012698-9df0-4d30-b4e4-1bebdbde0a0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6801","nonce":4070161993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6803","nonce":4070161993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6807","nonce":4070161993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":4070161993},{"type":"v1","addr":"172.21.15.161:6805","nonce":4070161993}]},"public_addr":"172.21.15.161:6801/4070161993","cluster_addr":"172.21.15.161:6803/4070161993","heartbeat_back_addr":"172.21.15.161:6807/4070161993","heartbeat_front_addr":"172.21.15.161:6805/4070161993","state":["exists","up"]},{"osd":3,"uuid":"794e7a63-96da-4d6e-b020-29d61db19359","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6811","nonce":2971789214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6813","nonce":2971789214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6817","nonce":2971789214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":2971789214},{"type":"v1","addr":"172.21.15.97:6815","nonce":2971789214}]},"public_addr":"172.21.15.97:6811/2971789214","cluster_addr":"172.21.15.97:6813/2971789214","heartbeat_back_addr":"172.21.15.97:6817/2971789214","heartbeat_front_addr":"172.21.15.97:6815/2971789214","state":["exists","up"]},{"osd":4,"uuid":"601a3bf7-cfee-4e6d-a838-7f04eccdb73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6809","nonce":1414652177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6811","nonce":1414652177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6815","nonce":1414652177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1414652177},{"type":"v1","addr":"172.21.15.114:6813","nonce":1414652177}]},"public_addr":"172.21.15.114:6809/1414652177","cluster_addr":"172.21.15.114:6811/1414652177","heartbeat_back_addr":"172.21.15.114:6815/1414652177","heartbeat_front_addr":"172.21.15.114:6813/1414652177","state":["exists","up"]},{"osd":5,"uuid":"0ce520f9-2aef-4901-aa82-edd7ca858028","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6809","nonce":2197634465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6811","nonce":2197634465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6815","nonce":2197634465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2197634465},{"type":"v1","addr":"172.21.15.161:6813","nonce":2197634465}]},"public_addr":"172.21.15.161:6809/2197634465","cluster_addr":"172.21.15.161:6811/2197634465","heartbeat_back_addr":"172.21.15.161:6815/2197634465","heartbeat_front_addr":"172.21.15.161:6813/2197634465","state":["exists","up"]},{"osd":6,"uuid":"609b0dd7-7cff-4f8e-b656-2a11e7652150","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6819","nonce":2094652517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6821","nonce":2094652517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6825","nonce":2094652517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2094652517},{"type":"v1","addr":"172.21.15.97:6823","nonce":2094652517}]},"public_addr":"172.21.15.97:6819/2094652517","cluster_addr":"172.21.15.97:6821/2094652517","heartbeat_back_addr":"172.21.15.97:6825/2094652517","heartbeat_front_addr":"172.21.15.97:6823/2094652517","state":["exists","up"]},{"osd":7,"uuid":"5a0d4de9-3628-4421-91b0-0c8cb7cdcf93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6817","nonce":3855344963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6819","nonce":3855344963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6823","nonce":3855344963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3855344963},{"type":"v1","addr":"172.21.15.114:6821","nonce":3855344963}]},"public_addr":"172.21.15.114:6817/3855344963","cluster_addr":"172.21.15.114:6819/3855344963","heartbeat_back_addr":"172.21.15.114:6823/3855344963","heartbeat_front_addr":"172.21.15.114:6821/3855344963","state":["exists","up"]},{"osd":8,"uuid":"2c0abbf6-d848-4911-beff-7b4d3ce7b330","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6817","nonce":4061144818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6819","nonce":4061144818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6823","nonce":4061144818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":4061144818},{"type":"v1","addr":"172.21.15.161:6821","nonce":4061144818}]},"public_addr":"172.21.15.161:6817/4061144818","cluster_addr":"172.21.15.161:6819/4061144818","heartbeat_back_addr":"172.21.15.161:6823/4061144818","heartbeat_front_addr":"172.21.15.161:6821/4061144818","state":["exists","up"]},{"osd":9,"uuid":"d55f416c-9d0f-4938-8e34-b364abeaedcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6827","nonce":2375353428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6829","nonce":2375353428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6833","nonce":2375353428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2375353428},{"type":"v1","addr":"172.21.15.97:6831","nonce":2375353428}]},"public_addr":"172.21.15.97:6827/2375353428","cluster_addr":"172.21.15.97:6829/2375353428","heartbeat_back_addr":"172.21.15.97:6833/2375353428","heartbeat_front_addr":"172.21.15.97:6831/2375353428","state":["exists","up"]},{"osd":10,"uuid":"27d60b08-8d2b-453e-841d-abb1339ce6e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6825","nonce":275586537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6827","nonce":275586537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6831","nonce":275586537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":275586537},{"type":"v1","addr":"172.21.15.114:6829","nonce":275586537}]},"public_addr":"172.21.15.114:6825/275586537","cluster_addr":"172.21.15.114:6827/275586537","heartbeat_back_addr":"172.21.15.114:6831/275586537","heartbeat_front_addr":"172.21.15.114:6829/275586537","state":["exists","up"]},{"osd":11,"uuid":"f16c85e3-0e4a-4ae3-894f-6dfe489e7325","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6825","nonce":1523383703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6827","nonce":1523383703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6831","nonce":1523383703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1523383703},{"type":"v1","addr":"172.21.15.161:6829","nonce":1523383703}]},"public_addr":"172.21.15.161:6825/1523383703","cluster_addr":"172.21.15.161:6827/1523383703","heartbeat_back_addr":"172.21.15.161:6831/1523383703","heartbeat_front_addr":"172.21.15.161:6829/1523383703","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:49:37.119378+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:01.858752+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:25.299226+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:50:58.031845+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:24.775315+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:51:49.514071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:23.090161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:52:49.808162+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:14.783233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:53:49.109616+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:15.554910+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T19:54:40.801319+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/206978801":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/3511489140":"2024-03-12T19:59:16.643130+0000","172.21.15.97:6801/3389151509":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/3879069067":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/4129219534":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/3932792931":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6801/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/1708150324":"2024-03-12T19:47:44.356323+0000","172.21.15.97:6800/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6800/3389151509":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/3236990094":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/839620487":"2024-03-12T19:59:16.643130+0000","172.21.15.97:0/1301912133":"2024-03-12T19:47:09.368819+0000","172.21.15.97:6801/3216682632":"2024-03-12T19:47:09.368819+0000","172.21.15.97:0/46019613":"2024-03-12T19:47:44.356323+0000","172.21.15.97:0/4194903751":"2024-03-12T19:47:09.368819+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T20:10:07.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.515+0000 7fcd12e00640 1 -- 172.21.15.97:0/282622601 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fccf00a1c20 msgr2=0x7fccf00a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:07.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.516+0000 7fcd12e00640 1 --2- 172.21.15.97:0/282622601 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fccf00a1c20 0x7fccf00a40e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c102650 tx=0x7fccf4009260 comp rx=0 tx=0).stop 2024-03-11T20:10:07.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.516+0000 7fcd12e00640 1 -- 172.21.15.97:0/282622601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0c103580 msgr2=0x7fcd0c19f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:07.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.516+0000 7fcd12e00640 1 --2- 172.21.15.97:0/282622601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0c103580 0x7fcd0c19f710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcd04036040 tx=0x7fcd04033610 comp rx=0 tx=0).stop 2024-03-11T20:10:07.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.516+0000 7fcd12e00640 1 -- 172.21.15.97:0/282622601 shutdown_connections 2024-03-11T20:10:07.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.516+0000 7fcd12e00640 1 --2- 172.21.15.97:0/282622601 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fccf00a1c20 0x7fccf00a40e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.516+0000 7fcd12e00640 1 --2- 172.21.15.97:0/282622601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcd0c1998c0 0x7fcd0c199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.516+0000 7fcd12e00640 1 --2- 172.21.15.97:0/282622601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcd0c108650 0x7fcd0c19fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.516+0000 7fcd12e00640 1 --2- 172.21.15.97:0/282622601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd0c103580 0x7fcd0c19f710 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.516+0000 7fcd12e00640 1 -- 172.21.15.97:0/282622601 >> 172.21.15.97:0/282622601 conn(0x7fcd0c0fede0 msgr2=0x7fcd0c101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:07.505 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.516+0000 7fcd12e00640 1 -- 172.21.15.97:0/282622601 shutdown_connections 2024-03-11T20:10:07.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.516+0000 7fcd12e00640 1 -- 172.21.15.97:0/282622601 wait complete. 2024-03-11T20:10:07.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-11T20:10:07.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:07 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2199508297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T20:10:07.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:07 smithi161 ceph-mon[36332]: pgmap v336: 129 pgs: 129 active+clean; 2.3 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-11T20:10:07.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.752+0000 7f509b200640 1 -- 172.21.15.97:0/2350247896 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509c10d660 msgr2=0x7f509c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:07.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.752+0000 7f509b200640 1 --2- 172.21.15.97:0/2350247896 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509c10d660 0x7f509c10da40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f508c009930 tx=0x7f508c02bcd0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.753+0000 7f509b200640 1 -- 172.21.15.97:0/2350247896 shutdown_connections 2024-03-11T20:10:07.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.753+0000 7f509b200640 1 --2- 172.21.15.97:0/2350247896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c108060 0x7f509c1144b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.753+0000 7f509b200640 1 --2- 172.21.15.97:0/2350247896 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f509c1076c0 0x7f509c107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.753+0000 7f509b200640 1 --2- 172.21.15.97:0/2350247896 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509c10d660 0x7f509c10da40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.753+0000 7f509b200640 1 -- 172.21.15.97:0/2350247896 >> 172.21.15.97:0/2350247896 conn(0x7f509c076fa0 msgr2=0x7f509c105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:07.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.753+0000 7f509b200640 1 -- 172.21.15.97:0/2350247896 shutdown_connections 2024-03-11T20:10:07.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.753+0000 7f509b200640 1 -- 172.21.15.97:0/2350247896 wait complete. 2024-03-11T20:10:07.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.754+0000 7f509b200640 1 Processor -- start 2024-03-11T20:10:07.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.754+0000 7f509b200640 1 -- start start 2024-03-11T20:10:07.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.754+0000 7f509b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c1076c0 0x7f509c1a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:07.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.754+0000 7f509b200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f509c108060 0x7f509c1a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:07.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.754+0000 7f509b200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509c10d660 0x7f509c19dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:07.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.754+0000 7f509b200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c19e110 con 0x7f509c108060 2024-03-11T20:10:07.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.754+0000 7f509b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c19e280 con 0x7f509c1076c0 2024-03-11T20:10:07.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.754+0000 7f509b200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c19e440 con 0x7f509c10d660 2024-03-11T20:10:07.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.754+0000 7f5099400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f509c108060 0x7f509c1a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:07.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.754+0000 7f5099e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c1076c0 0x7f509c1a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:07.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.754+0000 7f509a800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509c10d660 0x7f509c19dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:07.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.754+0000 7f5099400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f509c108060 0x7f509c1a3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45092/0 (socket says 172.21.15.97:45092) 2024-03-11T20:10:07.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.754+0000 7f5099400640 1 -- 172.21.15.97:0/2971302185 learned_addr learned my addr 172.21.15.97:0/2971302185 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:07.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.755+0000 7f5099400640 1 -- 172.21.15.97:0/2971302185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509c10d660 msgr2=0x7f509c19dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:07.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.755+0000 7f5099400640 1 --2- 172.21.15.97:0/2971302185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509c10d660 0x7f509c19dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.755+0000 7f5099400640 1 -- 172.21.15.97:0/2971302185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c1076c0 msgr2=0x7f509c1a38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:07.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.755+0000 7f5099400640 1 --2- 172.21.15.97:0/2971302185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c1076c0 0x7f509c1a38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.755+0000 7f5099400640 1 -- 172.21.15.97:0/2971302185 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c009560 con 0x7f509c108060 2024-03-11T20:10:07.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.755+0000 7f5099400640 1 --2- 172.21.15.97:0/2971302185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f509c108060 0x7f509c1a3e00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f508800e9f0 tx=0x7f508800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:07.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.755+0000 7f5092a00640 1 -- 172.21.15.97:0/2971302185 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088009700 con 0x7f509c108060 2024-03-11T20:10:07.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.756+0000 7f509b200640 1 -- 172.21.15.97:0/2971302185 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509c19e720 con 0x7f509c108060 2024-03-11T20:10:07.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.756+0000 7f5092a00640 1 -- 172.21.15.97:0/2971302185 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f508800cd80 con 0x7f509c108060 2024-03-11T20:10:07.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.756+0000 7f5092a00640 1 -- 172.21.15.97:0/2971302185 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088018920 con 0x7f509c108060 2024-03-11T20:10:07.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.756+0000 7f509b200640 1 -- 172.21.15.97:0/2971302185 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509c1aa960 con 0x7f509c108060 2024-03-11T20:10:07.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.757+0000 7f509b200640 1 -- 172.21.15.97:0/2971302185 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509c0682e0 con 0x7f509c108060 2024-03-11T20:10:07.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.759+0000 7f5092a00640 1 -- 172.21.15.97:0/2971302185 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5088018a80 con 0x7f509c108060 2024-03-11T20:10:07.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.760+0000 7f5092a00640 1 --2- 172.21.15.97:0/2971302185 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f50800a1ba0 0x7f50800a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:07.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.761+0000 7f5092a00640 1 -- 172.21.15.97:0/2971302185 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9533+0+0 (secure 0 0 0) 0x7f5088014070 con 0x7f509c108060 2024-03-11T20:10:07.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.761+0000 7f5099e00640 1 --2- 172.21.15.97:0/2971302185 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f50800a1ba0 0x7f50800a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:07.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.761+0000 7f5099e00640 1 --2- 172.21.15.97:0/2971302185 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f50800a1ba0 0x7f50800a4060 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f508c005ab0 tx=0x7f508c005fd0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:07.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:07.765+0000 7f5092a00640 1 -- 172.21.15.97:0/2971302185 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5088052d20 con 0x7f509c108060 2024-03-11T20:10:07.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.001+0000 7f509b200640 1 -- 172.21.15.97:0/2971302185 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f509c10fb00 con 0x7f509c108060 2024-03-11T20:10:07.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.005+0000 7f5092a00640 1 -- 172.21.15.97:0/2971302185 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v30) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5088096ae0 con 0x7f509c108060 2024-03-11T20:10:07.993 INFO:teuthology.orchestra.run.smithi097.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-11T20:10:07.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.009+0000 7f509b200640 1 -- 172.21.15.97:0/2971302185 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f50800a1ba0 msgr2=0x7f50800a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:07.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.009+0000 7f509b200640 1 --2- 172.21.15.97:0/2971302185 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f50800a1ba0 0x7f50800a4060 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f508c005ab0 tx=0x7f508c005fd0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.009+0000 7f509b200640 1 -- 172.21.15.97:0/2971302185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f509c108060 msgr2=0x7f509c1a3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:07.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.009+0000 7f509b200640 1 --2- 172.21.15.97:0/2971302185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f509c108060 0x7f509c1a3e00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f508800e9f0 tx=0x7f508800eeb0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.009+0000 7f509b200640 1 -- 172.21.15.97:0/2971302185 shutdown_connections 2024-03-11T20:10:07.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.009+0000 7f509b200640 1 --2- 172.21.15.97:0/2971302185 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f50800a1ba0 0x7f50800a4060 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.009+0000 7f509b200640 1 --2- 172.21.15.97:0/2971302185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509c10d660 0x7f509c19dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.009+0000 7f509b200640 1 --2- 172.21.15.97:0/2971302185 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f509c108060 0x7f509c1a3e00 secure :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f508800e9f0 tx=0x7f508800eeb0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.009+0000 7f509b200640 1 --2- 172.21.15.97:0/2971302185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f509c1076c0 0x7f509c1a38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:07.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.009+0000 7f509b200640 1 -- 172.21.15.97:0/2971302185 >> 172.21.15.97:0/2971302185 conn(0x7f509c076fa0 msgr2=0x7f509c078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:07.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.010+0000 7f509b200640 1 -- 172.21.15.97:0/2971302185 shutdown_connections 2024-03-11T20:10:08.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.011+0000 7f509b200640 1 -- 172.21.15.97:0/2971302185 wait complete. 2024-03-11T20:10:08.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-11T20:10:08.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.227+0000 7f14da800640 1 -- 172.21.15.97:0/733194884 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14d4103590 msgr2=0x7f14d4103970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:08.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.227+0000 7f14da800640 1 --2- 172.21.15.97:0/733194884 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14d4103590 0x7f14d4103970 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f14c4009930 tx=0x7f14c402bc30 comp rx=0 tx=0).stop 2024-03-11T20:10:08.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.227+0000 7f14da800640 1 -- 172.21.15.97:0/733194884 shutdown_connections 2024-03-11T20:10:08.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.227+0000 7f14da800640 1 --2- 172.21.15.97:0/733194884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14d4108660 0x7f14d4110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:08.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.227+0000 7f14da800640 1 --2- 172.21.15.97:0/733194884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14d4103f40 0x7f14d4107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:08.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.227+0000 7f14da800640 1 --2- 172.21.15.97:0/733194884 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14d4103590 0x7f14d4103970 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:08.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.227+0000 7f14da800640 1 -- 172.21.15.97:0/733194884 >> 172.21.15.97:0/733194884 conn(0x7f14d40fee30 msgr2=0x7f14d4101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:08.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.228+0000 7f14da800640 1 -- 172.21.15.97:0/733194884 shutdown_connections 2024-03-11T20:10:08.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.228+0000 7f14da800640 1 -- 172.21.15.97:0/733194884 wait complete. 2024-03-11T20:10:08.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.229+0000 7f14da800640 1 Processor -- start 2024-03-11T20:10:08.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.229+0000 7f14da800640 1 -- start start 2024-03-11T20:10:08.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.229+0000 7f14da800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14d4103590 0x7f14d419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:08.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.229+0000 7f14da800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14d4108660 0x7f14d419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:08.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.229+0000 7f14da800640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14d41998a0 0x7f14d4199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:08.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.229+0000 7f14da800640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d419a290 con 0x7f14d4108660 2024-03-11T20:10:08.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.229+0000 7f14da800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d419a400 con 0x7f14d4103590 2024-03-11T20:10:08.220 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.229+0000 7f14da800640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d419a5c0 con 0x7f14d41998a0 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.229+0000 7f14d2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14d4108660 0x7f14d419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.229+0000 7f14d2a00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14d4108660 0x7f14d419fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45130/0 (socket says 172.21.15.97:45130) 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.229+0000 7f14d2a00640 1 -- 172.21.15.97:0/2673427027 learned_addr learned my addr 172.21.15.97:0/2673427027 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.229+0000 7f14d3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14d4103590 0x7f14d419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.229+0000 7f14d3e00640 1 --2- 172.21.15.97:0/2673427027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14d41998a0 0x7f14d4199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.230+0000 7f14d2a00640 1 -- 172.21.15.97:0/2673427027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14d41998a0 msgr2=0x7f14d4199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.230+0000 7f14d2a00640 1 --2- 172.21.15.97:0/2673427027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14d41998a0 0x7f14d4199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.230+0000 7f14d2a00640 1 -- 172.21.15.97:0/2673427027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14d4103590 msgr2=0x7f14d419f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.230+0000 7f14d2a00640 1 --2- 172.21.15.97:0/2673427027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14d4103590 0x7f14d419f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.230+0000 7f14d2a00640 1 -- 172.21.15.97:0/2673427027 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c0009490 con 0x7f14d4108660 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.230+0000 7f14d2a00640 1 --2- 172.21.15.97:0/2673427027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14d4108660 0x7f14d419fc30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f14c000e9a0 tx=0x7f14c000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.230+0000 7f14cbe00640 1 -- 172.21.15.97:0/2673427027 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c000cd80 con 0x7f14d4108660 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.230+0000 7f14cbe00640 1 -- 172.21.15.97:0/2673427027 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14c00035f0 con 0x7f14d4108660 2024-03-11T20:10:08.221 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.230+0000 7f14cbe00640 1 -- 172.21.15.97:0/2673427027 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c0010690 con 0x7f14d4108660 2024-03-11T20:10:08.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.230+0000 7f14da800640 1 -- 172.21.15.97:0/2673427027 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c4009560 con 0x7f14d4108660 2024-03-11T20:10:08.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.231+0000 7f14da800640 1 -- 172.21.15.97:0/2673427027 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d41a6950 con 0x7f14d4108660 2024-03-11T20:10:08.222 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.232+0000 7f14da800640 1 -- 172.21.15.97:0/2673427027 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d40683c0 con 0x7f14d4108660 2024-03-11T20:10:08.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.237+0000 7f14cbe00640 1 -- 172.21.15.97:0/2673427027 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f14c0003ad0 con 0x7f14d4108660 2024-03-11T20:10:08.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.238+0000 7f14cbe00640 1 --2- 172.21.15.97:0/2673427027 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f14b80a1c20 0x7f14b80a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:08.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.239+0000 7f14cbe00640 1 -- 172.21.15.97:0/2673427027 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9875+0+0 (secure 0 0 0) 0x7f14c0014070 con 0x7f14d4108660 2024-03-11T20:10:08.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.239+0000 7f14d3400640 1 --2- 172.21.15.97:0/2673427027 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f14b80a1c20 0x7f14b80a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:08.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.240+0000 7f14d3400640 1 --2- 172.21.15.97:0/2673427027 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f14b80a1c20 0x7f14b80a40e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f14c4009900 tx=0x7f14c4005a20 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:08.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.243+0000 7f14cbe00640 1 -- 172.21.15.97:0/2673427027 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f14c0096ad0 con 0x7f14d4108660 2024-03-11T20:10:08.400 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a[31894]: 2024-03-11T20:10:08.002+0000 7f3a6cc00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T20:10:08.400 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/940373968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T20:10:08.400 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/282622601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:10:08.400 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2971302185' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-11T20:10:08.400 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-mon[31898]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T20:10:08.401 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-mon[31898]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T20:10:08.401 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2971302185' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-11T20:10:08.401 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-mon[31898]: osdmap e103: 12 total, 12 up, 12 in 2024-03-11T20:10:08.401 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-mon[31898]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-11T20:10:08.401 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-a[73765]: ignoring --setuser ceph since I am not root 2024-03-11T20:10:08.401 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-a[73765]: ignoring --setgroup ceph since I am not root 2024-03-11T20:10:08.401 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-mds[73787]: -- 172.21.15.97:0/2445704787 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557a53b661a0 con 0x557a53b3c800 2024-03-11T20:10:08.401 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-a[73765]: starting mds.a at 2024-03-11T20:10:08.402 INFO:journalctl@ceph.mds.d.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-d[72782]: ignoring --setuser ceph since I am not root 2024-03-11T20:10:08.402 INFO:journalctl@ceph.mds.d.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-d[72782]: ignoring --setgroup ceph since I am not root 2024-03-11T20:10:08.402 INFO:journalctl@ceph.mds.d.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-mds[72810]: -- 172.21.15.97:0/3019981808 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56071b85e1a0 con 0x56071b835000 2024-03-11T20:10:08.402 INFO:journalctl@ceph.mds.d.smithi097.stdout:Mar 11 20:10:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-d[72782]: starting mds.d at 2024-03-11T20:10:08.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:08.484+0000 7f14da800640 1 -- 172.21.15.97:0/2673427027 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f14d4103f40 con 0x7f14d4108660 2024-03-11T20:10:08.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/940373968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T20:10:08.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/282622601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:10:08.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2971302185' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-11T20:10:08.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-mon[36376]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T20:10:08.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-mon[36376]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T20:10:08.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2971302185' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-11T20:10:08.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-mon[36376]: osdmap e103: 12 total, 12 up, 12 in 2024-03-11T20:10:08.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-mon[36376]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-11T20:10:08.479 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-h[66370]: ignoring --setuser ceph since I am not root 2024-03-11T20:10:08.479 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-h[66370]: ignoring --setgroup ceph since I am not root 2024-03-11T20:10:08.479 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-h[66370]: starting mds.h at 2024-03-11T20:10:08.479 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b[66693]: ignoring --setuser ceph since I am not root 2024-03-11T20:10:08.479 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b[66693]: ignoring --setgroup ceph since I am not root 2024-03-11T20:10:08.479 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b[66693]: starting mds.b at 2024-03-11T20:10:08.480 INFO:journalctl@ceph.mds.e.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-e[66056]: ignoring --setuser ceph since I am not root 2024-03-11T20:10:08.480 INFO:journalctl@ceph.mds.e.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-e[66056]: ignoring --setgroup ceph since I am not root 2024-03-11T20:10:08.480 INFO:journalctl@ceph.mds.e.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-mds[66060]: -- 172.21.15.114:0/3327079562 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56121d9ee1a0 con 0x56121d9c4800 2024-03-11T20:10:08.480 INFO:journalctl@ceph.mds.e.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-e[66056]: starting mds.e at 2024-03-11T20:10:08.480 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-k[65736]: ignoring --setuser ceph since I am not root 2024-03-11T20:10:08.480 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-k[65736]: ignoring --setgroup ceph since I am not root 2024-03-11T20:10:08.480 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-mds[65740]: -- 172.21.15.114:0/2291785491 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dae08101a0 con 0x55dae07e7000 2024-03-11T20:10:08.480 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 20:10:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-k[65736]: starting mds.k at 2024-03-11T20:10:08.524 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-c[67247]: ignoring --setuser ceph since I am not root 2024-03-11T20:10:08.525 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-c[67247]: ignoring --setgroup ceph since I am not root 2024-03-11T20:10:08.525 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-c[67247]: starting mds.c at 2024-03-11T20:10:08.525 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-f[66940]: ignoring --setuser ceph since I am not root 2024-03-11T20:10:08.525 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-f[66940]: ignoring --setgroup ceph since I am not root 2024-03-11T20:10:08.525 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-mds[66944]: -- 172.21.15.161:0/837853999 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56025cecc1a0 con 0x56025cea3000 2024-03-11T20:10:08.525 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-f[66940]: starting mds.f at 2024-03-11T20:10:08.525 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-i[67556]: ignoring --setuser ceph since I am not root 2024-03-11T20:10:08.526 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-i[67556]: ignoring --setgroup ceph since I am not root 2024-03-11T20:10:08.526 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-mds[67560]: -- 172.21.15.161:0/83968861 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55733dce01a0 con 0x55733dcb7000 2024-03-11T20:10:08.526 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-i[67556]: starting mds.i at 2024-03-11T20:10:08.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/940373968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T20:10:08.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/282622601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:10:08.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2971302185' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-11T20:10:08.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-mon[36332]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T20:10:08.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-mon[36332]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T20:10:08.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2971302185' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-11T20:10:08.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-mon[36332]: osdmap e103: 12 total, 12 up, 12 in 2024-03-11T20:10:08.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-mon[36332]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-11T20:10:08.527 INFO:journalctl@ceph.mds.l.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-l[66632]: ignoring --setuser ceph since I am not root 2024-03-11T20:10:08.527 INFO:journalctl@ceph.mds.l.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-l[66632]: ignoring --setgroup ceph since I am not root 2024-03-11T20:10:08.527 INFO:journalctl@ceph.mds.l.smithi161.stdout:Mar 11 20:10:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-l[66632]: starting mds.l at 2024-03-11T20:10:09.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.072+0000 7f14cbe00640 1 -- 172.21.15.97:0/2673427027 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f14c0003760 con 0x7f14d4108660 2024-03-11T20:10:09.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.076+0000 7f14da800640 1 -- 172.21.15.97:0/2673427027 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f14b80a1c20 msgr2=0x7f14b80a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:09.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.076+0000 7f14da800640 1 --2- 172.21.15.97:0/2673427027 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f14b80a1c20 0x7f14b80a40e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f14c4009900 tx=0x7f14c4005a20 comp rx=0 tx=0).stop 2024-03-11T20:10:09.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.076+0000 7f14da800640 1 -- 172.21.15.97:0/2673427027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14d4108660 msgr2=0x7f14d419fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:09.065 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.076+0000 7f14da800640 1 --2- 172.21.15.97:0/2673427027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14d4108660 0x7f14d419fc30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f14c000e9a0 tx=0x7f14c000ee60 comp rx=0 tx=0).stop 2024-03-11T20:10:09.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.077+0000 7f14da800640 1 -- 172.21.15.97:0/2673427027 shutdown_connections 2024-03-11T20:10:09.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.077+0000 7f14da800640 1 --2- 172.21.15.97:0/2673427027 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f14b80a1c20 0x7f14b80a40e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:09.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.077+0000 7f14da800640 1 --2- 172.21.15.97:0/2673427027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14d41998a0 0x7f14d4199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:09.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.077+0000 7f14da800640 1 --2- 172.21.15.97:0/2673427027 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f14d4108660 0x7f14d419fc30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:09.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.077+0000 7f14da800640 1 --2- 172.21.15.97:0/2673427027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f14d4103590 0x7f14d419f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:09.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.077+0000 7f14da800640 1 -- 172.21.15.97:0/2673427027 >> 172.21.15.97:0/2673427027 conn(0x7f14d40fee30 msgr2=0x7f14d4101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:09.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.077+0000 7f14da800640 1 -- 172.21.15.97:0/2673427027 shutdown_connections 2024-03-11T20:10:09.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.077+0000 7f14da800640 1 -- 172.21.15.97:0/2673427027 wait complete. 2024-03-11T20:10:09.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-11T20:10:09.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.287+0000 7fac57200640 1 -- 172.21.15.97:0/117375447 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac501008c0 msgr2=0x7fac50104d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:09.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.287+0000 7fac57200640 1 --2- 172.21.15.97:0/117375447 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac501008c0 0x7fac50104d80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fac40009900 tx=0x7fac4002bd10 comp rx=0 tx=0).stop 2024-03-11T20:10:09.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.287+0000 7fac57200640 1 -- 172.21.15.97:0/117375447 shutdown_connections 2024-03-11T20:10:09.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.287+0000 7fac57200640 1 --2- 172.21.15.97:0/117375447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac501053a0 0x7fac50105830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:09.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.287+0000 7fac57200640 1 --2- 172.21.15.97:0/117375447 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac501008c0 0x7fac50104d80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:09.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.287+0000 7fac57200640 1 --2- 172.21.15.97:0/117375447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500fffa0 0x7fac50100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:09.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.287+0000 7fac57200640 1 -- 172.21.15.97:0/117375447 >> 172.21.15.97:0/117375447 conn(0x7fac500fb9d0 msgr2=0x7fac500fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:09.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.287+0000 7fac57200640 1 -- 172.21.15.97:0/117375447 shutdown_connections 2024-03-11T20:10:09.276 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.287+0000 7fac57200640 1 -- 172.21.15.97:0/117375447 wait complete. 2024-03-11T20:10:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.288+0000 7fac57200640 1 Processor -- start 2024-03-11T20:10:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.288+0000 7fac57200640 1 -- start start 2024-03-11T20:10:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.288+0000 7fac57200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500fffa0 0x7fac501a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.288+0000 7fac57200640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac501008c0 0x7fac501a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.288+0000 7fac57200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac501053a0 0x7fac5019dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.288+0000 7fac57200640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5019e100 con 0x7fac501053a0 2024-03-11T20:10:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.288+0000 7fac57200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5019e270 con 0x7fac500fffa0 2024-03-11T20:10:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.288+0000 7fac57200640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5019e430 con 0x7fac501008c0 2024-03-11T20:10:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.289+0000 7fac56800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac501053a0 0x7fac5019dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.289+0000 7fac55e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500fffa0 0x7fac501a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.289+0000 7fac56800640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac501053a0 0x7fac5019dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45138/0 (socket says 172.21.15.97:45138) 2024-03-11T20:10:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.289+0000 7fac55e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500fffa0 0x7fac501a3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.97:36110/0 (socket says 172.21.15.97:36110) 2024-03-11T20:10:09.279 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.289+0000 7fac56800640 1 -- 172.21.15.97:0/4239932235 learned_addr learned my addr 172.21.15.97:0/4239932235 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.289+0000 7fac55400640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac501008c0 0x7fac501a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.289+0000 7fac55e00640 1 -- 172.21.15.97:0/4239932235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac501008c0 msgr2=0x7fac501a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.289+0000 7fac55e00640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac501008c0 0x7fac501a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.289+0000 7fac55e00640 1 -- 172.21.15.97:0/4239932235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac501053a0 msgr2=0x7fac5019dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.289+0000 7fac55e00640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac501053a0 0x7fac5019dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.289+0000 7fac55e00640 1 -- 172.21.15.97:0/4239932235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac40009560 con 0x7fac500fffa0 2024-03-11T20:10:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.289+0000 7fac56800640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac501053a0 0x7fac5019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:10:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.289+0000 7fac55e00640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500fffa0 0x7fac501a3900 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fac4400e9f0 tx=0x7fac4400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.290+0000 7fac4aa00640 1 -- 172.21.15.97:0/4239932235 <== mon.2 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4400ccc0 con 0x7fac500fffa0 2024-03-11T20:10:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.290+0000 7fac4aa00640 1 -- 172.21.15.97:0/4239932235 <== mon.2 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac4400ce20 con 0x7fac500fffa0 2024-03-11T20:10:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.290+0000 7fac4aa00640 1 -- 172.21.15.97:0/4239932235 <== mon.2 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac44018920 con 0x7fac500fffa0 2024-03-11T20:10:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.290+0000 7fac57200640 1 -- 172.21.15.97:0/4239932235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5019e710 con 0x7fac500fffa0 2024-03-11T20:10:09.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.290+0000 7fac57200640 1 -- 172.21.15.97:0/4239932235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac501aa9a0 con 0x7fac500fffa0 2024-03-11T20:10:09.281 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.291+0000 7fac57200640 1 -- 172.21.15.97:0/4239932235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac50067e90 con 0x7fac500fffa0 2024-03-11T20:10:09.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.294+0000 7fac4aa00640 1 -- 172.21.15.97:0/4239932235 <== mon.2 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fac44018a80 con 0x7fac500fffa0 2024-03-11T20:10:09.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.296+0000 7fac4aa00640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fac380a1ba0 0x7fac380a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:09.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.296+0000 7fac4aa00640 1 -- 172.21.15.97:0/4239932235 <== mon.2 v2:172.21.15.114:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9875+0+0 (secure 0 0 0) 0x7fac44014070 con 0x7fac500fffa0 2024-03-11T20:10:09.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.296+0000 7fac55400640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fac380a1ba0 0x7fac380a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:09.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.296+0000 7fac55400640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fac380a1ba0 0x7fac380a4060 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fac40005ab0 tx=0x7fac40005a20 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:09.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.300+0000 7fac4aa00640 1 -- 172.21.15.97:0/4239932235 <== mon.2 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fac44052d20 con 0x7fac500fffa0 2024-03-11T20:10:09.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:09 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2673427027' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:10:09.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:09 smithi097 ceph-mon[31898]: pgmap v338: 129 pgs: 129 active+clean; 2.3 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 32 op/s 2024-03-11T20:10:09.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:09 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2673427027' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:10:09.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:09 smithi114 ceph-mon[36376]: pgmap v338: 129 pgs: 129 active+clean; 2.3 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 32 op/s 2024-03-11T20:10:09.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:09.496+0000 7fac57200640 1 -- 172.21.15.97:0/4239932235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fac50117770 con 0x7fac500fffa0 2024-03-11T20:10:09.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:09 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2673427027' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:10:09.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:09 smithi161 ceph-mon[36332]: pgmap v338: 129 pgs: 129 active+clean; 2.3 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 32 op/s 2024-03-11T20:10:10.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.093+0000 7fac4aa00640 1 -- 172.21.15.97:0/4239932235 <== mon.2 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v104) v1 ==== 181+0+0 (secure 0 0 0) 0x7fac4409a540 con 0x7fac500fffa0 2024-03-11T20:10:10.082 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_metadata' removed 2024-03-11T20:10:10.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.097+0000 7fac57200640 1 -- 172.21.15.97:0/4239932235 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fac380a1ba0 msgr2=0x7fac380a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:10.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.097+0000 7fac57200640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fac380a1ba0 0x7fac380a4060 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fac40005ab0 tx=0x7fac40005a20 comp rx=0 tx=0).stop 2024-03-11T20:10:10.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.098+0000 7fac57200640 1 -- 172.21.15.97:0/4239932235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500fffa0 msgr2=0x7fac501a3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:10.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.098+0000 7fac57200640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500fffa0 0x7fac501a3900 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fac4400e9f0 tx=0x7fac4400eeb0 comp rx=0 tx=0).stop 2024-03-11T20:10:10.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.098+0000 7fac57200640 1 -- 172.21.15.97:0/4239932235 shutdown_connections 2024-03-11T20:10:10.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.098+0000 7fac57200640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fac380a1ba0 0x7fac380a4060 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:10.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.098+0000 7fac57200640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac501053a0 0x7fac5019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:10.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.098+0000 7fac57200640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac501008c0 0x7fac501a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:10.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.098+0000 7fac57200640 1 --2- 172.21.15.97:0/4239932235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac500fffa0 0x7fac501a3900 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:10.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.098+0000 7fac57200640 1 -- 172.21.15.97:0/4239932235 >> 172.21.15.97:0/4239932235 conn(0x7fac500fb9d0 msgr2=0x7fac50102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:10.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.098+0000 7fac57200640 1 -- 172.21.15.97:0/4239932235 shutdown_connections 2024-03-11T20:10:10.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.098+0000 7fac57200640 1 -- 172.21.15.97:0/4239932235 wait complete. 2024-03-11T20:10:10.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-11T20:10:10.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.326+0000 7f5f24e00640 1 -- 172.21.15.97:0/4243157694 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f20103470 msgr2=0x7f5f20103850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:10.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.326+0000 7f5f24e00640 1 --2- 172.21.15.97:0/4243157694 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f20103470 0x7f5f20103850 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009930 tx=0x7f5f1002bc30 comp rx=0 tx=0).stop 2024-03-11T20:10:10.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.326+0000 7f5f24e00640 1 -- 172.21.15.97:0/4243157694 shutdown_connections 2024-03-11T20:10:10.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.326+0000 7f5f24e00640 1 --2- 172.21.15.97:0/4243157694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f20108540 0x7f5f20110110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:10.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.326+0000 7f5f24e00640 1 --2- 172.21.15.97:0/4243157694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f20103e20 0x7f5f20107e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:10.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.326+0000 7f5f24e00640 1 --2- 172.21.15.97:0/4243157694 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f20103470 0x7f5f20103850 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:10.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.326+0000 7f5f24e00640 1 -- 172.21.15.97:0/4243157694 >> 172.21.15.97:0/4243157694 conn(0x7f5f200fecb0 msgr2=0x7f5f201010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:10.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.326+0000 7f5f24e00640 1 -- 172.21.15.97:0/4243157694 shutdown_connections 2024-03-11T20:10:10.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.326+0000 7f5f24e00640 1 -- 172.21.15.97:0/4243157694 wait complete. 2024-03-11T20:10:10.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.327+0000 7f5f24e00640 1 Processor -- start 2024-03-11T20:10:10.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.327+0000 7f5f24e00640 1 -- start start 2024-03-11T20:10:10.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f24e00640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f20103470 0x7f5f2019f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:10.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f24e00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f20108540 0x7f5f2019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:10.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f24e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f20199720 0x7f5f20199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:10.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f24e00640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2019a1f0 con 0x7f5f20103470 2024-03-11T20:10:10.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f24e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2019a360 con 0x7f5f20199720 2024-03-11T20:10:10.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f24e00640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2019a520 con 0x7f5f20108540 2024-03-11T20:10:10.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f1f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f20103470 0x7f5f2019f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f16a00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f20108540 0x7f5f2019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f1f400640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f20103470 0x7f5f2019f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45156/0 (socket says 172.21.15.97:45156) 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f1f400640 1 -- 172.21.15.97:0/2455354709 learned_addr learned my addr 172.21.15.97:0/2455354709 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f1fe00640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f20199720 0x7f5f20199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f1f400640 1 -- 172.21.15.97:0/2455354709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f20108540 msgr2=0x7f5f2019fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f1f400640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f20108540 0x7f5f2019fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f1f400640 1 -- 172.21.15.97:0/2455354709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f20199720 msgr2=0x7f5f20199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f1f400640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f20199720 0x7f5f20199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f1f400640 1 -- 172.21.15.97:0/2455354709 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f10009560 con 0x7f5f20103470 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.328+0000 7f5f1fe00640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f20199720 0x7f5f20199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.329+0000 7f5f1f400640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f20103470 0x7f5f2019f660 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009230 tx=0x7f5f1002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.329+0000 7f5f1cc00640 1 -- 172.21.15.97:0/2455354709 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10004350 con 0x7f5f20103470 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.329+0000 7f5f1cc00640 1 -- 172.21.15.97:0/2455354709 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f100044b0 con 0x7f5f20103470 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.329+0000 7f5f24e00640 1 -- 172.21.15.97:0/2455354709 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f201a6460 con 0x7f5f20103470 2024-03-11T20:10:10.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.329+0000 7f5f1cc00640 1 -- 172.21.15.97:0/2455354709 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10003d70 con 0x7f5f20103470 2024-03-11T20:10:10.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.329+0000 7f5f24e00640 1 -- 172.21.15.97:0/2455354709 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f201a6940 con 0x7f5f20103470 2024-03-11T20:10:10.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.330+0000 7f5f24e00640 1 -- 172.21.15.97:0/2455354709 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f20104940 con 0x7f5f20103470 2024-03-11T20:10:10.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.332+0000 7f5f1cc00640 1 -- 172.21.15.97:0/2455354709 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5f10004df0 con 0x7f5f20103470 2024-03-11T20:10:10.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.333+0000 7f5f1cc00640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f5f040a1ba0 0x7f5f040a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:10.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.334+0000 7f5f1cc00640 1 -- 172.21.15.97:0/2455354709 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9342+0+0 (secure 0 0 0) 0x7f5f100efbb0 con 0x7f5f20103470 2024-03-11T20:10:10.322 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.334+0000 7f5f16a00640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f5f040a1ba0 0x7f5f040a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:10.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.334+0000 7f5f16a00640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f5f040a1ba0 0x7f5f040a4060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5f2019af70 tx=0x7f5f0800b390 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:10.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.338+0000 7f5f1cc00640 1 -- 172.21.15.97:0/2455354709 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5f100b8800 con 0x7f5f20103470 2024-03-11T20:10:10.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T20:10:10.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: Cluster is now healthy 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2673427027' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.161:6833/2880085155,v1:172.21.15.161:6835/2880085155] up:boot 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.114:6836/1324688731,v1:172.21.15.114:6837/1324688731] up:boot 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.161:6832/1057052333,v1:172.21.15.161:6834/1057052333] up:boot 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.114:6833/133447769,v1:172.21.15.114:6835/133447769] up:boot 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.114:6832/4037028954,v1:172.21.15.114:6834/4037028954] up:boot 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.161:6837/2862939556,v1:172.21.15.161:6839/2862939556] up:boot 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.114:6838/3323839850,v1:172.21.15.114:6839/3323839850] up:boot 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.161:6836/91897911,v1:172.21.15.161:6838/91897911] up:boot 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.97:6836/3319512217,v1:172.21.15.97:6837/3319512217] up:boot 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: mds.? [v2:172.21.15.97:6840/2284620822,v1:172.21.15.97:6841/2284620822] up:boot 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: fsmap 12 up:standby 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T20:10:10.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T20:10:10.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T20:10:10.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T20:10:10.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T20:10:10.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T20:10:10.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T20:10:10.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T20:10:10.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T20:10:10.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T20:10:10.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/4239932235' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:10:10.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:10 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:10:10.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T20:10:10.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T20:10:10.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: Cluster is now healthy 2024-03-11T20:10:10.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2673427027' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:10:10.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.161:6833/2880085155,v1:172.21.15.161:6835/2880085155] up:boot 2024-03-11T20:10:10.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.114:6836/1324688731,v1:172.21.15.114:6837/1324688731] up:boot 2024-03-11T20:10:10.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.161:6832/1057052333,v1:172.21.15.161:6834/1057052333] up:boot 2024-03-11T20:10:10.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.114:6833/133447769,v1:172.21.15.114:6835/133447769] up:boot 2024-03-11T20:10:10.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.114:6832/4037028954,v1:172.21.15.114:6834/4037028954] up:boot 2024-03-11T20:10:10.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.161:6837/2862939556,v1:172.21.15.161:6839/2862939556] up:boot 2024-03-11T20:10:10.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.114:6838/3323839850,v1:172.21.15.114:6839/3323839850] up:boot 2024-03-11T20:10:10.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.161:6836/91897911,v1:172.21.15.161:6838/91897911] up:boot 2024-03-11T20:10:10.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.97:6836/3319512217,v1:172.21.15.97:6837/3319512217] up:boot 2024-03-11T20:10:10.479 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: mds.? [v2:172.21.15.97:6840/2284620822,v1:172.21.15.97:6841/2284620822] up:boot 2024-03-11T20:10:10.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: fsmap 12 up:standby 2024-03-11T20:10:10.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T20:10:10.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T20:10:10.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T20:10:10.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T20:10:10.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T20:10:10.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T20:10:10.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T20:10:10.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T20:10:10.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T20:10:10.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T20:10:10.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/4239932235' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:10:10.480 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:10 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:10:10.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:10.534+0000 7f5f24e00640 1 -- 172.21.15.97:0/2455354709 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5f2010b770 con 0x7f5f20103470 2024-03-11T20:10:10.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T20:10:10.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T20:10:10.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: Cluster is now healthy 2024-03-11T20:10:10.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2673427027' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:10:10.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.161:6833/2880085155,v1:172.21.15.161:6835/2880085155] up:boot 2024-03-11T20:10:10.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.114:6836/1324688731,v1:172.21.15.114:6837/1324688731] up:boot 2024-03-11T20:10:10.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.161:6832/1057052333,v1:172.21.15.161:6834/1057052333] up:boot 2024-03-11T20:10:10.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.114:6833/133447769,v1:172.21.15.114:6835/133447769] up:boot 2024-03-11T20:10:10.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.114:6832/4037028954,v1:172.21.15.114:6834/4037028954] up:boot 2024-03-11T20:10:10.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.161:6837/2862939556,v1:172.21.15.161:6839/2862939556] up:boot 2024-03-11T20:10:10.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.114:6838/3323839850,v1:172.21.15.114:6839/3323839850] up:boot 2024-03-11T20:10:10.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.161:6836/91897911,v1:172.21.15.161:6838/91897911] up:boot 2024-03-11T20:10:10.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.97:6836/3319512217,v1:172.21.15.97:6837/3319512217] up:boot 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: mds.? [v2:172.21.15.97:6840/2284620822,v1:172.21.15.97:6841/2284620822] up:boot 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: fsmap 12 up:standby 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: from='mgr.15396 172.21.15.97:0/3559001974' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/4239932235' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:10:10.526 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:10 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:10:11.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.103+0000 7f5f1cc00640 1 -- 172.21.15.97:0/2455354709 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v105) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5f100bbec0 con 0x7f5f20103470 2024-03-11T20:10:11.092 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data' removed 2024-03-11T20:10:11.097 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.109+0000 7f5f24e00640 1 -- 172.21.15.97:0/2455354709 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f5f040a1ba0 msgr2=0x7f5f040a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.109+0000 7f5f24e00640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f5f040a1ba0 0x7f5f040a4060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5f2019af70 tx=0x7f5f0800b390 comp rx=0 tx=0).stop 2024-03-11T20:10:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.109+0000 7f5f24e00640 1 -- 172.21.15.97:0/2455354709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f20103470 msgr2=0x7f5f2019f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.109+0000 7f5f24e00640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f20103470 0x7f5f2019f660 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009230 tx=0x7f5f1002c6c0 comp rx=0 tx=0).stop 2024-03-11T20:10:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.109+0000 7f5f24e00640 1 -- 172.21.15.97:0/2455354709 shutdown_connections 2024-03-11T20:10:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.109+0000 7f5f24e00640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7f5f040a1ba0 0x7f5f040a4060 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.109+0000 7f5f24e00640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f20199720 0x7f5f20199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.109+0000 7f5f24e00640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5f20108540 0x7f5f2019fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.109+0000 7f5f24e00640 1 --2- 172.21.15.97:0/2455354709 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5f20103470 0x7f5f2019f660 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.109+0000 7f5f24e00640 1 -- 172.21.15.97:0/2455354709 >> 172.21.15.97:0/2455354709 conn(0x7f5f200fecb0 msgr2=0x7f5f20100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.109+0000 7f5f24e00640 1 -- 172.21.15.97:0/2455354709 shutdown_connections 2024-03-11T20:10:11.098 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.109+0000 7f5f24e00640 1 -- 172.21.15.97:0/2455354709 wait complete. 2024-03-11T20:10:11.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-11T20:10:11.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.334+0000 7fbf8e600640 1 -- 172.21.15.97:0/4084836849 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf8810d970 msgr2=0x7fbf8810fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:11.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.334+0000 7fbf8e600640 1 --2- 172.21.15.97:0/4084836849 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf8810d970 0x7fbf8810fd60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009ca0 tx=0x7fbf8002c100 comp rx=0 tx=0).stop 2024-03-11T20:10:11.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.334+0000 7fbf8e600640 1 -- 172.21.15.97:0/4084836849 shutdown_connections 2024-03-11T20:10:11.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.334+0000 7fbf8e600640 1 --2- 172.21.15.97:0/4084836849 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf8810d970 0x7fbf8810fd60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:11.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.334+0000 7fbf8e600640 1 --2- 172.21.15.97:0/4084836849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf88100770 0x7fbf8810d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:11.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.334+0000 7fbf8e600640 1 --2- 172.21.15.97:0/4084836849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf880ffdc0 0x7fbf881001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:11.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.334+0000 7fbf8e600640 1 -- 172.21.15.97:0/4084836849 >> 172.21.15.97:0/4084836849 conn(0x7fbf880fb9d0 msgr2=0x7fbf880fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:11.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.334+0000 7fbf8e600640 1 -- 172.21.15.97:0/4084836849 shutdown_connections 2024-03-11T20:10:11.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.334+0000 7fbf8e600640 1 -- 172.21.15.97:0/4084836849 wait complete. 2024-03-11T20:10:11.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.335+0000 7fbf8e600640 1 Processor -- start 2024-03-11T20:10:11.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.335+0000 7fbf8e600640 1 -- start start 2024-03-11T20:10:11.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.335+0000 7fbf8e600640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf880ffdc0 0x7fbf8819f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:11.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.335+0000 7fbf8e600640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf88100770 0x7fbf8819fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:11.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.335+0000 7fbf8e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8810d970 0x7fbf88199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:11.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.335+0000 7fbf8e600640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf88199e10 con 0x7fbf880ffdc0 2024-03-11T20:10:11.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.335+0000 7fbf8e600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf88199f80 con 0x7fbf8810d970 2024-03-11T20:10:11.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.335+0000 7fbf8e600640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8819a140 con 0x7fbf88100770 2024-03-11T20:10:11.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.336+0000 7fbf8d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf880ffdc0 0x7fbf8819f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:11.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.336+0000 7fbf8d200640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf880ffdc0 0x7fbf8819f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:45178/0 (socket says 172.21.15.97:45178) 2024-03-11T20:10:11.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.336+0000 7fbf8d200640 1 -- 172.21.15.97:0/243851154 learned_addr learned my addr 172.21.15.97:0/243851154 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-03-11T20:10:11.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.336+0000 7fbf8dc00640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8810d970 0x7fbf88199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:11.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.336+0000 7fbf87e00640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf88100770 0x7fbf8819fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:11.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.336+0000 7fbf8d200640 1 -- 172.21.15.97:0/243851154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf88100770 msgr2=0x7fbf8819fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:11.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.336+0000 7fbf8d200640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf88100770 0x7fbf8819fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:11.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.336+0000 7fbf8d200640 1 -- 172.21.15.97:0/243851154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8810d970 msgr2=0x7fbf88199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:11.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.336+0000 7fbf8d200640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8810d970 0x7fbf88199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:11.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.336+0000 7fbf8d200640 1 -- 172.21.15.97:0/243851154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf80009900 con 0x7fbf880ffdc0 2024-03-11T20:10:11.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.336+0000 7fbf8d200640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf880ffdc0 0x7fbf8819f570 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00e9f0 tx=0x7fbf7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:11.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.337+0000 7fbf85600640 1 -- 172.21.15.97:0/243851154 <== mon.0 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c009700 con 0x7fbf880ffdc0 2024-03-11T20:10:11.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.337+0000 7fbf85600640 1 -- 172.21.15.97:0/243851154 <== mon.0 v2:172.21.15.97:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf7c00cd80 con 0x7fbf880ffdc0 2024-03-11T20:10:11.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.337+0000 7fbf8e600640 1 -- 172.21.15.97:0/243851154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8819a420 con 0x7fbf880ffdc0 2024-03-11T20:10:11.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.337+0000 7fbf85600640 1 -- 172.21.15.97:0/243851154 <== mon.0 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c018920 con 0x7fbf880ffdc0 2024-03-11T20:10:11.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.337+0000 7fbf8e600640 1 -- 172.21.15.97:0/243851154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf881a64f0 con 0x7fbf880ffdc0 2024-03-11T20:10:11.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.338+0000 7fbf8e600640 1 -- 172.21.15.97:0/243851154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf88068bc0 con 0x7fbf880ffdc0 2024-03-11T20:10:11.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.340+0000 7fbf85600640 1 -- 172.21.15.97:0/243851154 <== mon.0 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fbf7c018a80 con 0x7fbf880ffdc0 2024-03-11T20:10:11.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.341+0000 7fbf85600640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fbf700a1ba0 0x7fbf700a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:10:11.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.341+0000 7fbf85600640 1 -- 172.21.15.97:0/243851154 <== mon.0 v2:172.21.15.97:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 8754+0+0 (secure 0 0 0) 0x7fbf7c014070 con 0x7fbf880ffdc0 2024-03-11T20:10:11.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.342+0000 7fbf87e00640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fbf700a1ba0 0x7fbf700a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:10:11.331 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.342+0000 7fbf87e00640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fbf700a1ba0 0x7fbf700a4060 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbf8819ae80 tx=0x7fbf74005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:10:11.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.346+0000 7fbf85600640 1 -- 172.21.15.97:0/243851154 <== mon.0 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbf7c052d20 con 0x7fbf880ffdc0 2024-03-11T20:10:11.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:11 smithi097 ceph-mon[31898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T20:10:11.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:11 smithi097 ceph-mon[31898]: osdmap e104: 12 total, 12 up, 12 in 2024-03-11T20:10:11.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:11 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2455354709' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:10:11.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:11 smithi097 ceph-mon[31898]: pgmap v340: 97 pgs: 97 active+clean; 577 KiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T20:10:11.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:11 smithi114 ceph-mon[36376]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T20:10:11.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:11 smithi114 ceph-mon[36376]: osdmap e104: 12 total, 12 up, 12 in 2024-03-11T20:10:11.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:11 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2455354709' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:10:11.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:11 smithi114 ceph-mon[36376]: pgmap v340: 97 pgs: 97 active+clean; 577 KiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T20:10:11.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:11 smithi161 ceph-mon[36332]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T20:10:11.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:11 smithi161 ceph-mon[36332]: osdmap e104: 12 total, 12 up, 12 in 2024-03-11T20:10:11.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:11 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2455354709' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:10:11.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:11 smithi161 ceph-mon[36332]: pgmap v340: 97 pgs: 97 active+clean; 577 KiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T20:10:11.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:11.550+0000 7fbf8e600640 1 -- 172.21.15.97:0/243851154 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbf8819ab60 con 0x7fbf880ffdc0 2024-03-11T20:10:12.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:12.115+0000 7fbf85600640 1 -- 172.21.15.97:0/243851154 <== mon.0 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v106) v1 ==== 178+0+0 (secure 0 0 0) 0x7fbf7c096ae0 con 0x7fbf880ffdc0 2024-03-11T20:10:12.104 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'cephfs_data_ec' removed 2024-03-11T20:10:12.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:12.120+0000 7fbf8e600640 1 -- 172.21.15.97:0/243851154 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fbf700a1ba0 msgr2=0x7fbf700a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:12.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:12.120+0000 7fbf8e600640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fbf700a1ba0 0x7fbf700a4060 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbf8819ae80 tx=0x7fbf74005f90 comp rx=0 tx=0).stop 2024-03-11T20:10:12.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:12.120+0000 7fbf8e600640 1 -- 172.21.15.97:0/243851154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf880ffdc0 msgr2=0x7fbf8819f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:10:12.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:12.120+0000 7fbf8e600640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf880ffdc0 0x7fbf8819f570 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00e9f0 tx=0x7fbf7c00eeb0 comp rx=0 tx=0).stop 2024-03-11T20:10:12.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:12.121+0000 7fbf8e600640 1 -- 172.21.15.97:0/243851154 shutdown_connections 2024-03-11T20:10:12.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:12.121+0000 7fbf8e600640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.97:6800/1085760108,v1:172.21.15.97:6801/1085760108] conn(0x7fbf700a1ba0 0x7fbf700a4060 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:12.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:12.121+0000 7fbf8e600640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf8810d970 0x7fbf88199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:12.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:12.121+0000 7fbf8e600640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf88100770 0x7fbf8819fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:12.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:12.121+0000 7fbf8e600640 1 --2- 172.21.15.97:0/243851154 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbf880ffdc0 0x7fbf8819f570 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:10:12.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:12.121+0000 7fbf8e600640 1 -- 172.21.15.97:0/243851154 >> 172.21.15.97:0/243851154 conn(0x7fbf880fb9d0 msgr2=0x7fbf880fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:10:12.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:12.122+0000 7fbf8e600640 1 -- 172.21.15.97:0/243851154 shutdown_connections 2024-03-11T20:10:12.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-11T20:10:12.122+0000 7fbf8e600640 1 -- 172.21.15.97:0/243851154 wait complete. 2024-03-11T20:10:12.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T20:10:12.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T20:10:12.266 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T20:10:12.303 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T20:10:12.303 DEBUG:teuthology.orchestra.run.smithi097:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T20:10:12.362 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T20:10:12.378 DEBUG:teuthology.orchestra.run.smithi161:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T20:10:12.395 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T20:10:12.396 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T20:10:12.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.a 2024-03-11T20:10:12.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:12 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/2455354709' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T20:10:12.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:12 smithi097 ceph-mon[31898]: osdmap e105: 12 total, 12 up, 12 in 2024-03-11T20:10:12.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:12 smithi097 ceph-mon[31898]: from='client.? 172.21.15.97:0/243851154' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:10:12.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:12 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/2455354709' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T20:10:12.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:12 smithi114 ceph-mon[36376]: osdmap e105: 12 total, 12 up, 12 in 2024-03-11T20:10:12.478 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:12 smithi114 ceph-mon[36376]: from='client.? 172.21.15.97:0/243851154' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:10:12.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:12 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/2455354709' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T20:10:12.524 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:12 smithi161 ceph-mon[36332]: osdmap e105: 12 total, 12 up, 12 in 2024-03-11T20:10:12.525 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:12 smithi161 ceph-mon[36332]: from='client.? 172.21.15.97:0/243851154' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:10:12.906 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:12 smithi097 systemd[1]: Stopping Ceph mon.a for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:10:13.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:12 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a[31894]: 2024-03-11T20:10:12.968+0000 7f3a73a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:10:13.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:12 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a[31894]: 2024-03-11T20:10:12.968+0000 7f3a73a00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T20:10:13.407 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:13 smithi097 podman[93567]: 2024-03-11 20:10:13.125382324 +0000 UTC m=+0.303116283 container died 3c2b428eeaee03d17e0bfd22c392aa61620169f3d699922426710cba08a3a226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T20:10:13.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:13 smithi097 podman[93567]: 2024-03-11 20:10:13.697469822 +0000 UTC m=+0.875203781 container cleanup 3c2b428eeaee03d17e0bfd22c392aa61620169f3d699922426710cba08a3a226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T20:10:13.907 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:13 smithi097 bash[93567]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a 2024-03-11T20:10:14.406 INFO:journalctl@ceph.mon.a.smithi097.stdout:Mar 11 20:10:13 smithi097 podman[93580]: 2024-03-11 20:10:13.949276082 +0000 UTC m=+0.809674000 container remove 3c2b428eeaee03d17e0bfd22c392aa61620169f3d699922426710cba08a3a226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T20:10:14.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T20:10:14.701 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:10:14.701 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T20:10:14.701 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-11T20:10:14.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.b 2024-03-11T20:10:15.047 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:14 smithi114 systemd[1]: Stopping Ceph mon.b for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:10:15.456 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:15 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-b[36372]: 2024-03-11T20:10:15.058+0000 7f352ee30700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:10:15.456 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:15 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-b[36372]: 2024-03-11T20:10:15.058+0000 7f352ee30700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T20:10:15.456 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:15 smithi114 podman[72606]: 2024-03-11 20:10:15.243287747 +0000 UTC m=+0.331582820 container died 02bc52469f3a26ff444c17563b5024af6d00eb9a8d0c3fa261bfa19d5d49458d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-11T20:10:15.728 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:15 smithi114 podman[72606]: 2024-03-11 20:10:15.567243903 +0000 UTC m=+0.655538968 container cleanup 02bc52469f3a26ff444c17563b5024af6d00eb9a8d0c3fa261bfa19d5d49458d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T20:10:15.728 INFO:journalctl@ceph.mon.b.smithi114.stdout:Mar 11 20:10:15 smithi114 bash[72606]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-b 2024-03-11T20:10:15.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T20:10:16.014 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:10:16.014 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-11T20:10:16.015 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-11T20:10:16.015 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.c 2024-03-11T20:10:16.357 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:16 smithi161 systemd[1]: Stopping Ceph mon.c for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:10:16.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:16 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-c[36313]: 2024-03-11T20:10:16.367+0000 7f7acf1c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:10:16.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:16 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-c[36313]: 2024-03-11T20:10:16.367+0000 7f7acf1c9700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T20:10:16.700 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:16 smithi161 podman[73629]: 2024-03-11 20:10:16.582801121 +0000 UTC m=+0.355406162 container died 767d2c35ced1708cdd6b6fbbcd75be93b8c5d91b179c7172174c0710223968a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-11T20:10:17.269 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:16 smithi161 podman[73629]: 2024-03-11 20:10:16.987148265 +0000 UTC m=+0.759753297 container cleanup 767d2c35ced1708cdd6b6fbbcd75be93b8c5d91b179c7172174c0710223968a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-94a5397) 2024-03-11T20:10:17.269 INFO:journalctl@ceph.mon.c.smithi161.stdout:Mar 11 20:10:16 smithi161 bash[73629]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mon-c 2024-03-11T20:10:17.386 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T20:10:17.433 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:10:17.433 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-11T20:10:17.434 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T20:10:17.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.x 2024-03-11T20:10:17.906 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 20:10:17 smithi097 systemd[1]: Stopping Ceph mgr.x for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:10:18.410 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 20:10:18 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T20:10:18.025+0000 7f801a200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:10:18.410 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 20:10:18 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x[32126]: 2024-03-11T20:10:18.025+0000 7f801a200700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-11T20:10:18.410 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 20:10:18 smithi097 podman[93685]: 2024-03-11 20:10:18.130113762 +0000 UTC m=+0.257505665 container died 69fd8238897fc4c56eccee52ceb85a235f2a09dd7c33bf2cf802e2f5ff7b6ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-11T20:10:18.906 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 20:10:18 smithi097 podman[93685]: 2024-03-11 20:10:18.720057237 +0000 UTC m=+0.847449163 container cleanup 69fd8238897fc4c56eccee52ceb85a235f2a09dd7c33bf2cf802e2f5ff7b6ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T20:10:18.907 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 20:10:18 smithi097 bash[93685]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x 2024-03-11T20:10:19.386 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 20:10:18 smithi097 podman[93699]: 2024-03-11 20:10:18.921388231 +0000 UTC m=+0.772308352 container remove 69fd8238897fc4c56eccee52ceb85a235f2a09dd7c33bf2cf802e2f5ff7b6ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-03-11T20:10:19.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T20:10:19.527 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:10:19.528 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T20:10:19.528 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T20:10:19.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.y 2024-03-11T20:10:19.855 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 20:10:19 smithi114 systemd[1]: Stopping Ceph mgr.y for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:10:20.176 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 20:10:19 smithi114 podman[72740]: 2024-03-11 20:10:19.867310262 +0000 UTC m=+0.145958236 container died 45ae7a66dab3b7dcf23002aca70a26b2d2285c855905cbecd9fccdc38cef8706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-11T20:10:20.478 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 20:10:20 smithi114 podman[72740]: 2024-03-11 20:10:20.188867346 +0000 UTC m=+0.467515311 container cleanup 45ae7a66dab3b7dcf23002aca70a26b2d2285c855905cbecd9fccdc38cef8706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS) 2024-03-11T20:10:20.478 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 20:10:20 smithi114 bash[72740]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y 2024-03-11T20:10:20.478 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 20:10:20 smithi114 podman[72754]: 2024-03-11 20:10:20.407093222 +0000 UTC m=+0.535832614 container remove 45ae7a66dab3b7dcf23002aca70a26b2d2285c855905cbecd9fccdc38cef8706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-11T20:10:20.478 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Mar 11 20:10:20 smithi114 systemd[1]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-11T20:10:20.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T20:10:20.681 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:10:20.681 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T20:10:20.681 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-11T20:10:20.681 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.z 2024-03-11T20:10:21.018 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 20:10:20 smithi161 systemd[1]: Stopping Ceph mgr.z for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:10:21.275 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 20:10:21 smithi161 podman[73753]: 2024-03-11 20:10:21.029842163 +0000 UTC m=+0.148006230 container died b71e0c963bb256d1d6cf47fae870b83bb7f934641eedf81f8cd81f4fd543d3bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:10:22.354 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 20:10:21 smithi161 podman[73753]: 2024-03-11 20:10:21.426540266 +0000 UTC m=+0.544704306 container cleanup b71e0c963bb256d1d6cf47fae870b83bb7f934641eedf81f8cd81f4fd543d3bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-11T20:10:22.354 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 20:10:21 smithi161 bash[73753]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z 2024-03-11T20:10:22.355 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.z.service' 2024-03-11T20:10:22.356 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 20:10:21 smithi161 podman[73767]: 2024-03-11 20:10:21.669779589 +0000 UTC m=+0.634571899 container remove b71e0c963bb256d1d6cf47fae870b83bb7f934641eedf81f8cd81f4fd543d3bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T20:10:22.356 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 20:10:21 smithi161 systemd[1]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-11T20:10:22.356 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 20:10:21 smithi161 systemd[1]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.z.service: Failed with result 'exit-code'. 2024-03-11T20:10:22.356 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 20:10:21 smithi161 systemd[1]: Stopped Ceph mgr.z for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T20:10:22.356 INFO:journalctl@ceph.mgr.z.smithi161.stdout:Mar 11 20:10:21 smithi161 systemd[1]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mgr.z.service: Consumed 20.909s CPU time. 2024-03-11T20:10:22.394 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:10:22.394 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-11T20:10:22.394 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T20:10:22.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.0 2024-03-11T20:10:22.910 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:22 smithi097 systemd[1]: Stopping Ceph osd.0 for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:10:23.406 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:22 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0[47090]: 2024-03-11T20:10:22.980+0000 7faea7400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:10:23.407 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:22 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0[47090]: 2024-03-11T20:10:22.980+0000 7faea7400700 -1 osd.0 106 *** Got signal Terminated *** 2024-03-11T20:10:23.407 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:22 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0[47090]: 2024-03-11T20:10:22.980+0000 7faea7400700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:10:28.844 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:28 smithi097 podman[93805]: 2024-03-11 20:10:28.515104713 +0000 UTC m=+5.709668754 container died 9091d43dab05a76d7984828aede746b651858abd5c23bb08b198f991c32524ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-11T20:10:29.156 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:29 smithi097 podman[93805]: 2024-03-11 20:10:29.017365086 +0000 UTC m=+6.211929159 container cleanup 9091d43dab05a76d7984828aede746b651858abd5c23bb08b198f991c32524ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397) 2024-03-11T20:10:29.156 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:29 smithi097 bash[93805]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0 2024-03-11T20:10:29.656 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:29 smithi097 podman[93826]: 2024-03-11 20:10:29.310542666 +0000 UTC m=+0.773497830 container remove 9091d43dab05a76d7984828aede746b651858abd5c23bb08b198f991c32524ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T20:10:30.322 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:30 smithi097 podman[93911]: 2024-03-11 20:10:29.977891804 +0000 UTC m=+0.036501495 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T20:10:30.322 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:30 smithi097 podman[93911]: 2024-03-11 20:10:30.151979536 +0000 UTC m=+0.210589257 container create eead4761d162efd546b3b6696b9d68a7887683862bc2102bd616c46c015b5f3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True) 2024-03-11T20:10:30.586 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:30 smithi097 podman[93911]: 2024-03-11 20:10:30.457449459 +0000 UTC m=+0.516059186 container init eead4761d162efd546b3b6696b9d68a7887683862bc2102bd616c46c015b5f3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0-deactivate, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T20:10:30.587 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:30 smithi097 podman[93911]: 2024-03-11 20:10:30.468372271 +0000 UTC m=+0.526981966 container start eead4761d162efd546b3b6696b9d68a7887683862bc2102bd616c46c015b5f3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-11T20:10:30.587 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:30 smithi097 podman[93911]: 2024-03-11 20:10:30.597932053 +0000 UTC m=+0.656541804 container attach eead4761d162efd546b3b6696b9d68a7887683862bc2102bd616c46c015b5f3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-11T20:10:31.156 INFO:journalctl@ceph.osd.0.smithi097.stdout:Mar 11 20:10:30 smithi097 podman[93911]: 2024-03-11 20:10:30.898741042 +0000 UTC m=+0.957350766 container died eead4761d162efd546b3b6696b9d68a7887683862bc2102bd616c46c015b5f3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:10:31.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T20:10:31.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:10:31.813 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T20:10:31.814 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T20:10:31.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.1 2024-03-11T20:10:32.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:31 smithi114 systemd[1]: Stopping Ceph osd.1 for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:10:32.478 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:32 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1[40946]: 2024-03-11T20:10:32.218+0000 7f59ba1cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:10:32.478 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:32 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1[40946]: 2024-03-11T20:10:32.218+0000 7f59ba1cd700 -1 osd.1 106 *** Got signal Terminated *** 2024-03-11T20:10:32.478 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:32 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1[40946]: 2024-03-11T20:10:32.218+0000 7f59ba1cd700 -1 osd.1 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:10:37.850 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:37 smithi114 podman[72864]: 2024-03-11 20:10:37.570337066 +0000 UTC m=+5.550361115 container died cb023e21741b80698d8caef6144de7938a50942a3b7c1c7e8b58c21977119838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:10:38.226 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:37 smithi114 podman[72864]: 2024-03-11 20:10:37.934863109 +0000 UTC m=+5.914887150 container cleanup cb023e21741b80698d8caef6144de7938a50942a3b7c1c7e8b58c21977119838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:10:38.226 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:37 smithi114 bash[72864]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1 2024-03-11T20:10:38.478 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:38 smithi114 podman[72882]: 2024-03-11 20:10:38.238522 +0000 UTC m=+0.665996853 container remove cb023e21741b80698d8caef6144de7938a50942a3b7c1c7e8b58c21977119838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T20:10:38.847 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:38 smithi114 podman[72974]: 2024-03-11 20:10:38.496926811 +0000 UTC m=+0.018207336 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T20:10:38.847 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:38 smithi114 podman[72974]: 2024-03-11 20:10:38.606068459 +0000 UTC m=+0.127349000 container create 41bf380fd73fa4992aaef00ee1f06238505b7df20fcc4d3267d0b8c9c8e6af03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397) 2024-03-11T20:10:38.847 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:38 smithi114 podman[72974]: 2024-03-11 20:10:38.858995915 +0000 UTC m=+0.380276436 container init 41bf380fd73fa4992aaef00ee1f06238505b7df20fcc4d3267d0b8c9c8e6af03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.schema-version=1.0) 2024-03-11T20:10:39.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:38 smithi114 podman[72974]: 2024-03-11 20:10:38.862528308 +0000 UTC m=+0.383808822 container start 41bf380fd73fa4992aaef00ee1f06238505b7df20fcc4d3267d0b8c9c8e6af03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:10:39.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:38 smithi114 podman[72974]: 2024-03-11 20:10:38.933977361 +0000 UTC m=+0.455257905 container attach 41bf380fd73fa4992aaef00ee1f06238505b7df20fcc4d3267d0b8c9c8e6af03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-03-11T20:10:39.395 INFO:journalctl@ceph.osd.1.smithi114.stdout:Mar 11 20:10:39 smithi114 podman[72974]: 2024-03-11 20:10:39.142423253 +0000 UTC m=+0.663703787 container died 41bf380fd73fa4992aaef00ee1f06238505b7df20fcc4d3267d0b8c9c8e6af03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:10:39.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T20:10:39.805 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:10:39.805 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T20:10:39.805 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T20:10:39.805 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.2 2024-03-11T20:10:40.185 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:39 smithi161 systemd[1]: Stopping Ceph osd.2 for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:10:40.452 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:40 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2[41453]: 2024-03-11T20:10:40.195+0000 7f81e1716700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:10:40.452 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:40 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2[41453]: 2024-03-11T20:10:40.195+0000 7f81e1716700 -1 osd.2 106 *** Got signal Terminated *** 2024-03-11T20:10:40.452 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:40 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2[41453]: 2024-03-11T20:10:40.195+0000 7f81e1716700 -1 osd.2 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:10:46.025 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:45 smithi161 podman[73874]: 2024-03-11 20:10:45.654753822 +0000 UTC m=+5.648495427 container died 54cba555377a1e2af93544d840d60ca03ec913d3a2dbad59019d32d082a252c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T20:10:46.395 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:46 smithi161 podman[73874]: 2024-03-11 20:10:46.155462605 +0000 UTC m=+6.149204207 container cleanup 54cba555377a1e2af93544d840d60ca03ec913d3a2dbad59019d32d082a252c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-11T20:10:46.395 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:46 smithi161 bash[73874]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2 2024-03-11T20:10:46.746 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:46 smithi161 podman[73903]: 2024-03-11 20:10:46.407628207 +0000 UTC m=+0.750167206 container remove 54cba555377a1e2af93544d840d60ca03ec913d3a2dbad59019d32d082a252c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-11T20:10:47.024 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:46 smithi161 podman[73993]: 2024-03-11 20:10:46.758810266 +0000 UTC m=+0.105928226 container create ef74471b35d9a68b2736e49c26fb2b543c0b0fa2e082329b0a51350be86ccaf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:10:47.024 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:46 smithi161 podman[73993]: 2024-03-11 20:10:46.671119003 +0000 UTC m=+0.018236944 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T20:10:47.024 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:46 smithi161 podman[73993]: 2024-03-11 20:10:46.994637897 +0000 UTC m=+0.341755856 container init ef74471b35d9a68b2736e49c26fb2b543c0b0fa2e082329b0a51350be86ccaf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-94a5397) 2024-03-11T20:10:47.024 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:46 smithi161 podman[73993]: 2024-03-11 20:10:46.997697042 +0000 UTC m=+0.344814983 container start ef74471b35d9a68b2736e49c26fb2b543c0b0fa2e082329b0a51350be86ccaf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-03-11T20:10:47.278 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:47 smithi161 podman[73993]: 2024-03-11 20:10:47.10337987 +0000 UTC m=+0.450497814 container attach ef74471b35d9a68b2736e49c26fb2b543c0b0fa2e082329b0a51350be86ccaf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True) 2024-03-11T20:10:47.645 INFO:journalctl@ceph.osd.2.smithi161.stdout:Mar 11 20:10:47 smithi161 podman[73993]: 2024-03-11 20:10:47.290630907 +0000 UTC m=+0.637748849 container died ef74471b35d9a68b2736e49c26fb2b543c0b0fa2e082329b0a51350be86ccaf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-11T20:10:47.972 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T20:10:48.016 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:10:48.016 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T20:10:48.016 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-11T20:10:48.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.3 2024-03-11T20:10:48.406 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:48 smithi097 systemd[1]: Stopping Ceph osd.3 for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:10:48.907 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:48 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3[53628]: 2024-03-11T20:10:48.585+0000 7fdce5e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:10:48.907 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:48 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3[53628]: 2024-03-11T20:10:48.585+0000 7fdce5e00700 -1 osd.3 106 *** Got signal Terminated *** 2024-03-11T20:10:48.907 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:48 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3[53628]: 2024-03-11T20:10:48.585+0000 7fdce5e00700 -1 osd.3 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:10:49.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:10:49 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:10:49.174+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:10:49.656 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:49 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3[53628]: 2024-03-11T20:10:49.198+0000 7fdce1600700 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.746547+0000 front 2024-03-11T20:10:26.746903+0000 (oldest deadline 2024-03-11T20:10:49.046458+0000) 2024-03-11T20:10:49.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:10:49 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:10:49.339+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:10:50.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:10:49 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:10:49.813+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:10:50.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:10:50 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:10:50.154+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:10:50.656 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:50 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3[53628]: 2024-03-11T20:10:50.236+0000 7fdce1600700 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.746547+0000 front 2024-03-11T20:10:26.746903+0000 (oldest deadline 2024-03-11T20:10:49.046458+0000) 2024-03-11T20:10:50.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:10:50 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:10:50.292+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:10:50.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:10:50 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:10:50.844+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:10:51.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:10:51 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:10:51.197+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:10:51.656 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:51 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3[53628]: 2024-03-11T20:10:51.270+0000 7fdce1600700 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.746547+0000 front 2024-03-11T20:10:26.746903+0000 (oldest deadline 2024-03-11T20:10:49.046458+0000) 2024-03-11T20:10:51.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:10:51 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:10:51.294+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:10:52.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:10:51 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:10:51.685+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:10:52.227 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:10:51 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:10:51.879+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:10:52.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:10:52 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:10:52.232+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:10:52.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:10:52 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:10:52.191+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:10:52.656 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:52 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3[53628]: 2024-03-11T20:10:52.232+0000 7fdce1600700 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.746547+0000 front 2024-03-11T20:10:26.746903+0000 (oldest deadline 2024-03-11T20:10:49.046458+0000) 2024-03-11T20:10:52.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:10:52 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:10:52.248+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:10:53.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:10:52 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:10:52.723+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:10:53.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:10:52 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:10:52.847+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:10:53.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:10:53 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:10:53.203+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:10:53.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:10:53 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:10:53.184+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:10:53.558 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:53 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3[53628]: 2024-03-11T20:10:53.255+0000 7fdce1600700 -1 osd.3 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.746547+0000 front 2024-03-11T20:10:26.746903+0000 (oldest deadline 2024-03-11T20:10:49.046458+0000) 2024-03-11T20:10:53.558 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:10:53 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:10:53.252+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:10:54.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:10:53 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:10:53.727+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:10:54.204 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:54 smithi097 podman[94018]: 2024-03-11 20:10:54.045792989 +0000 UTC m=+5.606362343 container died 8fa5f084638c897a9c7249aac4dbe43a2976694ccdf1477e7153f054bbd07de1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:10:54.227 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:10:53 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:10:53.848+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:10:54.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:10:54 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:10:54.172+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:10:54.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:10:54 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:10:54.185+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:10:54.526 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:10:54 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:10:54.215+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:10:54.778 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:54 smithi097 podman[94018]: 2024-03-11 20:10:54.538288051 +0000 UTC m=+6.098857413 container cleanup 8fa5f084638c897a9c7249aac4dbe43a2976694ccdf1477e7153f054bbd07de1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T20:10:54.778 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:54 smithi097 bash[94018]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3 2024-03-11T20:10:55.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:10:54 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:10:54.748+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:10:55.156 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:54 smithi097 podman[94041]: 2024-03-11 20:10:54.789604618 +0000 UTC m=+0.726857024 container remove 8fa5f084638c897a9c7249aac4dbe43a2976694ccdf1477e7153f054bbd07de1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T20:10:55.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:10:54 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:10:54.863+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:10:55.503 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:10:55 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:10:55.178+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:10:55.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:10:55 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:10:55.193+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:10:55.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:10:55 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:10:55.183+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:10:55.858 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:55 smithi097 podman[94134]: 2024-03-11 20:10:55.421331303 +0000 UTC m=+0.035000704 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T20:10:55.858 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:55 smithi097 podman[94134]: 2024-03-11 20:10:55.557496106 +0000 UTC m=+0.171165565 container create 3025bbde62733a124ce546f5de2a255fe2b47c838c5777465f1ae76d84cad4e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T20:10:55.859 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:55 smithi097 podman[94134]: 2024-03-11 20:10:55.869787669 +0000 UTC m=+0.483457118 container init 3025bbde62733a124ce546f5de2a255fe2b47c838c5777465f1ae76d84cad4e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-11T20:10:56.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:10:55 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:10:55.710+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:10:56.156 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:55 smithi097 podman[94134]: 2024-03-11 20:10:55.882249538 +0000 UTC m=+0.495918983 container start 3025bbde62733a124ce546f5de2a255fe2b47c838c5777465f1ae76d84cad4e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-11T20:10:56.156 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:55 smithi097 podman[94134]: 2024-03-11 20:10:55.976896709 +0000 UTC m=+0.590566195 container attach 3025bbde62733a124ce546f5de2a255fe2b47c838c5777465f1ae76d84cad4e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:10:56.227 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:10:55 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:10:55.819+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:10:56.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:10:56 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:10:56.153+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:10:56.525 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:10:56 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:10:56.169+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:10:56.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:10:56 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:10:56.207+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:10:56.657 INFO:journalctl@ceph.osd.3.smithi097.stdout:Mar 11 20:10:56 smithi097 podman[94134]: 2024-03-11 20:10:56.312340453 +0000 UTC m=+0.926009899 container died 3025bbde62733a124ce546f5de2a255fe2b47c838c5777465f1ae76d84cad4e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-3-deactivate, ceph=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:10:57.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:10:56 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:10:56.724+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:10:57.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.3.service' 2024-03-11T20:10:57.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:10:56 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:10:56.800+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:10:57.230 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:10:57.231 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-11T20:10:57.231 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-11T20:10:57.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.4 2024-03-11T20:10:57.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:10:57 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:10:57.249+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:10:57.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:10:57 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:10:57.147+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:10:57.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:10:57 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:10:57.166+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:10:57.528 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:10:57 smithi114 systemd[1]: Stopping Ceph osd.4 for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:10:57.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:10:57 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:10:57.761+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:10:57.978 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:10:57 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4[47754]: 2024-03-11T20:10:57.540+0000 7fb4ad7f0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:10:57.979 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:10:57 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4[47754]: 2024-03-11T20:10:57.540+0000 7fb4ad7f0700 -1 osd.4 106 *** Got signal Terminated *** 2024-03-11T20:10:57.979 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:10:57 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4[47754]: 2024-03-11T20:10:57.540+0000 7fb4ad7f0700 -1 osd.4 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:10:58.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:10:57 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:10:57.703+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:10:58.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:10:58 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:10:58.126+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:10:58.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:10:58 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:10:58.130+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:10:58.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:10:58 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:10:58.255+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:10:58.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:10:58 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:10:58.729+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:10:59.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:10:58 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:10:58.747+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:10:59.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:10:59 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:10:59.150+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:10:59.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:10:59 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:10:59.124+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:10:59.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:10:59 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:10:59.286+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:10:59.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:10:59 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:10:59.705+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:10:59.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:10:59 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:10:59.705+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:00.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:10:59 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:10:59.736+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:11:00.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:00 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:00.101+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:00.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:00 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:00.129+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:00.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:00 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:00.129+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:00.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:00 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:00.320+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:00.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:00 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:00.320+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:00.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:00 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:00.635+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:00.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:00 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:00.663+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:00.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:00 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:00.663+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:01.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:00 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:00.783+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:11:01.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:01 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:01.088+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:01.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:01 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:01.103+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:01.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:01 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:01.103+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:01.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:01 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:01.335+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:01.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:01 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:01.335+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:01.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:01 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:01.232+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:01.665 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:11:01 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4[47754]: 2024-03-11T20:11:01.412+0000 7fb4a9de4700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.567467+0000 front 2024-03-11T20:10:35.567482+0000 (oldest deadline 2024-03-11T20:11:00.768161+0000) 2024-03-11T20:11:01.665 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:01 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:01.638+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:01.665 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:01 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:01.638+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:01.977 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:01 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:01.685+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:02.056 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:01 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:01.804+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:11:02.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:02 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:02.068+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:02.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:02 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:02.073+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:02.524 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:02 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:02.073+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:02.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:02 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:02.248+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:02.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:02 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:02.291+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:02.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:02 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:02.291+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:02.665 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:11:02 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4[47754]: 2024-03-11T20:11:02.404+0000 7fb4a9de4700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.567467+0000 front 2024-03-11T20:10:35.567482+0000 (oldest deadline 2024-03-11T20:11:00.768161+0000) 2024-03-11T20:11:02.665 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:02 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:02.642+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:02.665 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:02 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:02.642+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:02.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:02 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:02.676+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:03.246 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:11:02 smithi114 podman[73105]: 2024-03-11 20:11:02.996183756 +0000 UTC m=+5.561080521 container died 8d193d08184126b7033b51fd877615fd4a1ea61f707ab246e267f7d611eebf6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T20:11:03.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:03 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:03.037+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:03.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:03 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:03.037+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:03.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:03 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:03.037+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:03.275 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:02 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:02.823+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:11:03.275 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:02 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:02.823+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.655398+0000 front 2024-03-11T20:10:36.655561+0000 (oldest deadline 2024-03-11T20:11:01.955294+0000) 2024-03-11T20:11:03.623 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:11:03 smithi114 podman[73105]: 2024-03-11 20:11:03.349148821 +0000 UTC m=+5.914045586 container cleanup 8d193d08184126b7033b51fd877615fd4a1ea61f707ab246e267f7d611eebf6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-11T20:11:03.623 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:11:03 smithi114 bash[73105]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4 2024-03-11T20:11:03.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:03 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:03.264+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:03.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:03 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:03.248+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:03.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:03 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:03.248+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:03.955 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:11:03 smithi114 podman[73122]: 2024-03-11 20:11:03.634094767 +0000 UTC m=+0.632572856 container remove 8d193d08184126b7033b51fd877615fd4a1ea61f707ab246e267f7d611eebf6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T20:11:03.955 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:03 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:03.702+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:03.956 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:03 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:03.664+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:03.956 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:03 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:03.664+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:04.075 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:03 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:03.988+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:04.075 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:03 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:03.988+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:04.075 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:03 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:03.812+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:11:04.075 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:03 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:03.812+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.655398+0000 front 2024-03-11T20:10:36.655561+0000 (oldest deadline 2024-03-11T20:11:01.955294+0000) 2024-03-11T20:11:04.217 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:11:03 smithi114 podman[73212]: 2024-03-11 20:11:03.870571812 +0000 UTC m=+0.017594638 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T20:11:04.217 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:11:03 smithi114 podman[73212]: 2024-03-11 20:11:03.993472768 +0000 UTC m=+0.140495581 container create 1038ec01f7a64d6d464cad05755600f2896add60a0b53dcd1c559017fb2c4498 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T20:11:04.217 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:11:04 smithi114 podman[73212]: 2024-03-11 20:11:04.229221844 +0000 UTC m=+0.376244672 container init 1038ec01f7a64d6d464cad05755600f2896add60a0b53dcd1c559017fb2c4498 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T20:11:04.478 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:11:04 smithi114 podman[73212]: 2024-03-11 20:11:04.232325378 +0000 UTC m=+0.379348182 container start 1038ec01f7a64d6d464cad05755600f2896add60a0b53dcd1c559017fb2c4498 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397) 2024-03-11T20:11:04.478 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:11:04 smithi114 podman[73212]: 2024-03-11 20:11:04.312663025 +0000 UTC m=+0.459685847 container attach 1038ec01f7a64d6d464cad05755600f2896add60a0b53dcd1c559017fb2c4498 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:11:04.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:04 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:04.087+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:04.637 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:04 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:04.256+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:04.638 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:04 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:04.294+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:04.638 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:04 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:04.294+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:04.811 INFO:journalctl@ceph.osd.4.smithi114.stdout:Mar 11 20:11:04 smithi114 podman[73212]: 2024-03-11 20:11:04.522764228 +0000 UTC m=+0.669787032 container died 1038ec01f7a64d6d464cad05755600f2896add60a0b53dcd1c559017fb2c4498 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-94a5397, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-11T20:11:04.812 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:04 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:04.682+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:04.812 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:04 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:04.682+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:04.812 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:04 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:04.738+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:05.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:04 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:04.763+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:11:05.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:04 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:04.763+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.655398+0000 front 2024-03-11T20:10:36.655561+0000 (oldest deadline 2024-03-11T20:11:01.955294+0000) 2024-03-11T20:11:05.024 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:04 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:04.988+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:05.024 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:04 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:04.988+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:05.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.4.service' 2024-03-11T20:11:05.158 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:11:05.158 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-11T20:11:05.158 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-11T20:11:05.159 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.5 2024-03-11T20:11:05.490 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:05 smithi161 systemd[1]: Stopping Ceph osd.5 for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:11:05.490 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:05 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:05.501+0000 7f33e86a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:11:05.491 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:05 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:05.501+0000 7f33e86a2700 -1 osd.5 106 *** Got signal Terminated *** 2024-03-11T20:11:05.491 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:05 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:05.501+0000 7f33e86a2700 -1 osd.5 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:11:05.491 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:05 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:05.122+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:05.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:05 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:05.281+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:05.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:05 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:05.249+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:05.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:05 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:05.249+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:05.774 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:05 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:05.730+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:11:05.774 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:05 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:05.730+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.655398+0000 front 2024-03-11T20:10:36.655561+0000 (oldest deadline 2024-03-11T20:11:01.955294+0000) 2024-03-11T20:11:05.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:05 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:05.714+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:05.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:05 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:05.701+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:05.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:05 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:05.701+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:06.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:06 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:06.075+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:06.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:05 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:05.983+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:06.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:05 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:05.983+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:06.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:06 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:06.258+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:06.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:06 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:06.221+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:06.657 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:06 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:06.221+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:06.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:06 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:06.695+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:06.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:06 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:06.695+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:06.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:06 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:06.700+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:06.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:06 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:06.700+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:07.024 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:06 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:06.992+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:07.024 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:06 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:06.992+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:07.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:06 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:06.779+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:11:07.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:06 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:06.779+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.655398+0000 front 2024-03-11T20:10:36.655561+0000 (oldest deadline 2024-03-11T20:11:01.955294+0000) 2024-03-11T20:11:07.025 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:06 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:06.779+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.955640+0000 front 2024-03-11T20:10:41.955595+0000 (oldest deadline 2024-03-11T20:11:06.655682+0000) 2024-03-11T20:11:07.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:07 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:07.075+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:07.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:07 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:07.075+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:07.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:07 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:07.245+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:07.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:07 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:07.225+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:07.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:07 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:07.225+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:07.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:07 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:07.649+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:07.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:07 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:07.649+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:07.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:07 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:07.692+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:07.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:07 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:07.692+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:08.278 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:08.018+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:08.278 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:08.018+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:08.278 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:08.048+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:08.279 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:08.048+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:08.279 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:07 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:07.805+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:11:08.279 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:07 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:07.805+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.655398+0000 front 2024-03-11T20:10:36.655561+0000 (oldest deadline 2024-03-11T20:11:01.955294+0000) 2024-03-11T20:11:08.279 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:07 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:07.805+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.955640+0000 front 2024-03-11T20:10:41.955595+0000 (oldest deadline 2024-03-11T20:11:06.655682+0000) 2024-03-11T20:11:08.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:08.261+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:08.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:08.261+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.642913+0000 front 2024-03-11T20:10:42.642575+0000 (oldest deadline 2024-03-11T20:11:07.942839+0000) 2024-03-11T20:11:08.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:08.208+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:08.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:08 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:08.208+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:08.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:08.630+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:08.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:08.630+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:08.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:08.739+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:08.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:08.739+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:08.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:08 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:08.739+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:09.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:08.773+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:11:09.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:08.773+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.655398+0000 front 2024-03-11T20:10:36.655561+0000 (oldest deadline 2024-03-11T20:11:01.955294+0000) 2024-03-11T20:11:09.024 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:08 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:08.773+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.955640+0000 front 2024-03-11T20:10:41.955595+0000 (oldest deadline 2024-03-11T20:11:06.655682+0000) 2024-03-11T20:11:09.024 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:09.009+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:09.025 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:09.009+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:09.025 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:09.009+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:09.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:09.251+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:09.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:09.251+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.642913+0000 front 2024-03-11T20:10:42.642575+0000 (oldest deadline 2024-03-11T20:11:07.942839+0000) 2024-03-11T20:11:09.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:09.164+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:09.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:09.164+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:09.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:09.164+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:09.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:09.086+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:09.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:09.086+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:09.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:09.601+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:09.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:09.601+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:09.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:09.758+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:09.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:09.758+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:09.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:09 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:09.758+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:10.123 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:09.998+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:10.123 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:09.998+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:10.123 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:09.998+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:10.123 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:09.813+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:26.753801+0000 front 2024-03-11T20:10:26.753828+0000 (oldest deadline 2024-03-11T20:10:51.453711+0000) 2024-03-11T20:11:10.123 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:09.813+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.655398+0000 front 2024-03-11T20:10:36.655561+0000 (oldest deadline 2024-03-11T20:11:01.955294+0000) 2024-03-11T20:11:10.123 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:09 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5[48447]: 2024-03-11T20:11:09.813+0000 7f33e0c8e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.955640+0000 front 2024-03-11T20:10:41.955595+0000 (oldest deadline 2024-03-11T20:11:06.655682+0000) 2024-03-11T20:11:10.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:10 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:10.298+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:10.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:10 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:10.298+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.642913+0000 front 2024-03-11T20:10:42.642575+0000 (oldest deadline 2024-03-11T20:11:07.942839+0000) 2024-03-11T20:11:10.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:10 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:10.139+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:10.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:10 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:10.139+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:10.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:10 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:10.139+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:10.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:10 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:10.135+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:10.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:10 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:10.135+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:10.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:10 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:10.610+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:10.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:10 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:10.610+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:10.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:10 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:10.723+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:10.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:10 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:10.723+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:10.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:10 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:10.723+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:11.249 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:11.107+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:11.249 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:11.107+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:11.249 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:10 smithi161 podman[74099]: 2024-03-11 20:11:10.969066172 +0000 UTC m=+5.600747898 container died 83084f0fbfd2c5c2f5aa9a9b73c290755f727c9dc6f5285327a0ec3062442102 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-11T20:11:11.250 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:10 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:10.961+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:11.250 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:10 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:10.961+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:11.250 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:10 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:10.961+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:11.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:11 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:11.316+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:11.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:11 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:11.316+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.642913+0000 front 2024-03-11T20:10:42.642575+0000 (oldest deadline 2024-03-11T20:11:07.942839+0000) 2024-03-11T20:11:11.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:11 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:11.162+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:11.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:11 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:11.162+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:11.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:11 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:11.162+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:11.524 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:11 smithi161 podman[74099]: 2024-03-11 20:11:11.368755975 +0000 UTC m=+6.000437702 container cleanup 83084f0fbfd2c5c2f5aa9a9b73c290755f727c9dc6f5285327a0ec3062442102 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:11:11.524 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:11 smithi161 bash[74099]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5 2024-03-11T20:11:11.914 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:11 smithi161 podman[74116]: 2024-03-11 20:11:11.636849888 +0000 UTC m=+0.657730805 container remove 83084f0fbfd2c5c2f5aa9a9b73c290755f727c9dc6f5285327a0ec3062442102 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:11:11.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:11.638+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:11.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:11.638+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:11.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:11.757+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:11.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:11.757+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:11.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:11 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:11.757+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:12.180 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:12 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:12.069+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:12.180 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:12 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:12.069+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:12.180 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:12 smithi161 podman[74219]: 2024-03-11 20:11:11.928896026 +0000 UTC m=+0.018484587 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T20:11:12.180 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:12 smithi161 podman[74219]: 2024-03-11 20:11:12.046168457 +0000 UTC m=+0.135756997 container create e1da378e91a37910a51a949da105c8446518724053e81d90956c85cf0cfc9151 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T20:11:12.180 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:11.925+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:12.181 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:11.925+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:12.181 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:11 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:11.925+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:12.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:12 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:12.297+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:12.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:12 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:12.297+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.642913+0000 front 2024-03-11T20:10:42.642575+0000 (oldest deadline 2024-03-11T20:11:07.942839+0000) 2024-03-11T20:11:12.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:12 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:12.118+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:12.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:12 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:12.118+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:12.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:12 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:12.118+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:12.595 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:12 smithi161 podman[74219]: 2024-03-11 20:11:12.298699829 +0000 UTC m=+0.388288374 container init e1da378e91a37910a51a949da105c8446518724053e81d90956c85cf0cfc9151 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-11T20:11:12.596 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:12 smithi161 podman[74219]: 2024-03-11 20:11:12.301906817 +0000 UTC m=+0.391495372 container start e1da378e91a37910a51a949da105c8446518724053e81d90956c85cf0cfc9151 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-11T20:11:12.596 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:12 smithi161 podman[74219]: 2024-03-11 20:11:12.390308902 +0000 UTC m=+0.479897465 container attach e1da378e91a37910a51a949da105c8446518724053e81d90956c85cf0cfc9151 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T20:11:12.872 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:12 smithi161 podman[74219]: 2024-03-11 20:11:12.608002743 +0000 UTC m=+0.697591292 container died e1da378e91a37910a51a949da105c8446518724053e81d90956c85cf0cfc9151 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T20:11:12.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:12 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:12.604+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:12.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:12 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:12.604+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:12.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:12 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:12.781+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:12.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:12 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:12.781+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:12.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:12 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:12.781+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:13.132 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:13 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:13.113+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:13.132 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:13 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:13.113+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:13.132 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:12 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:12.885+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:13.132 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:12 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:12.885+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:13.132 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:12 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:12.885+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:13.133 INFO:journalctl@ceph.osd.5.smithi161.stdout:Mar 11 20:11:13 smithi161 podman[74246]: 2024-03-11 20:11:13.127219681 +0000 UTC m=+0.513905045 container remove e1da378e91a37910a51a949da105c8446518724053e81d90956c85cf0cfc9151 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-11T20:11:13.137 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.5.service' 2024-03-11T20:11:13.179 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:11:13.179 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-11T20:11:13.179 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-11T20:11:13.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.6 2024-03-11T20:11:13.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:13 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:13.315+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:13.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:13 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:13.315+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.642913+0000 front 2024-03-11T20:10:42.642575+0000 (oldest deadline 2024-03-11T20:11:07.942839+0000) 2024-03-11T20:11:13.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:13 smithi097 systemd[1]: Stopping Ceph osd.6 for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:11:13.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:13 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:13.081+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:13.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:13 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:13.081+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:13.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:13 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:13.081+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:13.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:13 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:13.626+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:13.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:13 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:13.626+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:13.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:13 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:13.745+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:13.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:13 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:13.745+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:13.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:13 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:13.745+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:14.100 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:13 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:13.774+0000 7f1cca400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:11:14.100 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:13 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:13.774+0000 7f1cca400700 -1 osd.6 106 *** Got signal Terminated *** 2024-03-11T20:11:14.100 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:13 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:13.774+0000 7f1cca400700 -1 osd.6 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:11:14.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:14 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:14.105+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:14.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:14 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:14.105+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:14.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:13 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:13.883+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:14.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:13 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:13.883+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:14.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:13 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:13.883+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:14.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:14 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:14.279+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:14.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:14 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:14.279+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.642913+0000 front 2024-03-11T20:10:42.642575+0000 (oldest deadline 2024-03-11T20:11:07.942839+0000) 2024-03-11T20:11:14.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:14 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:14.110+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:14.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:14 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:14.110+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:14.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:14 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:14.110+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:14.977 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:14 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:14.604+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:14.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:14 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:14.604+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:14.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:14 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:14.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:14.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:14 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:14.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:14.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:14 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:14.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:15.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:15 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:15.076+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:15.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:15 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:15.076+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:15.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:15 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:15.076+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:15.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:14 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:14.887+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:15.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:14 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:14.887+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:15.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:14 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:14.887+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:15.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:15.304+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:15.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:15.304+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.642913+0000 front 2024-03-11T20:10:42.642575+0000 (oldest deadline 2024-03-11T20:11:07.942839+0000) 2024-03-11T20:11:15.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:15.068+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:15.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:15.068+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:15.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:15 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:15.068+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:15.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:15 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:15.612+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:15.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:15 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:15.612+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:15.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:15 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:15.801+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:15.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:15 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:15.801+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:15.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:15 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:15.801+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:15.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:15 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:15.801+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:16.106 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:15 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:15.841+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:16.106 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:15 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:15.841+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:16.107 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:15 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:15.841+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:16.304 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:16.030+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:16.304 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:16.030+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:16.304 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:16.030+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:16.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:16 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:16.117+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:16.525 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:16 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:16.117+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:16.525 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:16 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:16.117+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:16.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:16.314+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:16.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:16 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:16.314+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.642913+0000 front 2024-03-11T20:10:42.642575+0000 (oldest deadline 2024-03-11T20:11:07.942839+0000) 2024-03-11T20:11:16.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:16 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:16.582+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:16.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:16 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:16.582+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:16.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:16 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:16.798+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:16.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:16 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:16.798+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:16.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:16 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:16.798+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:16.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:16 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:16.798+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:17.139 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:16 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:16.840+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:17.140 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:16 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:16.840+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:17.140 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:16 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:16.840+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:17.325 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:17 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:17.068+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:17.326 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:17 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:17.068+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:17.326 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:17 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:17.068+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:17.326 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:17 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:17.068+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:17.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:17 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:17.151+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:17.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:17 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:17.151+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:17.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:17 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:17.151+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:17.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:17 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:17.336+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:17.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:17 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:17.336+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.642913+0000 front 2024-03-11T20:10:42.642575+0000 (oldest deadline 2024-03-11T20:11:07.942839+0000) 2024-03-11T20:11:17.833 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:17 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:17.545+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:17.833 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:17 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:17.545+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:18.112 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:17 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:17.853+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:18.113 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:17 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:17.853+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:18.113 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:17 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:17.853+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:18.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:17 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:17.845+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:18.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:17 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:17.845+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:18.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:17 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:17.845+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:18.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:17 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:17.845+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:18.309 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:18 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:18.025+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:18.309 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:18 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:18.025+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:18.309 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:18 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:18.025+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:18.309 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:18 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:18.025+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:18.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:18 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:18.123+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:18.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:18 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:18.123+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:18.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:18 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:18.124+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:18.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:18 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:18.319+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:31.539243+0000 front 2024-03-11T20:10:31.539358+0000 (oldest deadline 2024-03-11T20:11:00.340393+0000) 2024-03-11T20:11:18.656 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:18 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6[60183]: 2024-03-11T20:11:18.319+0000 7f1cc5c00700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.642913+0000 front 2024-03-11T20:10:42.642575+0000 (oldest deadline 2024-03-11T20:11:07.942839+0000) 2024-03-11T20:11:18.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:18 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:18.579+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:18.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:18 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:18.579+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:18.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:18 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:18.579+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.374378+0000 front 2024-03-11T20:10:52.374448+0000 (oldest deadline 2024-03-11T20:11:18.274255+0000) 2024-03-11T20:11:18.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:18 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:18.827+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:18.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:18 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:18.827+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:18.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:18 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:18.827+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:18.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:18 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:18.827+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:19.088 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:18 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:18.822+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:19.088 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:18 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:18.822+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:19.088 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:18 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:18.822+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:19.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:19 smithi097 podman[94243]: 2024-03-11 20:11:19.183404233 +0000 UTC m=+5.562333277 container died 5f774f361093ee057e0324270a26221a76ca141962bed513d5ff6d9795cd86bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T20:11:19.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:19 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:19.049+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:19.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:19 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:19.049+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:19.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:19 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:19.049+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:19.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:19 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:19.049+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:19.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:19 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:19.099+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:19.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:19 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:19.099+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:19.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:19 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:19.099+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:19.906 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:19 smithi097 podman[94243]: 2024-03-11 20:11:19.700665488 +0000 UTC m=+6.079594595 container cleanup 5f774f361093ee057e0324270a26221a76ca141962bed513d5ff6d9795cd86bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-11T20:11:19.906 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:19 smithi097 bash[94243]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6 2024-03-11T20:11:19.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:19 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:19.610+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:19.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:19 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:19.610+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:19.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:19 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:19.610+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.374378+0000 front 2024-03-11T20:10:52.374448+0000 (oldest deadline 2024-03-11T20:11:18.274255+0000) 2024-03-11T20:11:19.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:19 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:19.781+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:19.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:19 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:19.781+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:19.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:19 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:19.781+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:19.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:19 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:19.781+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:20.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:20 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:20.071+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:20.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:20 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:20.071+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:20.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:20 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:20.071+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:20.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:19 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:19.831+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:20.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:19 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:19.831+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:20.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:19 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:19.831+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:20.406 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:20 smithi097 podman[94262]: 2024-03-11 20:11:20.010562342 +0000 UTC m=+0.814456168 container remove 5f774f361093ee057e0324270a26221a76ca141962bed513d5ff6d9795cd86bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-11T20:11:20.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:20 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:20.050+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:20.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:20 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:20.050+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:20.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:20 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:20.050+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:20.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:20 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:20.050+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:20.894 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:20 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:20.651+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:20.895 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:20 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:20.651+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:20.895 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:20 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:20.651+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.374378+0000 front 2024-03-11T20:10:52.374448+0000 (oldest deadline 2024-03-11T20:11:18.274255+0000) 2024-03-11T20:11:20.895 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:20 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:20.818+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:20.895 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:20 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:20.818+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:20.896 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:20 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:20.818+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:20.896 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:20 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:20.818+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:21.094 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:20 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:20.840+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:21.094 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:20 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:20.840+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:21.095 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:20 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:20.840+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:21.156 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:20 smithi097 podman[94353]: 2024-03-11 20:11:20.902833244 +0000 UTC m=+0.125762054 container create c16735b9f04fef37e81566d9302400e86976d5d2adb5d448d5185e1b7eee658b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6-deactivate, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-11T20:11:21.156 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:20 smithi097 podman[94353]: 2024-03-11 20:11:20.808073107 +0000 UTC m=+0.031001880 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T20:11:21.157 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:21 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:21.072+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:21.157 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:21 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:21.072+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:21.157 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:21 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:21.072+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:21.157 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:21 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:21.072+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:21.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:21 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:21.105+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:21.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:21 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:21.105+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:21.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:21 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:21.105+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:21.654 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:21 smithi097 podman[94353]: 2024-03-11 20:11:21.224152283 +0000 UTC m=+0.447081100 container init c16735b9f04fef37e81566d9302400e86976d5d2adb5d448d5185e1b7eee658b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-11T20:11:21.654 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:21 smithi097 podman[94353]: 2024-03-11 20:11:21.237415225 +0000 UTC m=+0.460344017 container start c16735b9f04fef37e81566d9302400e86976d5d2adb5d448d5185e1b7eee658b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6-deactivate, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T20:11:21.654 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:21 smithi097 podman[94353]: 2024-03-11 20:11:21.382287043 +0000 UTC m=+0.605215903 container attach c16735b9f04fef37e81566d9302400e86976d5d2adb5d448d5185e1b7eee658b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T20:11:21.906 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 20:11:21 smithi097 podman[94353]: 2024-03-11 20:11:21.666550283 +0000 UTC m=+0.889479113 container died c16735b9f04fef37e81566d9302400e86976d5d2adb5d448d5185e1b7eee658b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-11T20:11:21.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:21 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:21.640+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:21.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:21 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:21.640+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:21.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:21 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:21.640+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.374378+0000 front 2024-03-11T20:10:52.374448+0000 (oldest deadline 2024-03-11T20:11:18.274255+0000) 2024-03-11T20:11:21.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:21 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:21.818+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:21.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:21 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:21.818+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:21.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:21 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:21.818+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:21.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:21 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:21.818+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:22.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:22 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:22.083+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:22.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:22 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:22.083+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:22.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:22 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:22.083+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:22.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:21 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:21.858+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:22.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:21 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:21.858+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:22.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:21 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:21.858+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:22.400 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:22 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:22.080+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:22.400 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:22 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:22.080+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:22.400 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:22 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:22.080+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:22.400 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:22 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:22.080+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:22.441 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.6.service' 2024-03-11T20:11:22.553 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:11:22.554 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-11T20:11:22.554 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-11T20:11:22.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.7 2024-03-11T20:11:22.874 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:22 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:22.824+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:22.874 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:22 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:22.824+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:22.874 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:22 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:22.824+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:22.874 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:22 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:22.824+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:22.875 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:22 smithi114 systemd[1]: Stopping Ceph osd.7 for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:11:22.875 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:22 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:22.645+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:22.875 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:22 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:22.645+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:22.875 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:22 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:22.645+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.374378+0000 front 2024-03-11T20:10:52.374448+0000 (oldest deadline 2024-03-11T20:11:18.274255+0000) 2024-03-11T20:11:22.875 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:22 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:22.886+0000 7f625850a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:11:22.875 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:22 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:22.886+0000 7f625850a700 -1 osd.7 106 *** Got signal Terminated *** 2024-03-11T20:11:22.876 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:22 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:22.886+0000 7f625850a700 -1 osd.7 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:11:23.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:23 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:23.045+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:23.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:23 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:23.045+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:23.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:23 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:23.045+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:23.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:22 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:22.816+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:23.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:22 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:22.816+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:23.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:22 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:22.816+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:23.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:23 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:23.038+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:23.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:23 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:23.038+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:23.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:23 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:23.038+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:23.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:23 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:23.038+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:23.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:23 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:23.653+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:23.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:23 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:23.653+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:23.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:23 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:23.653+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.374378+0000 front 2024-03-11T20:10:52.374448+0000 (oldest deadline 2024-03-11T20:11:18.274255+0000) 2024-03-11T20:11:23.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:23 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:23.653+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.274618+0000 front 2024-03-11T20:10:58.274650+0000 (oldest deadline 2024-03-11T20:11:23.574799+0000) 2024-03-11T20:11:23.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:23 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:23.785+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:23.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:23 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:23.785+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:23.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:23 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:23.785+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:23.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:23 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:23.785+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:24.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:24 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:24.035+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:24.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:24 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:24.035+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:24.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:24 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:24.035+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:24.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:23 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:23.862+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:24.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:23 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:23.862+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:24.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:23 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:23.862+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:24.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:24 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:24.082+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:24.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:24 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:24.082+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:24.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:24 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:24.082+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:24.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:24 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:24.082+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:24.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:24 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:24.082+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:24.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:24 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:24.631+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:24.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:24 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:24.631+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:24.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:24 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:24.631+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.374378+0000 front 2024-03-11T20:10:52.374448+0000 (oldest deadline 2024-03-11T20:11:18.274255+0000) 2024-03-11T20:11:24.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:24 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:24.631+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.274618+0000 front 2024-03-11T20:10:58.274650+0000 (oldest deadline 2024-03-11T20:11:23.574799+0000) 2024-03-11T20:11:24.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:24 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:24.738+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:24.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:24 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:24.738+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:24.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:24 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:24.738+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:24.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:24 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:24.738+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:24.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:24 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:24.738+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:25.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:25 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:25.050+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:25.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:25 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:25.050+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:25.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:25 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:25.050+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:25.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:24 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:24.827+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:25.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:24 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:24.827+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:25.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:24 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:24.827+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:25.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:25 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:25.072+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:25.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:25 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:25.072+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:25.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:25 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:25.072+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:25.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:25 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:25.072+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:25.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:25 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:25.072+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:25.977 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:25 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:25.621+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:25.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:25 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:25.621+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:25.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:25 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:25.621+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.374378+0000 front 2024-03-11T20:10:52.374448+0000 (oldest deadline 2024-03-11T20:11:18.274255+0000) 2024-03-11T20:11:25.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:25 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:25.621+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.274618+0000 front 2024-03-11T20:10:58.274650+0000 (oldest deadline 2024-03-11T20:11:23.574799+0000) 2024-03-11T20:11:25.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:25 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:25.775+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:25.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:25 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:25.775+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:25.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:25 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:25.775+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:25.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:25 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:25.775+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:25.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:25 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:25.775+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:26.059 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:25 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:25.809+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:26.060 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:25 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:25.809+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:26.060 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:25 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:25.809+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:26.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:26 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:26.040+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:26.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:26 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:26.040+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:26.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:26 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:26.040+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:26.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:26 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:26.040+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:26.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:26 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:26.040+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:26.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:26 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:26.071+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:26.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:26 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:26.071+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:26.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:26 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:26.071+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:26.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:26 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:26.604+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:26.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:26 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:26.604+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:26.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:26 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:26.604+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.374378+0000 front 2024-03-11T20:10:52.374448+0000 (oldest deadline 2024-03-11T20:11:18.274255+0000) 2024-03-11T20:11:26.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:26 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:26.604+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.274618+0000 front 2024-03-11T20:10:58.274650+0000 (oldest deadline 2024-03-11T20:11:23.574799+0000) 2024-03-11T20:11:26.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:26 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:26.734+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:26.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:26 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:26.734+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:26.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:26 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:26.734+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:26.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:26 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:26.734+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:26.979 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:26 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:26.734+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:27.049 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:26 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:26.805+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:27.049 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:26 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:26.805+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:27.049 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:26 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:26.805+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:27.049 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:26 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:26.805+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:27.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:27 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:27.026+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:27.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:27 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:27.026+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:27.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:27 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:27.026+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:27.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:27 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:27.026+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:27.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:27 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:27.026+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:27.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:27 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:27.061+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:27.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:27 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:27.061+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:27.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:27 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:27.061+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:27.977 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:27 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:27.568+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.371979+0000 front 2024-03-11T20:10:35.372026+0000 (oldest deadline 2024-03-11T20:11:00.072159+0000) 2024-03-11T20:11:27.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:27 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:27.568+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:41.172982+0000 front 2024-03-11T20:10:41.172957+0000 (oldest deadline 2024-03-11T20:11:06.473106+0000) 2024-03-11T20:11:27.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:27 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:27.568+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.374378+0000 front 2024-03-11T20:10:52.374448+0000 (oldest deadline 2024-03-11T20:11:18.274255+0000) 2024-03-11T20:11:27.978 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:27 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7[54329]: 2024-03-11T20:11:27.568+0000 7f6250af6700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.274618+0000 front 2024-03-11T20:10:58.274650+0000 (oldest deadline 2024-03-11T20:11:23.574799+0000) 2024-03-11T20:11:27.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:27 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:27.691+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:27.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:27 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:27.691+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:27.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:27 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:27.691+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:27.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:27 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:27.691+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:27.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:27 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:27.691+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:28.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:28 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:28.021+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:28.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:28 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:28.021+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:28.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:28 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:28.021+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:28.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:28 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:28.021+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.410743+0000 front 2024-03-11T20:11:02.410695+0000 (oldest deadline 2024-03-11T20:11:27.110789+0000) 2024-03-11T20:11:28.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:27 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:27.789+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:28.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:27 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:27.789+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:28.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:27 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:27.789+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:28.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:27 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:27.789+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:28.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:27 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:27.988+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:28.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:27 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:27.988+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:28.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:27 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:27.988+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:28.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:27 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:27.988+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:28.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:27 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:27.988+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:28.477 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:28 smithi114 podman[73329]: 2024-03-11 20:11:28.239458464 +0000 UTC m=+5.487392359 container died 47e62e2f4d5d2d17b5b7a53f4e41484d1d98302b1c59d839b4d6b9475441d9b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T20:11:28.841 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:28 smithi114 podman[73329]: 2024-03-11 20:11:28.610839727 +0000 UTC m=+5.858773613 container cleanup 47e62e2f4d5d2d17b5b7a53f4e41484d1d98302b1c59d839b4d6b9475441d9b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T20:11:28.841 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:28 smithi114 bash[73329]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7 2024-03-11T20:11:28.842 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:28 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:28.725+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:28.842 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:28 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:28.725+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:28.842 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:28 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:28.725+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:28.842 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:28 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:28.725+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:28.842 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:28 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:28.725+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:29.157 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:28 smithi114 podman[73346]: 2024-03-11 20:11:28.853568929 +0000 UTC m=+0.610902730 container remove 47e62e2f4d5d2d17b5b7a53f4e41484d1d98302b1c59d839b4d6b9475441d9b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T20:11:29.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:29 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:29.042+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:29.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:29 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:29.042+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:29.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:29 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:29.042+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:29.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:29 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:29.042+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.410743+0000 front 2024-03-11T20:11:02.410695+0000 (oldest deadline 2024-03-11T20:11:27.110789+0000) 2024-03-11T20:11:29.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:28 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:28.836+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:29.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:28 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:28.836+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:29.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:28 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:28.836+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:29.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:28 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:28.836+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:29.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:28 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:28.989+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:29.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:28 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:28.989+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:29.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:28 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:28.989+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:29.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:28 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:28.989+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:29.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:28 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:28.989+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:29.428 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:29 smithi114 podman[73441]: 2024-03-11 20:11:29.072761842 +0000 UTC m=+0.017454689 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T20:11:29.428 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:29 smithi114 podman[73441]: 2024-03-11 20:11:29.204467994 +0000 UTC m=+0.149160827 container create 701fd146e78cf84c71663fba45ad4e7cb37fa4cfddffa2dfef2dd191baebfba2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:11:29.429 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:29 smithi114 podman[73441]: 2024-03-11 20:11:29.440462337 +0000 UTC m=+0.385155192 container init 701fd146e78cf84c71663fba45ad4e7cb37fa4cfddffa2dfef2dd191baebfba2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:11:29.717 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:29 smithi114 podman[73441]: 2024-03-11 20:11:29.444517431 +0000 UTC m=+0.389210275 container start 701fd146e78cf84c71663fba45ad4e7cb37fa4cfddffa2dfef2dd191baebfba2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-11T20:11:29.717 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:29 smithi114 podman[73441]: 2024-03-11 20:11:29.540561439 +0000 UTC m=+0.485254290 container attach 701fd146e78cf84c71663fba45ad4e7cb37fa4cfddffa2dfef2dd191baebfba2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7-deactivate, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:11:29.977 INFO:journalctl@ceph.osd.7.smithi114.stdout:Mar 11 20:11:29 smithi114 podman[73441]: 2024-03-11 20:11:29.729829609 +0000 UTC m=+0.674522452 container died 701fd146e78cf84c71663fba45ad4e7cb37fa4cfddffa2dfef2dd191baebfba2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-11T20:11:29.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:29 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:29.749+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:29.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:29 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:29.749+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:29.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:29 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:29.749+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:29.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:29 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:29.749+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:29.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:29 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:29.749+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:30.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:30 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:30.000+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:30.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:30 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:30.000+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:30.275 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:30 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:30.000+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:30.275 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:30 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:30.000+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.410743+0000 front 2024-03-11T20:11:02.410695+0000 (oldest deadline 2024-03-11T20:11:27.110789+0000) 2024-03-11T20:11:30.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:29 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:29.866+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:30.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:29 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:29.866+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:30.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:29 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:29.866+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:30.276 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:29 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:29.866+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:30.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.7.service' 2024-03-11T20:11:30.380 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:11:30.380 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-11T20:11:30.381 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-11T20:11:30.381 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.8 2024-03-11T20:11:30.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:30 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:30.026+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:30.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:30 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:30.026+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:30.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:30 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:30.026+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:30.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:30 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:30.026+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:30.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:30 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:30.026+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:30.684 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:30 smithi161 systemd[1]: Stopping Ceph osd.8 for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:11:30.990 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:30 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:30.849+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:30.990 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:30 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:30.849+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:30.990 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:30 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:30.849+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:30.990 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:30 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:30.849+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:30.991 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:30 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:30.695+0000 7fc974a5a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:11:30.991 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:30 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:30.695+0000 7fc974a5a700 -1 osd.8 106 *** Got signal Terminated *** 2024-03-11T20:11:30.991 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:30 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:30.695+0000 7fc974a5a700 -1 osd.8 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:11:30.991 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:31 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:31.001+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:30.991 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:31 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:31.001+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:30.991 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:31 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:31.001+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:30.991 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:31 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:31.001+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.410743+0000 front 2024-03-11T20:11:02.410695+0000 (oldest deadline 2024-03-11T20:11:27.110789+0000) 2024-03-11T20:11:31.227 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:30 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:30.742+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:31.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:30 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:30.742+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:31.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:30 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:30.742+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:31.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:30 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:30.742+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:31.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:30 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:30.742+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:31.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:30 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:30.994+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:31.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:30 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:30.994+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:31.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:30 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:30.994+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:31.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:30 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:30.994+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:31.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:30 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:30.994+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:32.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:31 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:31.753+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:32.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:31 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:31.753+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:32.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:31 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:31.753+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:32.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:31 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:31.753+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:32.229 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:31 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:31.753+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:32.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:31 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:31.832+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:32.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:31 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:31.832+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:32.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:31 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:31.832+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:32.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:31 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:31.832+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:32.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:31 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:31.985+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:32.275 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:31 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:31.985+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:32.275 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:31 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:31.985+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:32.275 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:31 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:31.985+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.410743+0000 front 2024-03-11T20:11:02.410695+0000 (oldest deadline 2024-03-11T20:11:27.110789+0000) 2024-03-11T20:11:32.275 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:31 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:31.985+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:07.111237+0000 front 2024-03-11T20:11:07.111282+0000 (oldest deadline 2024-03-11T20:11:31.811213+0000) 2024-03-11T20:11:32.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:31 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:31.981+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:32.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:31 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:31.981+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:32.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:31 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:31.982+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:32.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:31 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:31.982+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:32.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:31 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:31.982+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:32.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:32 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:32.705+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:32.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:32 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:32.705+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:32.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:32 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:32.705+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:32.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:32 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:32.705+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:32.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:32 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:32.705+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:33.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:32 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:32.804+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:33.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:32 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:32.804+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:33.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:32 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:32.804+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:33.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:32 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:32.804+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:33.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:32 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:32.978+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:33.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:32 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:32.978+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:33.275 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:32 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:32.978+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:33.275 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:32 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:32.978+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.410743+0000 front 2024-03-11T20:11:02.410695+0000 (oldest deadline 2024-03-11T20:11:27.110789+0000) 2024-03-11T20:11:33.275 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:32 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:32.978+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:07.111237+0000 front 2024-03-11T20:11:07.111282+0000 (oldest deadline 2024-03-11T20:11:31.811213+0000) 2024-03-11T20:11:33.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:32 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:32.971+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:33.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:32 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:32.971+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:33.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:32 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:32.971+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:33.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:32 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:32.971+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:33.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:32 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:32.971+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:33.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:33 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:33.708+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:33.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:33 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:33.708+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:33.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:33 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:33.708+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:33.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:33 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:33.708+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:33.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:33 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:33.708+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:33.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:33 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:33.708+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:34.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:33 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:33.844+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:34.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:33 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:33.844+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:34.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:33 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:33.844+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:34.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:33 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:33.844+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:34.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:34 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:34.008+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:34.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:34 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:34.008+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:34.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:34 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:34.008+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:34.275 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:34 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:34.008+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.410743+0000 front 2024-03-11T20:11:02.410695+0000 (oldest deadline 2024-03-11T20:11:27.110789+0000) 2024-03-11T20:11:34.275 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:34 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:34.008+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:07.111237+0000 front 2024-03-11T20:11:07.111282+0000 (oldest deadline 2024-03-11T20:11:31.811213+0000) 2024-03-11T20:11:34.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:33 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:33.979+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:34.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:33 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:33.979+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:34.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:33 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:33.979+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:34.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:33 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:33.979+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:34.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:33 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:33.979+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:34.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:34 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:34.708+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:34.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:34 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:34.708+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:34.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:34 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:34.708+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:34.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:34 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:34.708+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:34.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:34 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:34.708+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:34.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:34 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:34.708+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:35.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:35 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:35.048+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.304673+0000 front 2024-03-11T20:10:23.304364+0000 (oldest deadline 2024-03-11T20:10:48.604247+0000) 2024-03-11T20:11:35.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:35 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:35.048+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:42.606904+0000 front 2024-03-11T20:10:42.607017+0000 (oldest deadline 2024-03-11T20:11:06.107100+0000) 2024-03-11T20:11:35.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:35 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:35.048+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.008455+0000 front 2024-03-11T20:10:52.008638+0000 (oldest deadline 2024-03-11T20:11:14.308306+0000) 2024-03-11T20:11:35.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:35 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:35.048+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.410743+0000 front 2024-03-11T20:11:02.410695+0000 (oldest deadline 2024-03-11T20:11:27.110789+0000) 2024-03-11T20:11:35.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:35 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8[55305]: 2024-03-11T20:11:35.048+0000 7fc96d046700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:07.111237+0000 front 2024-03-11T20:11:07.111282+0000 (oldest deadline 2024-03-11T20:11:31.811213+0000) 2024-03-11T20:11:35.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:34 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:34.875+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:35.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:34 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:34.875+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:35.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:34 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:34.875+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:35.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:34 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:34.875+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:35.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:34 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:34.875+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:35.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:34 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:34.994+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:35.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:34 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:34.994+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:35.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:34 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:34.994+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:35.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:34 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:34.994+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:35.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:34 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:34.994+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:35.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:35 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:35.730+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:35.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:35 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:35.730+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:35.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:35 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:35.730+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:35.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:35 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:35.730+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:35.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:35 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:35.730+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:35.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:35 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:35.730+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:36.274 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:36 smithi161 podman[74326]: 2024-03-11 20:11:36.017017703 +0000 UTC m=+5.435376999 container died 99b754ca2fc4c6a144bd4a8a10fd225d58185a0d66a42b6758a31b829b8bddd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=) 2024-03-11T20:11:36.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:35 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:35.828+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:36.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:35 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:35.828+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:36.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:35 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:35.828+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:36.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:35 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:35.828+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:36.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:35 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:35.828+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:36.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:35 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:35.964+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:36.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:35 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:35.965+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:36.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:35 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:35.965+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:36.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:35 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:35.965+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:36.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:35 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:35.965+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:36.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:35 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:35.965+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.750499+0000 front 2024-03-11T20:11:09.750372+0000 (oldest deadline 2024-03-11T20:11:35.550882+0000) 2024-03-11T20:11:36.642 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:36 smithi161 podman[74326]: 2024-03-11 20:11:36.411937925 +0000 UTC m=+5.830297228 container cleanup 99b754ca2fc4c6a144bd4a8a10fd225d58185a0d66a42b6758a31b829b8bddd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:11:36.642 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:36 smithi161 bash[74326]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8 2024-03-11T20:11:36.964 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:36 smithi161 podman[74343]: 2024-03-11 20:11:36.654541465 +0000 UTC m=+0.633050802 container remove 99b754ca2fc4c6a144bd4a8a10fd225d58185a0d66a42b6758a31b829b8bddd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-11T20:11:36.965 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:36 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:36.806+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:36.965 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:36 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:36.806+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:36.965 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:36 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:36.806+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:36.965 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:36 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:36.806+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:36.965 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:36 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:36.806+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:37.227 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:36 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:36.744+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:37.227 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:36 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:36.744+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:37.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:36 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:36.744+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:37.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:36 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:36.744+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:37.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:36 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:36.744+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:37.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:36 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:36.744+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:37.254 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:36 smithi161 podman[74435]: 2024-03-11 20:11:36.879333946 +0000 UTC m=+0.017362777 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T20:11:37.254 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:36 smithi161 podman[74435]: 2024-03-11 20:11:36.996918817 +0000 UTC m=+0.134947638 container create 6a94ff326941640f9dfd37ef2f32d762ca769888a0042023bfee0afd6c95a2ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-11T20:11:37.254 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:37 smithi161 podman[74435]: 2024-03-11 20:11:37.191097485 +0000 UTC m=+0.329126308 container init 6a94ff326941640f9dfd37ef2f32d762ca769888a0042023bfee0afd6c95a2ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8-deactivate, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:11:37.254 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:37 smithi161 podman[74435]: 2024-03-11 20:11:37.194142125 +0000 UTC m=+0.332170942 container start 6a94ff326941640f9dfd37ef2f32d762ca769888a0042023bfee0afd6c95a2ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T20:11:37.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:36 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:36.981+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:37.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:36 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:36.981+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:37.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:36 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:36.981+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:37.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:36 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:36.981+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:37.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:36 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:36.981+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:37.407 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:36 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:36.981+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.750499+0000 front 2024-03-11T20:11:09.750372+0000 (oldest deadline 2024-03-11T20:11:35.550882+0000) 2024-03-11T20:11:37.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:37 smithi161 podman[74435]: 2024-03-11 20:11:37.265998515 +0000 UTC m=+0.404027369 container attach 6a94ff326941640f9dfd37ef2f32d762ca769888a0042023bfee0afd6c95a2ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-11T20:11:37.524 INFO:journalctl@ceph.osd.8.smithi161.stdout:Mar 11 20:11:37 smithi161 podman[74435]: 2024-03-11 20:11:37.488135462 +0000 UTC m=+0.626164285 container died 6a94ff326941640f9dfd37ef2f32d762ca769888a0042023bfee0afd6c95a2ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T20:11:37.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:37 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:37.695+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:37.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:37 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:37.695+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:37.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:37 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:37.695+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:37.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:37 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:37.695+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:37.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:37 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:37.695+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:37.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:37 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:37.695+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:38.007 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:37 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:37.839+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:38.007 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:37 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:37.839+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:38.007 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:37 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:37.839+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:38.007 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:37 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:37.839+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:38.007 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:37 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:37.839+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:38.028 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.8.service' 2024-03-11T20:11:38.070 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:11:38.070 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-11T20:11:38.070 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-11T20:11:38.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.9 2024-03-11T20:11:38.200 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:37 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:37.938+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:38.200 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:37 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:37.938+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:38.200 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:37 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:37.938+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:38.200 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:37 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:37.938+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:38.200 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:37 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:37.938+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:38.200 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:37 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:37.938+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.750499+0000 front 2024-03-11T20:11:09.750372+0000 (oldest deadline 2024-03-11T20:11:35.550882+0000) 2024-03-11T20:11:38.656 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:38 smithi097 systemd[1]: Stopping Ceph osd.9 for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:11:38.959 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:38.677+0000 7f05e7600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:11:38.959 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:38.677+0000 7f05e7600700 -1 osd.9 106 *** Got signal Terminated *** 2024-03-11T20:11:38.959 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:38.677+0000 7f05e7600700 -1 osd.9 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:11:38.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:38 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:38.694+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:38.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:38 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:38.694+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:38.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:38 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:38.694+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:38.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:38 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:38.694+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:38.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:38 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:38.694+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:38.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:38 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:38.694+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:39.213 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:38 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:38.791+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:39.213 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:38 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:38.791+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:39.213 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:38 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:38.791+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:39.213 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:38 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:38.791+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:39.213 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:38 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:38.791+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:39.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:38.970+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:39.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:38.970+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:39.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:38.970+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:39.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:38.970+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:39.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:38.970+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:39.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:38 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:38.970+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.750499+0000 front 2024-03-11T20:11:09.750372+0000 (oldest deadline 2024-03-11T20:11:35.550882+0000) 2024-03-11T20:11:39.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:39 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:39.709+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:39.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:39 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:39.709+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:39.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:39 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:39.709+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:39.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:39 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:39.709+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:39.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:39 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:39.709+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:39.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:39 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:39.709+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:40.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:39 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:39.812+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:40.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:39 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:39.812+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:40.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:39 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:39.812+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:40.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:39 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:39.812+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:40.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:39 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:39.812+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:40.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:40 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:40.009+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:40.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:40 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:40.009+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:40.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:40 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:40.009+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:40.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:40 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:40.009+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:40.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:40 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:40.009+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:40.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:40 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:40.009+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.750499+0000 front 2024-03-11T20:11:09.750372+0000 (oldest deadline 2024-03-11T20:11:35.550882+0000) 2024-03-11T20:11:41.227 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:40 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:40.753+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:41.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:40 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:40.753+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:41.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:40 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:40.753+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:41.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:40 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:40.753+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:41.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:40 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:40.753+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:41.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:40 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:40.753+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:41.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:40 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:40.835+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:41.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:40 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:40.835+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:41.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:40 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:40.835+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:41.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:40 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:40.835+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:41.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:40 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:40.835+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:41.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:41 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:41.015+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:41.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:41 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:41.015+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:41.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:41 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:41.015+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:41.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:41 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:41.015+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:41.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:41 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:41.015+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:41.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:41 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:41.015+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.750499+0000 front 2024-03-11T20:11:09.750372+0000 (oldest deadline 2024-03-11T20:11:35.550882+0000) 2024-03-11T20:11:42.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:41 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:41.761+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:42.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:41 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:41.761+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:42.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:41 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:41.761+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:42.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:41 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:41.761+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:42.229 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:41 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:41.761+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:42.229 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:41 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:41.761+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:42.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:41 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:41.871+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:42.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:41 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:41.871+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:42.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:41 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:41.871+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:42.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:41 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:41.871+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:42.276 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:41 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:41.871+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:42.276 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:41 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:41.871+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:42.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:42 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:42.048+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:42.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:42 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:42.048+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:42.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:42 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:42.048+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:42.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:42 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:42.048+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:42.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:42 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:42.048+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:42.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:42 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:42.048+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.750499+0000 front 2024-03-11T20:11:09.750372+0000 (oldest deadline 2024-03-11T20:11:35.550882+0000) 2024-03-11T20:11:43.227 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:42 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:42.741+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:43.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:42 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:42.741+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:43.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:42 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:42.741+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:43.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:42 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:42.741+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:43.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:42 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:42.741+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:43.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:42 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:42.741+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:43.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:42 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:42.912+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:43.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:42 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:42.912+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:43.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:42 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:42.912+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:43.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:42 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:42.912+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:43.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:42 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:42.912+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:43.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:42 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:42.912+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:43.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:43 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:43.038+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:22.734378+0000 front 2024-03-11T20:10:22.734587+0000 (oldest deadline 2024-03-11T20:10:49.135500+0000) 2024-03-11T20:11:43.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:43 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:43.038+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:34.939138+0000 front 2024-03-11T20:10:34.939023+0000 (oldest deadline 2024-03-11T20:11:00.239208+0000) 2024-03-11T20:11:43.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:43 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:43.038+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.841429+0000 front 2024-03-11T20:10:44.841536+0000 (oldest deadline 2024-03-11T20:11:08.341539+0000) 2024-03-11T20:11:43.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:43 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:43.038+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:53.446368+0000 front 2024-03-11T20:10:53.446196+0000 (oldest deadline 2024-03-11T20:11:16.346304+0000) 2024-03-11T20:11:43.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:43 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:43.038+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:02.148740+0000 front 2024-03-11T20:11:02.148992+0000 (oldest deadline 2024-03-11T20:11:23.848546+0000) 2024-03-11T20:11:43.406 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:43 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9[66779]: 2024-03-11T20:11:43.038+0000 7f05dde00700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.750499+0000 front 2024-03-11T20:11:09.750372+0000 (oldest deadline 2024-03-11T20:11:35.550882+0000) 2024-03-11T20:11:43.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:43 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:43.735+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:43.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:43 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:43.735+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:43.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:43 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:43.735+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:43.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:43 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:43.735+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:43.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:43 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:43.735+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:43.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:43 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:43.735+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:43.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:43 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:43.735+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:17.728599+0000 front 2024-03-11T20:11:17.728701+0000 (oldest deadline 2024-03-11T20:11:42.928816+0000) 2024-03-11T20:11:44.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:43 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:43.919+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:44.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:43 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:43.919+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:44.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:43 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:43.919+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:44.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:43 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:43.919+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:44.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:43 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:43.919+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:44.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:43 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:43.919+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:44.506 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:44 smithi097 podman[94470]: 2024-03-11 20:11:44.17982004 +0000 UTC m=+5.662635421 container died e0b34c2d0e31765222e412cb53ffbdb3519a3db2e53bc667c5f5af44ace3c5c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:11:44.850 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:44 smithi097 podman[94470]: 2024-03-11 20:11:44.677028839 +0000 UTC m=+6.159844257 container cleanup e0b34c2d0e31765222e412cb53ffbdb3519a3db2e53bc667c5f5af44ace3c5c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T20:11:44.850 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:44 smithi097 bash[94470]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9 2024-03-11T20:11:45.155 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:44 smithi097 podman[94489]: 2024-03-11 20:11:44.862031232 +0000 UTC m=+0.669185810 container remove e0b34c2d0e31765222e412cb53ffbdb3519a3db2e53bc667c5f5af44ace3c5c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, io.buildah.version=1.33.5) 2024-03-11T20:11:45.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:44 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:44.763+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:45.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:44 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:44.763+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:45.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:44 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:44.763+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:45.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:44 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:44.763+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:45.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:44 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:44.763+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:45.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:44 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:44.763+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:45.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:44 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:44.763+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:17.728599+0000 front 2024-03-11T20:11:17.728701+0000 (oldest deadline 2024-03-11T20:11:42.928816+0000) 2024-03-11T20:11:45.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:44 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:44.932+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:45.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:44 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:44.932+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:45.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:44 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:44.932+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:45.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:44 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:44.932+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:45.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:44 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:44.932+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:45.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:44 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:44.932+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:45.942 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:45 smithi097 podman[94573]: 2024-03-11 20:11:45.60024299 +0000 UTC m=+0.035456231 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T20:11:45.943 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:45 smithi097 podman[94573]: 2024-03-11 20:11:45.696018233 +0000 UTC m=+0.131231503 container create 1fe0f5bfb73c36201fcf19ebb0889217929e779931295715beceb50484470a1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9-deactivate, RELEASE=squid-94a5397, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:11:45.943 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:45 smithi097 podman[94573]: 2024-03-11 20:11:45.953855286 +0000 UTC m=+0.389068596 container init 1fe0f5bfb73c36201fcf19ebb0889217929e779931295715beceb50484470a1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:11:46.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:45 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:45.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:46.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:45 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:45.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:46.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:45 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:45.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:46.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:45 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:45.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:46.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:45 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:45.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:46.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:45 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:45.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:46.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:45 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:45.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:17.728599+0000 front 2024-03-11T20:11:17.728701+0000 (oldest deadline 2024-03-11T20:11:42.928816+0000) 2024-03-11T20:11:46.273 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:45 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:45.927+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:46.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:45 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:45.927+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:46.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:45 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:45.927+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:46.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:45 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:45.927+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:46.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:45 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:45.927+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:46.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:45 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:45.927+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:46.385 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:45 smithi097 podman[94573]: 2024-03-11 20:11:45.964802915 +0000 UTC m=+0.400016189 container start 1fe0f5bfb73c36201fcf19ebb0889217929e779931295715beceb50484470a1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-11T20:11:46.385 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:46 smithi097 podman[94573]: 2024-03-11 20:11:46.048586007 +0000 UTC m=+0.483799291 container attach 1fe0f5bfb73c36201fcf19ebb0889217929e779931295715beceb50484470a1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-11T20:11:46.655 INFO:journalctl@ceph.osd.9.smithi097.stdout:Mar 11 20:11:46 smithi097 podman[94573]: 2024-03-11 20:11:46.398022878 +0000 UTC m=+0.833236152 container died 1fe0f5bfb73c36201fcf19ebb0889217929e779931295715beceb50484470a1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T20:11:47.176 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.9.service' 2024-03-11T20:11:47.227 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:46 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:46.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:47.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:46 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:46.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:47.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:46 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:46.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:47.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:46 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:46.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:47.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:46 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:46.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:47.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:46 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:46.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:47.228 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:46 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:46.750+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:17.728599+0000 front 2024-03-11T20:11:17.728701+0000 (oldest deadline 2024-03-11T20:11:42.928816+0000) 2024-03-11T20:11:47.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:46 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:46.921+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:47.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:46 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:46.921+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:47.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:46 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:46.921+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:47.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:46 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:46.921+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:47.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:46 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:46.921+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:47.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:46 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:46.921+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:47.284 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:11:47.284 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-11T20:11:47.284 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-11T20:11:47.284 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.10 2024-03-11T20:11:47.575 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:47 smithi114 systemd[1]: Stopping Ceph osd.10 for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:11:47.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:47 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:47.586+0000 7f6aab259700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:11:47.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:47 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:47.586+0000 7f6aab259700 -1 osd.10 106 *** Got signal Terminated *** 2024-03-11T20:11:47.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:47 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:47.586+0000 7f6aab259700 -1 osd.10 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:11:47.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:47 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:47.779+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:47.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:47 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:47.779+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:47.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:47 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:47.779+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:47.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:47 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:47.779+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:47.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:47 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:47.779+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:47.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:47 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:47.779+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:47.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:47 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:47.779+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:17.728599+0000 front 2024-03-11T20:11:17.728701+0000 (oldest deadline 2024-03-11T20:11:42.928816+0000) 2024-03-11T20:11:48.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:47 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:47.935+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:48.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:47 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:47.935+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:48.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:47 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:47.935+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:48.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:47 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:47.935+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:48.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:47 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:47.935+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:48.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:47 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:47.935+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:48.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:48 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:48.738+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:48.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:48 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:48.738+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:48.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:48 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:48.738+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:48.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:48 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:48.738+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:48.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:48 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:48.738+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:48.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:48 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:48.738+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:48.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:48 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:48.738+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:17.728599+0000 front 2024-03-11T20:11:17.728701+0000 (oldest deadline 2024-03-11T20:11:42.928816+0000) 2024-03-11T20:11:49.024 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:48 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:48.888+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:49.024 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:48 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:48.888+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:49.024 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:48 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:48.888+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:49.024 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:48 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:48.888+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:49.024 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:48 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:48.888+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:49.024 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:48 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:48.888+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:49.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:49 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:49.712+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:49.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:49 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:49.712+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:49.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:49 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:49.712+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:49.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:49 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:49.712+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:49.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:49 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:49.712+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:49.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:49 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:49.712+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:49.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:49 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:49.712+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:17.728599+0000 front 2024-03-11T20:11:17.728701+0000 (oldest deadline 2024-03-11T20:11:42.928816+0000) 2024-03-11T20:11:50.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:49 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:49.861+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:50.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:49 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:49.861+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:50.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:49 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:49.861+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:50.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:49 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:49.861+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:50.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:49 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:49.861+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:50.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:49 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:49.861+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:50.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:50 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:50.676+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:50.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:50 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:50.676+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:50.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:50 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:50.676+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:50.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:50 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:50.676+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:50.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:50 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:50.676+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:50.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:50 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:50.676+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:50.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:50 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:50.676+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:17.728599+0000 front 2024-03-11T20:11:17.728701+0000 (oldest deadline 2024-03-11T20:11:42.928816+0000) 2024-03-11T20:11:51.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:50 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:50.822+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:51.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:50 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:50.822+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:51.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:50 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:50.822+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:51.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:50 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:50.822+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:51.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:50 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:50.822+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:51.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:50 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:50.822+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:51.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:51 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:51.663+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:23.620013+0000 front 2024-03-11T20:10:23.619727+0000 (oldest deadline 2024-03-11T20:10:48.919661+0000) 2024-03-11T20:11:51.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:51 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:51.663+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:36.521607+0000 front 2024-03-11T20:10:36.521696+0000 (oldest deadline 2024-03-11T20:10:59.421756+0000) 2024-03-11T20:11:51.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:51 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:51.663+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:44.722686+0000 front 2024-03-11T20:10:44.722862+0000 (oldest deadline 2024-03-11T20:11:08.222789+0000) 2024-03-11T20:11:51.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:51 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:51.663+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6814 osd.3 since back 2024-03-11T20:10:52.824374+0000 front 2024-03-11T20:10:52.824607+0000 (oldest deadline 2024-03-11T20:11:15.724212+0000) 2024-03-11T20:11:51.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:51 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:51.663+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:10:58.624899+0000 front 2024-03-11T20:10:58.624952+0000 (oldest deadline 2024-03-11T20:11:24.425414+0000) 2024-03-11T20:11:51.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:51 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:51.663+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:09.026788+0000 front 2024-03-11T20:11:09.026878+0000 (oldest deadline 2024-03-11T20:11:33.126769+0000) 2024-03-11T20:11:51.978 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:51 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10[61419]: 2024-03-11T20:11:51.663+0000 7f6aa3845700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:17.728599+0000 front 2024-03-11T20:11:17.728701+0000 (oldest deadline 2024-03-11T20:11:42.928816+0000) 2024-03-11T20:11:52.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:51 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:51.800+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:52.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:51 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:51.800+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:52.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:51 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:51.800+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:52.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:51 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:51.800+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:52.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:51 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:51.800+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:52.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:51 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:51.800+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:52.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:51 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:51.800+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6820 osd.7 since back 2024-03-11T20:11:25.782339+0000 front 2024-03-11T20:11:25.782474+0000 (oldest deadline 2024-03-11T20:11:51.582785+0000) 2024-03-11T20:11:53.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:52 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:52.847+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:53.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:52 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:52.847+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:53.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:52 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:52.847+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:53.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:52 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:52.847+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:53.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:52 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:52.847+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:53.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:52 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:52.847+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:53.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:52 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:52.847+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6820 osd.7 since back 2024-03-11T20:11:25.782339+0000 front 2024-03-11T20:11:25.782474+0000 (oldest deadline 2024-03-11T20:11:51.582785+0000) 2024-03-11T20:11:53.326 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:53 smithi114 podman[73547]: 2024-03-11 20:11:53.049744308 +0000 UTC m=+5.562895947 container died cf5216ee07067ff499ca66e94b3632e2e6356c7b32e1c5ec8613d91a895826d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397) 2024-03-11T20:11:53.668 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:53 smithi114 podman[73547]: 2024-03-11 20:11:53.437546304 +0000 UTC m=+5.950697943 container cleanup cf5216ee07067ff499ca66e94b3632e2e6356c7b32e1c5ec8613d91a895826d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T20:11:53.668 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:53 smithi114 bash[73547]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10 2024-03-11T20:11:53.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:53 smithi114 podman[73576]: 2024-03-11 20:11:53.680483879 +0000 UTC m=+0.624122804 container remove cf5216ee07067ff499ca66e94b3632e2e6356c7b32e1c5ec8613d91a895826d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-11T20:11:54.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:53 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:53.799+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:54.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:53 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:53.799+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:54.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:53 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:53.799+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:54.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:53 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:53.799+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:54.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:53 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:53.799+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:54.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:53 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:53.799+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:54.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:53 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:53.799+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6820 osd.7 since back 2024-03-11T20:11:25.782339+0000 front 2024-03-11T20:11:25.782474+0000 (oldest deadline 2024-03-11T20:11:51.582785+0000) 2024-03-11T20:11:54.432 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:53 smithi114 podman[73668]: 2024-03-11 20:11:53.892869267 +0000 UTC m=+0.017809418 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T20:11:54.433 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:53 smithi114 podman[73668]: 2024-03-11 20:11:53.997937638 +0000 UTC m=+0.122877819 container create 2afd7a5e40417e8e3835150075d058b1e055978482eacf3ba25a8703f58d3283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:11:54.433 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:54 smithi114 podman[73668]: 2024-03-11 20:11:54.158818089 +0000 UTC m=+0.283758252 container init 2afd7a5e40417e8e3835150075d058b1e055978482eacf3ba25a8703f58d3283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-94a5397) 2024-03-11T20:11:54.433 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:54 smithi114 podman[73668]: 2024-03-11 20:11:54.161761443 +0000 UTC m=+0.286701598 container start 2afd7a5e40417e8e3835150075d058b1e055978482eacf3ba25a8703f58d3283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10-deactivate, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T20:11:54.433 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:54 smithi114 podman[73668]: 2024-03-11 20:11:54.233515299 +0000 UTC m=+0.358455446 container attach 2afd7a5e40417e8e3835150075d058b1e055978482eacf3ba25a8703f58d3283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-11T20:11:54.723 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:54 smithi114 podman[73668]: 2024-03-11 20:11:54.445217276 +0000 UTC m=+0.570157412 container died 2afd7a5e40417e8e3835150075d058b1e055978482eacf3ba25a8703f58d3283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T20:11:54.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.10.service' 2024-03-11T20:11:54.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:54 smithi114 podman[73693]: 2024-03-11 20:11:54.953453961 +0000 UTC m=+0.503062754 container remove 2afd7a5e40417e8e3835150075d058b1e055978482eacf3ba25a8703f58d3283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T20:11:54.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:54 smithi114 systemd[1]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.10.service: Deactivated successfully. 2024-03-11T20:11:54.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:54 smithi114 systemd[1]: Stopped Ceph osd.10 for e81ca0d2-dfdf-11ee-95c8-87774f69a715. 2024-03-11T20:11:54.977 INFO:journalctl@ceph.osd.10.smithi114.stdout:Mar 11 20:11:54 smithi114 systemd[1]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.10.service: Consumed 19.366s CPU time. 2024-03-11T20:11:54.996 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:11:54.996 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-11T20:11:54.996 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-11T20:11:54.996 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.11 2024-03-11T20:11:55.273 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:54 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:54.835+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:55.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:54 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:54.835+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:55.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:54 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:54.835+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:55.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:54 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:54.835+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:55.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:54 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:54.835+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:55.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:54 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:54.835+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:55.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:54 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:54.835+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6820 osd.7 since back 2024-03-11T20:11:25.782339+0000 front 2024-03-11T20:11:25.782474+0000 (oldest deadline 2024-03-11T20:11:51.582785+0000) 2024-03-11T20:11:55.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:55 smithi161 systemd[1]: Stopping Ceph osd.11 for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:11:55.773 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:55 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:55.312+0000 7fdd27a12700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:11:55.774 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:55 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:55.312+0000 7fdd27a12700 -1 osd.11 106 *** Got signal Terminated *** 2024-03-11T20:11:55.774 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:55 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:55.312+0000 7fdd27a12700 -1 osd.11 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T20:11:56.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:55 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:55.867+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:56.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:55 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:55.867+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:56.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:55 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:55.867+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:56.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:55 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:55.867+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:56.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:55 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:55.867+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:56.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:55 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:55.867+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:56.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:55 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:55.867+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6820 osd.7 since back 2024-03-11T20:11:25.782339+0000 front 2024-03-11T20:11:25.782474+0000 (oldest deadline 2024-03-11T20:11:51.582785+0000) 2024-03-11T20:11:57.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:56 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:56.869+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:57.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:56 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:56.869+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:57.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:56 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:56.869+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:57.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:56 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:56.869+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:57.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:56 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:56.869+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:57.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:56 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:56.869+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:57.275 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:56 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:56.869+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6820 osd.7 since back 2024-03-11T20:11:25.782339+0000 front 2024-03-11T20:11:25.782474+0000 (oldest deadline 2024-03-11T20:11:51.582785+0000) 2024-03-11T20:11:58.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:57 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:57.865+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:58.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:57 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:57.865+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:58.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:57 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:57.865+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:58.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:57 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:57.865+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:58.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:57 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:57.865+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:58.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:57 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:57.865+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:58.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:57 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:57.865+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6820 osd.7 since back 2024-03-11T20:11:25.782339+0000 front 2024-03-11T20:11:25.782474+0000 (oldest deadline 2024-03-11T20:11:51.582785+0000) 2024-03-11T20:11:59.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:58 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:58.911+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:11:59.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:58 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:58.911+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:11:59.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:58 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:58.911+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:11:59.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:58 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:58.911+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:11:59.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:58 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:58.911+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:11:59.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:58 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:58.911+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:11:59.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:58 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:58.911+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6820 osd.7 since back 2024-03-11T20:11:25.782339+0000 front 2024-03-11T20:11:25.782474+0000 (oldest deadline 2024-03-11T20:11:51.582785+0000) 2024-03-11T20:11:59.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:58 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:58.911+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6820 osd.8 since back 2024-03-11T20:11:35.683823+0000 front 2024-03-11T20:11:35.683857+0000 (oldest deadline 2024-03-11T20:11:57.983856+0000) 2024-03-11T20:12:00.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:59 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:59.939+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-03-11T20:10:25.972594+0000 front 2024-03-11T20:10:25.972631+0000 (oldest deadline 2024-03-11T20:10:51.872554+0000) 2024-03-11T20:12:00.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:59 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:59.939+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6804 osd.1 since back 2024-03-11T20:10:35.874386+0000 front 2024-03-11T20:10:35.874278+0000 (oldest deadline 2024-03-11T20:10:59.974368+0000) 2024-03-11T20:12:00.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:59 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:59.939+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6804 osd.2 since back 2024-03-11T20:10:45.075983+0000 front 2024-03-11T20:10:45.076048+0000 (oldest deadline 2024-03-11T20:11:08.576137+0000) 2024-03-11T20:12:00.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:59 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:59.939+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6812 osd.4 since back 2024-03-11T20:11:01.478503+0000 front 2024-03-11T20:11:01.478421+0000 (oldest deadline 2024-03-11T20:11:26.178429+0000) 2024-03-11T20:12:00.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:59 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:59.939+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6812 osd.5 since back 2024-03-11T20:11:06.679379+0000 front 2024-03-11T20:11:06.679500+0000 (oldest deadline 2024-03-11T20:11:34.279944+0000) 2024-03-11T20:12:00.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:59 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:59.939+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.97:6822 osd.6 since back 2024-03-11T20:11:15.381019+0000 front 2024-03-11T20:11:15.381098+0000 (oldest deadline 2024-03-11T20:11:41.280734+0000) 2024-03-11T20:12:00.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:59 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:59.939+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.114:6820 osd.7 since back 2024-03-11T20:11:25.782339+0000 front 2024-03-11T20:11:25.782474+0000 (oldest deadline 2024-03-11T20:11:51.582785+0000) 2024-03-11T20:12:00.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:11:59 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11[62068]: 2024-03-11T20:11:59.939+0000 7fdd1fffe700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.161:6820 osd.8 since back 2024-03-11T20:11:35.683823+0000 front 2024-03-11T20:11:35.683857+0000 (oldest deadline 2024-03-11T20:11:57.983856+0000) 2024-03-11T20:12:00.992 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:12:00 smithi161 podman[74550]: 2024-03-11 20:12:00.70695325 +0000 UTC m=+5.505372897 container stop 9151dbf7de92fed4e52c25a1bdd27c2f3ae6f570d62f01ff61566a3a9f6e992f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-11T20:12:00.993 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:12:00 smithi161 podman[74550]: 2024-03-11 20:12:00.732685952 +0000 UTC m=+5.531105603 container died 9151dbf7de92fed4e52c25a1bdd27c2f3ae6f570d62f01ff61566a3a9f6e992f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T20:12:01.273 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:12:01 smithi161 podman[74550]: 2024-03-11 20:12:01.120604205 +0000 UTC m=+5.919023848 container cleanup 9151dbf7de92fed4e52c25a1bdd27c2f3ae6f570d62f01ff61566a3a9f6e992f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T20:12:01.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:12:01 smithi161 bash[74550]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11 2024-03-11T20:12:01.634 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:12:01 smithi161 podman[74567]: 2024-03-11 20:12:01.338246105 +0000 UTC m=+0.619483386 container remove 9151dbf7de92fed4e52c25a1bdd27c2f3ae6f570d62f01ff61566a3a9f6e992f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T20:12:01.937 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:12:01 smithi161 podman[74659]: 2024-03-11 20:12:01.647381594 +0000 UTC m=+0.100335437 container create 3f405bb01687a62e75b24ae7ab371245c9f906c123498dfd99644f840a87ed5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397) 2024-03-11T20:12:01.938 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:12:01 smithi161 podman[74659]: 2024-03-11 20:12:01.564744227 +0000 UTC m=+0.017698016 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T20:12:01.938 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:12:01 smithi161 podman[74659]: 2024-03-11 20:12:01.866021295 +0000 UTC m=+0.318975094 container init 3f405bb01687a62e75b24ae7ab371245c9f906c123498dfd99644f840a87ed5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-11T20:12:01.938 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:12:01 smithi161 podman[74659]: 2024-03-11 20:12:01.869251162 +0000 UTC m=+0.322204945 container start 3f405bb01687a62e75b24ae7ab371245c9f906c123498dfd99644f840a87ed5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397) 2024-03-11T20:12:02.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:12:01 smithi161 podman[74659]: 2024-03-11 20:12:01.949420785 +0000 UTC m=+0.402374571 container attach 3f405bb01687a62e75b24ae7ab371245c9f906c123498dfd99644f840a87ed5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T20:12:02.274 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:12:02 smithi161 podman[74659]: 2024-03-11 20:12:02.163102424 +0000 UTC m=+0.616056210 container died 3f405bb01687a62e75b24ae7ab371245c9f906c123498dfd99644f840a87ed5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-11T20:12:02.761 INFO:journalctl@ceph.osd.11.smithi161.stdout:Mar 11 20:12:02 smithi161 podman[74685]: 2024-03-11 20:12:02.760575748 +0000 UTC m=+0.589533158 container remove 3f405bb01687a62e75b24ae7ab371245c9f906c123498dfd99644f840a87ed5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-osd-11-deactivate, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-11T20:12:02.766 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@osd.11.service' 2024-03-11T20:12:02.808 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:12:02.809 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-11T20:12:02.809 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-11T20:12:02.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.a 2024-03-11T20:12:03.156 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 20:12:02 smithi097 systemd[1]: Stopping Ceph mds.a for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:12:03.655 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 20:12:03 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-a[73765]: 2024-03-11T20:12:03.410+0000 7f7d2b800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:12:03.656 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 20:12:03 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-a[73765]: 2024-03-11T20:12:03.410+0000 7f7d2b800700 -1 mds.a *** got signal Terminated *** 2024-03-11T20:12:07.738 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 20:12:07 smithi097 podman[94700]: 2024-03-11 20:12:07.42720123 +0000 UTC m=+4.149899204 container died f4167ce20b3adef6134dea02ab9b4bbb3bde611795f344af866a0d11092717a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-a, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-11T20:12:08.134 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 20:12:07 smithi097 podman[94700]: 2024-03-11 20:12:07.861571491 +0000 UTC m=+4.584269455 container cleanup f4167ce20b3adef6134dea02ab9b4bbb3bde611795f344af866a0d11092717a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-a, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T20:12:08.134 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 20:12:07 smithi097 bash[94700]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-a 2024-03-11T20:12:08.405 INFO:journalctl@ceph.mds.a.smithi097.stdout:Mar 11 20:12:08 smithi097 podman[94714]: 2024-03-11 20:12:08.146166611 +0000 UTC m=+0.703599797 container remove f4167ce20b3adef6134dea02ab9b4bbb3bde611795f344af866a0d11092717a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T20:12:08.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.a.service' 2024-03-11T20:12:08.692 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:12:08.692 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-11T20:12:08.692 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-11T20:12:08.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.d 2024-03-11T20:12:08.905 INFO:journalctl@ceph.mds.d.smithi097.stdout:Mar 11 20:12:08 smithi097 systemd[1]: Stopping Ceph mds.d for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:12:09.405 INFO:journalctl@ceph.mds.d.smithi097.stdout:Mar 11 20:12:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-d[72782]: 2024-03-11T20:12:09.221+0000 7f2b62400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:12:09.406 INFO:journalctl@ceph.mds.d.smithi097.stdout:Mar 11 20:12:09 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-d[72782]: 2024-03-11T20:12:09.221+0000 7f2b62400700 -1 mds.d *** got signal Terminated *** 2024-03-11T20:12:12.405 INFO:journalctl@ceph.mds.d.smithi097.stdout:Mar 11 20:12:12 smithi097 podman[94821]: 2024-03-11 20:12:12.118644649 +0000 UTC m=+3.029467409 container died 8295e80e4b3ae98fd9017ef1ee49820e3ae6177fe129b8caca862d4dfcc77424 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-11T20:12:12.829 INFO:journalctl@ceph.mds.d.smithi097.stdout:Mar 11 20:12:12 smithi097 podman[94821]: 2024-03-11 20:12:12.57336175 +0000 UTC m=+3.484184556 container cleanup 8295e80e4b3ae98fd9017ef1ee49820e3ae6177fe129b8caca862d4dfcc77424 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-11T20:12:12.830 INFO:journalctl@ceph.mds.d.smithi097.stdout:Mar 11 20:12:12 smithi097 bash[94821]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-d 2024-03-11T20:12:13.156 INFO:journalctl@ceph.mds.d.smithi097.stdout:Mar 11 20:12:12 smithi097 podman[94835]: 2024-03-11 20:12:12.84110836 +0000 UTC m=+0.705232915 container remove 8295e80e4b3ae98fd9017ef1ee49820e3ae6177fe129b8caca862d4dfcc77424 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-11T20:12:13.354 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.d.service' 2024-03-11T20:12:13.472 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:12:13.472 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-11T20:12:13.472 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-11T20:12:13.472 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.g 2024-03-11T20:12:13.655 INFO:journalctl@ceph.mds.g.smithi097.stdout:Mar 11 20:12:13 smithi097 systemd[1]: Stopping Ceph mds.g for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:12:14.406 INFO:journalctl@ceph.mds.g.smithi097.stdout:Mar 11 20:12:14 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-g[73399]: 2024-03-11T20:12:14.003+0000 7f9e00a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:12:14.406 INFO:journalctl@ceph.mds.g.smithi097.stdout:Mar 11 20:12:14 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-g[73399]: 2024-03-11T20:12:14.003+0000 7f9e00a00700 -1 mds.g *** got signal Terminated *** 2024-03-11T20:12:16.277 INFO:journalctl@ceph.mds.g.smithi097.stdout:Mar 11 20:12:15 smithi097 podman[94940]: 2024-03-11 20:12:15.959579335 +0000 UTC m=+2.082247249 container died 7d64ddc1dbc3bbb38abd6ae14f310d750a913ae1072b14a9dc540a63ca64e2b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-11T20:12:16.655 INFO:journalctl@ceph.mds.g.smithi097.stdout:Mar 11 20:12:16 smithi097 podman[94940]: 2024-03-11 20:12:16.410148571 +0000 UTC m=+2.532816486 container cleanup 7d64ddc1dbc3bbb38abd6ae14f310d750a913ae1072b14a9dc540a63ca64e2b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:12:16.655 INFO:journalctl@ceph.mds.g.smithi097.stdout:Mar 11 20:12:16 smithi097 bash[94940]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-g 2024-03-11T20:12:17.061 INFO:journalctl@ceph.mds.g.smithi097.stdout:Mar 11 20:12:16 smithi097 podman[94954]: 2024-03-11 20:12:16.70412132 +0000 UTC m=+0.728648988 container remove 7d64ddc1dbc3bbb38abd6ae14f310d750a913ae1072b14a9dc540a63ca64e2b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-g, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-03-11T20:12:17.092 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.g.service' 2024-03-11T20:12:17.208 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:12:17.209 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-11T20:12:17.209 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-11T20:12:17.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.j 2024-03-11T20:12:17.655 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 20:12:17 smithi097 systemd[1]: Stopping Ceph mds.j for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:12:18.155 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 20:12:17 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-j[72416]: 2024-03-11T20:12:17.893+0000 7f1ccb400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:12:18.155 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 20:12:17 smithi097 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-j[72416]: 2024-03-11T20:12:17.893+0000 7f1ccb400700 -1 mds.j *** got signal Terminated *** 2024-03-11T20:12:21.655 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 20:12:21 smithi097 podman[95061]: 2024-03-11 20:12:21.31598398 +0000 UTC m=+3.610078656 container died 80123fcb5e22646809d9a0330bcf9322fcc9a03c9401c8ca999bee1f56101614 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-11T20:12:21.905 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 20:12:21 smithi097 podman[95061]: 2024-03-11 20:12:21.830470287 +0000 UTC m=+4.124564961 container cleanup 80123fcb5e22646809d9a0330bcf9322fcc9a03c9401c8ca999bee1f56101614 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-94a5397) 2024-03-11T20:12:21.905 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 20:12:21 smithi097 bash[95061]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-j 2024-03-11T20:12:22.405 INFO:journalctl@ceph.mds.j.smithi097.stdout:Mar 11 20:12:22 smithi097 podman[95088]: 2024-03-11 20:12:22.02290057 +0000 UTC m=+0.695867452 container remove 80123fcb5e22646809d9a0330bcf9322fcc9a03c9401c8ca999bee1f56101614 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T20:12:22.489 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.j.service' 2024-03-11T20:12:22.594 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:12:22.594 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-11T20:12:22.595 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-11T20:12:22.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.b 2024-03-11T20:12:22.882 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 20:12:22 smithi114 systemd[1]: Stopping Ceph mds.b for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:12:23.145 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 20:12:22 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b[66693]: 2024-03-11T20:12:22.894+0000 7f263a71f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:12:23.146 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 20:12:22 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b[66693]: 2024-03-11T20:12:22.894+0000 7f263a71f700 -1 mds.b *** got signal Terminated *** 2024-03-11T20:12:24.325 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 20:12:24 smithi114 podman[73774]: 2024-03-11 20:12:24.085697506 +0000 UTC m=+1.288922641 container died 1d053fe741d3a18d257e4dc71628c7495208c1034a44539eb859847543088af1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-11T20:12:24.667 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 20:12:24 smithi114 podman[73774]: 2024-03-11 20:12:24.428954716 +0000 UTC m=+1.632179857 container cleanup 1d053fe741d3a18d257e4dc71628c7495208c1034a44539eb859847543088af1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-03-11T20:12:24.667 INFO:journalctl@ceph.mds.b.smithi114.stdout:Mar 11 20:12:24 smithi114 bash[73774]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-b 2024-03-11T20:12:24.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.b.service' 2024-03-11T20:12:24.877 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:12:24.877 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-11T20:12:24.877 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-11T20:12:24.878 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.e 2024-03-11T20:12:25.227 INFO:journalctl@ceph.mds.e.smithi114.stdout:Mar 11 20:12:24 smithi114 systemd[1]: Stopping Ceph mds.e for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:12:25.227 INFO:journalctl@ceph.mds.e.smithi114.stdout:Mar 11 20:12:25 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-e[66056]: 2024-03-11T20:12:25.198+0000 7f3f222cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:12:25.227 INFO:journalctl@ceph.mds.e.smithi114.stdout:Mar 11 20:12:25 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-e[66056]: 2024-03-11T20:12:25.198+0000 7f3f222cd700 -1 mds.e *** got signal Terminated *** 2024-03-11T20:12:28.328 INFO:journalctl@ceph.mds.e.smithi114.stdout:Mar 11 20:12:28 smithi114 podman[73908]: 2024-03-11 20:12:28.06947376 +0000 UTC m=+2.962257355 container died 8ef026329e7f2ea4d2996e7c055a8a6030f6750991c56f913b0d0bcf259d9d47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-e, RELEASE=squid-94a5397, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-11T20:12:28.712 INFO:journalctl@ceph.mds.e.smithi114.stdout:Mar 11 20:12:28 smithi114 podman[73908]: 2024-03-11 20:12:28.440991302 +0000 UTC m=+3.333774888 container cleanup 8ef026329e7f2ea4d2996e7c055a8a6030f6750991c56f913b0d0bcf259d9d47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-e, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T20:12:28.712 INFO:journalctl@ceph.mds.e.smithi114.stdout:Mar 11 20:12:28 smithi114 bash[73908]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-e 2024-03-11T20:12:28.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.e.service' 2024-03-11T20:12:28.888 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:12:28.889 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-11T20:12:28.889 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-11T20:12:28.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.h 2024-03-11T20:12:29.227 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 20:12:28 smithi114 systemd[1]: Stopping Ceph mds.h for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:12:29.227 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 20:12:29 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-h[66370]: 2024-03-11T20:12:29.210+0000 7fa4f139f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:12:29.227 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 20:12:29 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-h[66370]: 2024-03-11T20:12:29.210+0000 7fa4f139f700 -1 mds.h *** got signal Terminated *** 2024-03-11T20:12:32.477 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 20:12:32 smithi114 podman[74031]: 2024-03-11 20:12:32.080012521 +0000 UTC m=+2.965056106 container died cbd981c5cca1f85f50cab2d02ff65d5397c4f0bbb74110e69558b41413269e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-h, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-11T20:12:32.799 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 20:12:32 smithi114 podman[74031]: 2024-03-11 20:12:32.577849751 +0000 UTC m=+3.462893355 container cleanup cbd981c5cca1f85f50cab2d02ff65d5397c4f0bbb74110e69558b41413269e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-h, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T20:12:32.799 INFO:journalctl@ceph.mds.h.smithi114.stdout:Mar 11 20:12:32 smithi114 bash[74031]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-h 2024-03-11T20:12:32.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.h.service' 2024-03-11T20:12:32.962 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:12:32.962 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-11T20:12:32.962 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-11T20:12:32.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.k 2024-03-11T20:12:33.477 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 20:12:33 smithi114 systemd[1]: Stopping Ceph mds.k for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:12:33.477 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 20:12:33 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-k[65736]: 2024-03-11T20:12:33.297+0000 7f52918fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:12:33.477 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 20:12:33 smithi114 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-k[65736]: 2024-03-11T20:12:33.297+0000 7f52918fa700 -1 mds.k *** got signal Terminated *** 2024-03-11T20:12:36.369 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 20:12:36 smithi114 podman[74150]: 2024-03-11 20:12:36.073974882 +0000 UTC m=+2.878615419 container died 80803eebc596ba60064a0fa12ffff3713a88a20654aea32ce6d8c1d30f511266 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:12:36.685 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 20:12:36 smithi114 podman[74150]: 2024-03-11 20:12:36.46484725 +0000 UTC m=+3.269487789 container cleanup 80803eebc596ba60064a0fa12ffff3713a88a20654aea32ce6d8c1d30f511266 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-k, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-11T20:12:36.686 INFO:journalctl@ceph.mds.k.smithi114.stdout:Mar 11 20:12:36 smithi114 bash[74150]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-k 2024-03-11T20:12:36.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.k.service' 2024-03-11T20:12:36.862 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:12:36.863 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-11T20:12:36.863 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-11T20:12:36.863 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.c 2024-03-11T20:12:37.259 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 20:12:36 smithi161 systemd[1]: Stopping Ceph mds.c for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:12:37.523 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 20:12:37 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-c[67247]: 2024-03-11T20:12:37.276+0000 7f8dec047700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:12:37.523 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 20:12:37 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-c[67247]: 2024-03-11T20:12:37.276+0000 7f8dec047700 -1 mds.c *** got signal Terminated *** 2024-03-11T20:12:40.524 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 20:12:40 smithi161 podman[74765]: 2024-03-11 20:12:40.093108591 +0000 UTC m=+3.023146562 container died 1a128a11cdb93320cb689a669d9bb3a4d32aeab2c086a2fc751ba604b4575491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-11T20:12:40.931 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 20:12:40 smithi161 podman[74765]: 2024-03-11 20:12:40.693226968 +0000 UTC m=+3.623264935 container cleanup 1a128a11cdb93320cb689a669d9bb3a4d32aeab2c086a2fc751ba604b4575491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T20:12:40.931 INFO:journalctl@ceph.mds.c.smithi161.stdout:Mar 11 20:12:40 smithi161 bash[74765]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-c 2024-03-11T20:12:41.053 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.c.service' 2024-03-11T20:12:41.092 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:12:41.092 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-11T20:12:41.092 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-11T20:12:41.092 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.f 2024-03-11T20:12:41.273 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 20:12:41 smithi161 systemd[1]: Stopping Ceph mds.f for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:12:41.773 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 20:12:41 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-f[66940]: 2024-03-11T20:12:41.421+0000 7f09e1c77700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:12:41.773 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 20:12:41 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-f[66940]: 2024-03-11T20:12:41.421+0000 7f09e1c77700 -1 mds.f *** got signal Terminated *** 2024-03-11T20:12:44.443 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 20:12:44 smithi161 podman[74895]: 2024-03-11 20:12:44.095128923 +0000 UTC m=+2.774211791 container died 7764197c160f3108a63d0eef42a22565f7a84d02f6c8d8e704cd70e5d7e1e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-f, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-11T20:12:44.773 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 20:12:44 smithi161 podman[74895]: 2024-03-11 20:12:44.563228669 +0000 UTC m=+3.242311535 container cleanup 7764197c160f3108a63d0eef42a22565f7a84d02f6c8d8e704cd70e5d7e1e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-f, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-11T20:12:44.774 INFO:journalctl@ceph.mds.f.smithi161.stdout:Mar 11 20:12:44 smithi161 bash[74895]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-f 2024-03-11T20:12:44.975 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.f.service' 2024-03-11T20:12:45.010 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:12:45.011 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-11T20:12:45.011 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-11T20:12:45.011 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.i 2024-03-11T20:12:45.523 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 20:12:45 smithi161 systemd[1]: Stopping Ceph mds.i for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:12:45.523 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 20:12:45 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-i[67556]: 2024-03-11T20:12:45.316+0000 7fa8fcaeb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:12:45.523 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 20:12:45 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-i[67556]: 2024-03-11T20:12:45.316+0000 7fa8fcaeb700 -1 mds.i *** got signal Terminated *** 2024-03-11T20:12:48.346 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 20:12:48 smithi161 podman[75015]: 2024-03-11 20:12:48.065839647 +0000 UTC m=+2.830500585 container died 2e4833f4daa648d3f915e8621d8fa3dc19e910bfeb7af0ec823850b96f63a99b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-i, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-11T20:12:48.679 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 20:12:48 smithi161 podman[75015]: 2024-03-11 20:12:48.44171391 +0000 UTC m=+3.206374848 container cleanup 2e4833f4daa648d3f915e8621d8fa3dc19e910bfeb7af0ec823850b96f63a99b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-i, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:12:48.679 INFO:journalctl@ceph.mds.i.smithi161.stdout:Mar 11 20:12:48 smithi161 bash[75015]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-i 2024-03-11T20:12:48.805 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.i.service' 2024-03-11T20:12:48.840 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:12:48.840 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-11T20:12:48.840 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-11T20:12:48.840 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.l 2024-03-11T20:12:48.966 INFO:journalctl@ceph.mds.l.smithi161.stdout:Mar 11 20:12:48 smithi161 systemd[1]: Stopping Ceph mds.l for e81ca0d2-dfdf-11ee-95c8-87774f69a715... 2024-03-11T20:12:49.273 INFO:journalctl@ceph.mds.l.smithi161.stdout:Mar 11 20:12:49 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-l[66632]: 2024-03-11T20:12:49.177+0000 7fb016764700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:12:49.273 INFO:journalctl@ceph.mds.l.smithi161.stdout:Mar 11 20:12:49 smithi161 ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-l[66632]: 2024-03-11T20:12:49.177+0000 7fb016764700 -1 mds.l *** got signal Terminated *** 2024-03-11T20:12:52.339 INFO:journalctl@ceph.mds.l.smithi161.stdout:Mar 11 20:12:52 smithi161 podman[75136]: 2024-03-11 20:12:52.067153601 +0000 UTC m=+2.992381343 container died 4691c5ad38edfd0cc8dcf4b66ce5e2ce58cd4da6964494ae5e13b024e9a0ce27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-l, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:12:52.340 INFO:journalctl@ceph.mds.l.smithi161.stdout:Mar 11 20:12:52 smithi161 podman[75136]: 2024-03-11 20:12:52.352709266 +0000 UTC m=+3.277937004 container cleanup 4691c5ad38edfd0cc8dcf4b66ce5e2ce58cd4da6964494ae5e13b024e9a0ce27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-l, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:12:52.690 INFO:journalctl@ceph.mds.l.smithi161.stdout:Mar 11 20:12:52 smithi161 bash[75136]: ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-l 2024-03-11T20:12:52.690 INFO:journalctl@ceph.mds.l.smithi161.stdout:Mar 11 20:12:52 smithi161 podman[75150]: 2024-03-11 20:12:52.595645092 +0000 UTC m=+0.523191325 container remove 4691c5ad38edfd0cc8dcf4b66ce5e2ce58cd4da6964494ae5e13b024e9a0ce27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715-mds-l, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T20:12:52.705 DEBUG:teuthology.orchestra.run.smithi161:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e81ca0d2-dfdf-11ee-95c8-87774f69a715@mds.l.service' 2024-03-11T20:12:52.739 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T20:12:52.739 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-11T20:12:52.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T20:12:53.061 INFO:teuthology.orchestra.run.smithi097.stdout:Deleting cluster with fsid: e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T20:12:57.872 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T20:12:58.060 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T20:13:00.543 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T20:13:00.736 INFO:teuthology.orchestra.run.smithi161.stdout:Deleting cluster with fsid: e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T20:13:03.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T20:13:03.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T20:13:03.327 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T20:13:03.355 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T20:13:03.357 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293/remote/smithi097/crash 2024-03-11T20:13:03.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/crash -- . 2024-03-11T20:13:03.467 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T20:13:03.467 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2024-03-11T20:13:03.472 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293/remote/smithi114/crash 2024-03-11T20:13:03.473 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/crash -- . 2024-03-11T20:13:03.526 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T20:13:03.526 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now 2024-03-11T20:13:03.528 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293/remote/smithi161/crash 2024-03-11T20:13:03.529 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/crash -- . 2024-03-11T20:13:03.578 INFO:teuthology.orchestra.run.smithi161.stderr:tar: /var/lib/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T20:13:03.579 INFO:teuthology.orchestra.run.smithi161.stderr:tar: Error is not recoverable: exiting now 2024-03-11T20:13:03.580 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T20:13:03.580 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-11T20:13:03.689 INFO:teuthology.orchestra.run.smithi097.stdout:2024-03-11T19:48:53.729879+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T20:13:03.690 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T20:13:03.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[SEC\]' /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-11T20:13:03.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[ERR\]' /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-11T20:13:03.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[WRN\]' /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-11T20:13:03.938 INFO:teuthology.orchestra.run.smithi097.stdout:2024-03-11T19:48:53.729879+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T20:13:03.939 INFO:tasks.cephadm:Compressing logs... 2024-03-11T20:13:03.940 DEBUG:teuthology.orchestra.run.smithi097:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T20:13:03.944 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T20:13:03.946 DEBUG:teuthology.orchestra.run.smithi161:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T20:13:03.975 INFO:teuthology.orchestra.run.smithi161.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T20:13:03.975 INFO:teuthology.orchestra.run.smithi161.stderr:: No such file or directory 2024-03-11T20:13:03.975 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.l.log 2024-03-11T20:13:03.975 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T20:13:03.975 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.i.log 2024-03-11T20:13:03.975 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.l.log: /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.11.log 2024-03-11T20:13:03.975 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mgr.z.log 2024-03-11T20:13:03.975 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.i.log: /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.8.log 2024-03-11T20:13:03.978 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose 92.4% -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-11T20:13:03.978 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T20:13:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-03-11T20:13:03.981 INFO:teuthology.orchestra.run.smithi114.stderr: /var/log/ceph/cephadm.log 2024-03-11T20:13:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.10.log 2024-03-11T20:13:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T20:13:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.b.log 2024-03-11T20:13:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-11T20:13:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T20:13:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.b.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T20:13:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.k.log 2024-03-11T20:13:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.h.log 2024-03-11T20:13:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-11T20:13:03.981 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.e.log 2024-03-11T20:13:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.7.log: 90.1% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T20:13:03.982 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T20:13:03.986 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.f.log 2024-03-11T20:13:03.986 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T20:13:03.986 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.f.log: 90.1% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T20:13:03.986 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.log 2024-03-11T20:13:03.986 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.cephadm.log: 83.2% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T20:13:03.986 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T20:13:03.986 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.log: 87.4% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T20:13:03.986 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.c.log 2024-03-11T20:13:03.986 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.2.log: 92.0% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mgr.z.log.gz 2024-03-11T20:13:03.986 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T20:13:03.988 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T20:13:03.998 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T20:13:04.003 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mgr.y.log: 91.9%gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.log 2024-03-11T20:13:04.003 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.cephadm.log: 83.3% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T20:13:04.005 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-11T20:13:04.005 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T20:13:04.006 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-11T20:13:04.007 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-11T20:13:04.008 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T20:13:04.009 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.6.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-11T20:13:04.009 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T20:13:04.010 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.g.log 2024-03-11T20:13:04.011 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T20:13:04.012 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.j.log 2024-03-11T20:13:04.013 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-11T20:13:04.016 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.j.log: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T20:13:04.016 INFO:teuthology.orchestra.run.smithi097.stderr: 90.0% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T20:13:04.016 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T20:13:04.017 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T20:13:04.018 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mgr.x.log: 94.4% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.g.log.gz 2024-03-11T20:13:04.020 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T20:13:04.020 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.0.log: 94.4% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.j.log.gz 2024-03-11T20:13:04.021 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T20:13:04.022 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.log 2024-03-11T20:13:04.023 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.log: 87.4% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T20:13:04.024 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T20:13:04.024 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T20:13:04.025 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T20:13:04.026 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.log: /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.9.log 2024-03-11T20:13:04.028 INFO:teuthology.orchestra.run.smithi097.stderr: 87.4% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T20:13:04.029 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.d.log 2024-03-11T20:13:04.031 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.a.log 2024-03-11T20:13:04.036 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T20:13:04.040 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-volume.log: 93.2% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.k.log.gz 2024-03-11T20:13:04.042 INFO:teuthology.orchestra.run.smithi161.stderr: 94.5% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.l.log.gz 2024-03-11T20:13:04.044 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mon.b.log: 93.0% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.h.log.gz 2024-03-11T20:13:04.045 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.d.log: /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.a.log: 90.9% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T20:13:04.045 INFO:teuthology.orchestra.run.smithi114.stderr: 94.4% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.e.log.gz 2024-03-11T20:13:04.048 INFO:teuthology.orchestra.run.smithi161.stderr: 93.5% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.i.log.gz 2024-03-11T20:13:04.049 INFO:teuthology.orchestra.run.smithi114.stderr: 90.9% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T20:13:04.062 INFO:teuthology.orchestra.run.smithi161.stderr: 94.7% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.f.log.gz 2024-03-11T20:13:04.077 INFO:teuthology.orchestra.run.smithi161.stderr: 93.9% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.c.log.gz 2024-03-11T20:13:04.116 INFO:teuthology.orchestra.run.smithi097.stderr: 92.8% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.a.log.gz 2024-03-11T20:13:04.152 INFO:teuthology.orchestra.run.smithi097.stderr: 94.4% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.d.log.gz 2024-03-11T20:13:04.306 INFO:teuthology.orchestra.run.smithi114.stderr: 93.1% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mds.b.log.gz 2024-03-11T20:13:04.378 INFO:teuthology.orchestra.run.smithi161.stderr: 91.9% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T20:13:04.379 INFO:teuthology.orchestra.run.smithi097.stderr: 89.9% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T20:13:04.451 INFO:teuthology.orchestra.run.smithi114.stderr: 91.7% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T20:13:05.349 INFO:teuthology.orchestra.run.smithi097.stderr: 90.5% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T20:13:05.831 INFO:teuthology.orchestra.run.smithi114.stderr: 92.2% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T20:13:06.079 INFO:teuthology.orchestra.run.smithi161.stderr: 92.9% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.11.log.gz 2024-03-11T20:13:06.481 INFO:teuthology.orchestra.run.smithi097.stderr: 91.9% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-11T20:13:06.560 INFO:teuthology.orchestra.run.smithi161.stderr: 92.4% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.8.log.gz 2024-03-11T20:13:07.053 INFO:teuthology.orchestra.run.smithi161.stderr: 92.4% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T20:13:07.266 INFO:teuthology.orchestra.run.smithi097.stderr: 92.5% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T20:13:07.390 INFO:teuthology.orchestra.run.smithi114.stderr: 92.4% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-11T20:13:07.408 INFO:teuthology.orchestra.run.smithi161.stderr: 92.5% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-11T20:13:07.410 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-11T20:13:07.410 INFO:teuthology.orchestra.run.smithi161.stderr:real 0m3.449s 2024-03-11T20:13:07.410 INFO:teuthology.orchestra.run.smithi161.stderr:user 0m11.470s 2024-03-11T20:13:07.410 INFO:teuthology.orchestra.run.smithi161.stderr:sys 0m0.400s 2024-03-11T20:13:07.462 INFO:teuthology.orchestra.run.smithi114.stderr: 92.4% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.10.log.gz 2024-03-11T20:13:07.826 INFO:teuthology.orchestra.run.smithi097.stderr: 91.7% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-11T20:13:07.828 INFO:teuthology.orchestra.run.smithi114.stderr: 91.7% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-11T20:13:07.831 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-11T20:13:07.831 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m3.870s 2024-03-11T20:13:07.831 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m13.122s 2024-03-11T20:13:07.831 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m0.435s 2024-03-11T20:13:08.158 INFO:teuthology.orchestra.run.smithi097.stderr: 92.1% -- replaced with /var/log/ceph/e81ca0d2-dfdf-11ee-95c8-87774f69a715/ceph-osd.9.log.gz 2024-03-11T20:13:08.163 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-11T20:13:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:real 0m4.188s 2024-03-11T20:13:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:user 0m14.432s 2024-03-11T20:13:08.163 INFO:teuthology.orchestra.run.smithi097.stderr:sys 0m1.173s 2024-03-11T20:13:08.163 INFO:tasks.cephadm:Archiving logs... 2024-03-11T20:13:08.164 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293/remote/smithi097/log 2024-03-11T20:13:08.165 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T20:13:10.684 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293/remote/smithi114/log 2024-03-11T20:13:10.685 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T20:13:12.828 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293/remote/smithi161/log 2024-03-11T20:13:12.830 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T20:13:14.714 INFO:tasks.cephadm:Removing cluster... 2024-03-11T20:13:14.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 --force 2024-03-11T20:13:15.044 INFO:teuthology.orchestra.run.smithi097.stdout:Deleting cluster with fsid: e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T20:13:15.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 --force 2024-03-11T20:13:15.864 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T20:13:16.138 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e81ca0d2-dfdf-11ee-95c8-87774f69a715 --force 2024-03-11T20:13:16.332 INFO:teuthology.orchestra.run.smithi161.stdout:Deleting cluster with fsid: e81ca0d2-dfdf-11ee-95c8-87774f69a715 2024-03-11T20:13:16.596 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T20:13:17.286 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T20:13:17.341 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T20:13:17.360 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T20:13:17.378 INFO:tasks.cephadm:Teardown complete 2024-03-11T20:13:17.378 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-11T20:13:17.895 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-11T20:13:17.896 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T20:13:17.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T20:13:17.908 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T20:13:18.063 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T20:13:18.079 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T20:13:18.080 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T20:13:18.088 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T20:13:18.090 DEBUG:teuthology.orchestra.run.smithi161:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T20:13:18.104 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-03-11T20:13:18.107 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-03-11T20:13:18.109 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T20:13:18.109 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-03-11T20:13:18.109 INFO:teuthology.orchestra.run.smithi114.stdout:^- hv01.front.sepia.ceph.com 2 6 377 17 +352us[ +352us] +/- 163ms 2024-03-11T20:13:18.109 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv02.front.sepia.ceph.com 3 7 377 79 -1759us[-1827us] +/- 63ms 2024-03-11T20:13:18.109 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 +2393us[+2393us] +/- 71ms 2024-03-11T20:13:18.109 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T20:13:18.111 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T20:13:18.112 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-03-11T20:13:18.112 INFO:teuthology.orchestra.run.smithi161.stdout:^- hv01.front.sepia.ceph.com 2 6 377 17 +420us[ +326us] +/- 163ms 2024-03-11T20:13:18.112 INFO:teuthology.orchestra.run.smithi161.stdout:^* hv02.front.sepia.ceph.com 3 7 377 14 -1707us[-1802us] +/- 64ms 2024-03-11T20:13:18.112 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +2431us[+2431us] +/- 71ms 2024-03-11T20:13:18.112 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T20:13:18.134 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-03-11T20:13:18.144 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T20:13:18.144 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-03-11T20:13:18.144 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 +328us[ +328us] +/- 163ms 2024-03-11T20:13:18.144 INFO:teuthology.orchestra.run.smithi097.stdout:^* hv02.front.sepia.ceph.com 3 8 377 19 -1789us[-1815us] +/- 64ms 2024-03-11T20:13:18.144 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +2322us[+2322us] +/- 71ms 2024-03-11T20:13:18.144 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T20:13:18.145 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T20:13:18.156 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T20:13:18.157 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T20:13:18.179 DEBUG:teuthology.orchestra.run.smithi097:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T20:13:18.407 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T20:13:18.484 DEBUG:teuthology.orchestra.run.smithi161:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T20:13:18.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T20:13:18.657 INFO:teuthology.orchestra.run.smithi097.stdout:type=AVC msg=audit(1710185749.273:199): avc: denied { checkpoint_restore } for pid=1164 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1 2024-03-11T20:13:18.658 DEBUG:teuthology.task.selinux:ubuntu@smithi097.front.sepia.ceph.com has 1 denials 2024-03-11T20:13:18.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T20:13:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:13:18.692 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T20:13:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:13:18.724 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 166, in teardown self.get_new_denials() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 215, in get_new_denials raise SELinuxError(node=remote, teuthology.exceptions.SELinuxError: SELinux denials found on ubuntu@smithi097.front.sepia.ceph.com: ['type=AVC msg=audit(1710185749.273:199): avc: denied { checkpoint_restore } for pid=1164 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-11T20:13:18.725 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T20:13:18.735 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T20:13:18.756 INFO:teuthology.task.internal:Duration was 2250.978177 seconds 2024-03-11T20:13:18.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T20:13:18.776 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T20:13:18.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T20:13:18.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T20:13:18.787 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T20:13:18.831 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T20:13:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T20:13:18.940 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T20:13:19.484 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T20:13:19.485 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2024-03-11T20:13:19.485 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T20:13:19.546 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-03-11T20:13:19.547 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T20:13:19.586 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com 2024-03-11T20:13:19.587 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T20:13:19.614 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T20:13:19.614 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T20:13:19.621 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T20:13:19.628 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T20:13:19.713 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T20:13:19.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T20:13:19.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T20:13:19.720 DEBUG:teuthology.orchestra.run.smithi161:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T20:13:20.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T20:13:20.281 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T20:13:20.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T20:13:20.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T20:13:20.404 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T20:13:20.433 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T20:13:20.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T20:13:20.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T20:13:20.454 DEBUG:teuthology.orchestra.run.smithi161:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T20:13:20.477 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-03-11T20:13:20.500 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = core 2024-03-11T20:13:20.553 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2024-03-11T20:13:20.607 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T20:13:20.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:13:20.655 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T20:13:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:13:20.670 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T20:13:20.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:13:20.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T20:13:20.697 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T20:13:20.698 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293/remote/smithi097 2024-03-11T20:13:20.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T20:13:20.815 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293/remote/smithi114 2024-03-11T20:13:20.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T20:13:20.855 DEBUG:teuthology.misc:Transferring archived files from smithi161:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293/remote/smithi161 2024-03-11T20:13:20.855 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T20:13:20.894 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T20:13:20.894 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T20:13:20.902 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T20:13:20.904 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T20:13:20.952 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T20:13:20.965 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T20:13:20.965 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T20:13:20.984 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T20:13:20.984 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T20:13:20.991 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T20:13:20.993 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T20:13:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 20:13 /home/ubuntu/cephtest 2024-03-11T20:13:21.007 INFO:teuthology.orchestra.run.smithi161.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 20:13 /home/ubuntu/cephtest 2024-03-11T20:13:21.040 INFO:teuthology.orchestra.run.smithi097.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 20:13 /home/ubuntu/cephtest 2024-03-11T20:13:21.044 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T20:13:21.055 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T20:13:21.070 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: SELinuxError: SELinux denials found on ubuntu@smithi097.front.sepia.ceph.com: ['type=AVC msg=audit(1710185749.273:199): avc: denied { checkpoint_restore } for pid=1164 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-11T20:13:21.070 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T20:13:21.096 DEBUG:teuthology.nuke:shortname: smithi097 2024-03-11T20:13:21.096 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T20:13:21.117 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 19:26:50.755900', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJlNkDZxM21kiCbCyIGT8AvJPTJYmU5+z6GdS64/hYfKQSwUfkugIdbhrNHJ2xMRntb0q5YkB74XuEcHdyFDCiY='} 2024-03-11T20:13:21.120 DEBUG:teuthology.nuke:shortname: smithi114 2024-03-11T20:13:21.120 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T20:13:21.141 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 19:26:50.754045', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBJ/ri8dGEp9lL36CgRPgII75UmcwO8RWA7h7c4PWmi4uOXvfSO7VKlYI6RxG49mGU4XOl1fiEPHfJ+apzkF91Y='} 2024-03-11T20:13:21.146 DEBUG:teuthology.nuke:shortname: smithi161 2024-03-11T20:13:21.146 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T20:13:21.170 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 19:26:50.757091', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJaVzOJvnEd6jcckfqck5YksHEswmVBjlrHM8bwCvd8txKxJyeXQDSDhBQnBUwDA6mSh3j0tXJkKCiyrt/oPwmU='} 2024-03-11T20:13:21.180 INFO:teuthology.orchestra.console.smithi097:Power off 2024-03-11T20:13:21.180 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T20:13:21.202 DEBUG:teuthology.orchestra.console.smithi097:power off output: Chassis Power Control: Down/Off 2024-03-11T20:13:21.202 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T20:13:21.218 INFO:teuthology.orchestra.console.smithi114:Power off 2024-03-11T20:13:21.218 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T20:13:21.230 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-03-11T20:13:21.243 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2024-03-11T20:13:21.243 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T20:13:21.258 INFO:teuthology.orchestra.console.smithi161:Power off 2024-03-11T20:13:21.258 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T20:13:21.268 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-11T20:13:21.280 DEBUG:teuthology.orchestra.console.smithi161:power off output: Chassis Power Control: Down/Off 2024-03-11T20:13:21.280 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T20:13:21.303 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-03-11T20:13:25.231 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T20:13:25.270 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T20:13:25.304 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T20:13:25.342 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-03-11T20:13:25.379 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-11T20:13:25.414 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-03-11T20:13:29.347 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T20:13:29.380 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T20:13:29.415 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T20:13:29.456 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-03-11T20:13:29.490 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-11T20:13:29.525 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-03-11T20:13:33.461 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T20:13:33.491 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T20:13:33.526 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T20:13:33.573 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is off 2024-03-11T20:13:33.601 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2024-03-11T20:13:33.636 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is off 2024-03-11T20:13:33.674 INFO:teuthology.orchestra.console.smithi097:Power off completed 2024-03-11T20:13:33.701 INFO:teuthology.orchestra.console.smithi114:Power off completed 2024-03-11T20:13:33.738 INFO:teuthology.orchestra.console.smithi161:Power off completed 2024-03-11T20:13:33.843 INFO:teuthology.run:Summary data: client.0-kernel-sha1: dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} duration: 2250.9781773090363 failure_reason: '"2024-03-11T19:48:53.729879+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-03-11T20:13:33.843 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T20:13:33.927 INFO:teuthology.run:FAIL